upvote
Moat is not the harness. Harness itself is temporary until the models get better and slowly the code in harness will go down.

Note that the biggest GPU providers in the world are the hyper scalers and even they couldn’t allocate more if you pay for it. Because the rich companies and well funded ones are gobbling them up to the point where if tomorrow a 5T model that smokes every other model in the world is released you just can’t afford inference.

reply
I meant that companies like Anthropic are locking in users with proprietary formats in their harness where it's hard to leave.
reply
Agree. Harness can not be a moat. There are many open harnesses and they are at least on par with the providers ones. It looks like Anthropic/OpenAI's approach to vendor lock-in is not so much the inference or the harness it is functional integration across the individuals and teams in a company. I don't think this will be a moat either, but I think it's all they have outside of compute.
reply
Surely the moat is the training data... with the data you can explore new architectures much easier and get step changes in performance.
reply
The training data, at least up to now, is very abundant and basically every lab has the same data from scraping the Internet. RLHF data is what's now valuable.
reply
Arguably the majority of codebases are not available on the public internet
reply
Is it RLHF if there is no human feedback?
reply
And yet investment is continuing. What are they counting on?
reply
Bubbles aren't always rational.
reply
One possibility is that they're counting on another bailout package like the banks and others got in 2008.

The bets are absolutely huge and have been spread throughout the financial system via securitization. If the number does not continue to go up, a lot of people are going to find themselves backing their numbers with money that simply isn't there anymore. Therefore, the number will continue to go up and we (meaning American taxpayers) will back whatever it takes for that to happen.

Now, were the bets stupid to begin with? Of course. Who wants a machine that potentially unemploys major percentages of the population? No one. But the bets were made regardless, and then more bets were made on the hypothetical returns of the first round of bets, and so on and so on.

reply