Will this continue? The field is moving too fast to tell. Kimi, Qwen, Deepseek also produce very capable models but that doesn't automatically translate into trillion dollar valuations.
However, trillion dollar valuations on Anthropic and OpenAI, such new companies, never publicly traded and such huge valuations decided just by investors. This is just asking for trouble.
1. https://docs.github.com/en/copilot/reference/ai-models/suppo...
Why is Google a monopoly? Do they have so advanced software that no one can outperform? I doubt it. What they have is a giant user base that generate loads of real-time data, which make Google services more accurate.
So how AI company can build a moat? Exactly the same way: by making a giant user base produce loads of real time data. Just imagine a service that will generate answers not only based on data they were trained on, but on all data from all user conversations. Imagine being at a concert, looking for a certain type of beer and instantly receiving an answer from an AI assistant about that only because some other guy in a crowd looking for exactly the same thing said to his agent "ah, here they are!".
It is not happening just yet because of making it secure and private is not yet solved, but it's just a matter of time I think.
A more appropriate term is “stateless”. LLM responses are certainly not idempotent, as they are not even deterministic.
https://earendil.com/posts/session-portability/
“Just take the session thread to another provider” might not be feasible anymore soon-ish.
The fundamental technology of LLMs and arguably AI in general strongly cuts against that sort of lockin. Handoff is a fundamental capability. There's no option to encrypt the docs or write it in some dialect only one model understands because humans need to understand it to, which stops that whole line dead in its tracks for at least the forseeable future. An AI can already today pick up such pieces, how much more easily will they do it tomorrow?
If they want to lock me in, they're going to need to provide a feature that I need so badly I can't switch and nobody else has. It is hard to see what that would be, other than being a generally better model.
(Only half-/s)
Which is why folks should be jumping to pi / oh my pi as soon as possible
This is also why anthropic no finger lets you use the flat rate sub in non-claude-code harnesses
(They do but you are charged at api rates)
This is harder to do on other architectures that themselves aren't fully deterministic though.
My understanding is that it's best effort because even if you control the seed/temperature, the model version and hardware specs you still have floating point math that will introduce rounding differences
an article explaining this in detail : https://news.ycombinator.com/item?id=45200925
You did mention, seed, which I haven't seen available anywhere else (in my limited experience) and it can explain a reproducible inference result, but I feel like this matches the "letter of the law" meaning of deterministic rather than the "spirit of the law".
Greedy decoding a single batch in most libraries will give you mostly deterministic outputs. Higher batch sizes can increase variance.
But all of this is down to CUDA and/or kernel implementation issues.
Isn't that more due to an optimization and not how the LLM itself runs?
Like a MoE LLM run on a single input should give the same output each time. But this is inefficient, as any given token is hitting 1 (or maybe 2 or 3) experts at a time, meaning all the other experts are doing absolutely nothing. So you upgrade it to take in multiple requests. But then any given expert can become a bottleneck, so when too many requests need a given expert, some of them are routed to a second or third best expert instead. Within the context of any single request, this looks like non-determinism, but it is still deterministic when considering the full batch.
For everyday users and everyday use cases, that is enough to treat it as non-deterministic (the harness might also send in unique data like current time which means one can never have the exact same request twice), but when talking about LLMs more theoretically, I think we need to consider they can still be ran deterministically even if that isn't as optimized.
Similar with temperature. 0 means deterministic, but anything higher with a seeded value is deterministic. If anything, temperature is us purposefully adding non-determinism to agents because they were too deterministic.
In the simplest possible case, a distributed addition of floating point numbers is not deterministic if you don't specify the order of the addition operations. If floats are added in a first-come first-serve fashion (the simplest implementation of a reduction operation) you already lose determinism. These could be activations from multiple experts, but it could also be adding chunks of a matrix operation that uses multiple CUDA cores.
If you manage to achieve determinism in temperature zero, it is possible to extend it to determinism at higher temperatures, because at that point you only have to keep the pseudo-random-number generator state in sync across parallel instances and this problem has been solved.
https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...
Note that the biggest GPU providers in the world are the hyper scalers and even they couldn’t allocate more if you pay for it. Because the rich companies and well funded ones are gobbling them up to the point where if tomorrow a 5T model that smokes every other model in the world is released you just can’t afford inference.
The bets are absolutely huge and have been spread throughout the financial system via securitization. If the number does not continue to go up, a lot of people are going to find themselves backing their numbers with money that simply isn't there anymore. Therefore, the number will continue to go up and we (meaning American taxpayers) will back whatever it takes for that to happen.
Now, were the bets stupid to begin with? Of course. Who wants a machine that potentially unemploys major percentages of the population? No one. But the bets were made regardless, and then more bets were made on the hypothetical returns of the first round of bets, and so on and so on.
They aren't, not even if we forget about the capable Chinese models.
I suspect Anthropic will implode soon when employees are unable to get the cash-out that they expected. Having so much compensation locked up in company stock is risky on a good day.
The first is the compute. OpenAI and Anthropic secured huge amounts of compute, Google, Meta and xAI have their own huge datacenters. Now anyone can rent some cloud machines and start serving Kimi K3, but it's going to be impossible to get to a similar scale as the big 5 above. And inference has economies of scale: the more people you serve in parallel, the more efficient you are.
The second is the data. By now (and maybe even by one year ago), all the data on the internet has been used for training. You need new data. The big AI companies sit on top of trillions or quadrillions of tokens that they have generated over the years. They can use that to train new models. That data is gold, and the proof is that SpaceX was happy to pay $60B to acquire Cursor.
If you want to overtake the frontier labs, you have 2 options: use their models to generate synthetic data, and provide lots of (cheap, maybe below cost) inference to generate your own new data. The frontier labs know about the first, and I'm sure they try to limit how much others milk their models. As for the second, that's the "honest" way to compete, but it's not easy.
They don't. The moat will mainly be the tooling around AI, not the AI itself. You don't hear any company claiming their moat is the Internet.
These companies have AI and enough money to lobby the Pope. They can afford to reanimate members of congress and push some tactical legislation through.
But all the money in the world cannot move government too quickly.
Other moats exist too. OS or browser can undermine performance and availability of alternatives.
Generally speaking they do, at least from my experience when switching from one model to the other - their performance decreases, and they often do large refactors outside of the requested scope as they try to bring the code closer to 'their' style.
Which makes sense imo - they'v been trained to iterate over the code they wrote, and not code that was modified by someone else in the interim.
Recursive self-improvement changes the parameters a bit, especially for the market-leaders, and it's the one thing that makes me wonder if they'll be able to extend their lead faster than the smaller labs can keep up, but it's an option available to everyone.
This isn't without it's challenges however.
1. This will increase costs drastically, since they would need to a run a sandbox per use to ensure data isolation.
2. Increased latency, and this directly limits how much of the harness can be moved to the cloud before the users notice sluggishness
It’s more like a bunch of people are placing different bets. Only a few bets are going to generate a return, possibly only one, but the profit on that one bet will make it all worthwhile. That’s the theory, anyway.
Fine tunes are a possibility but I think it offers very little uplift for the vast majority of uses beyond just stuffing enough context.
Which anyone who hasn't been sitting in the SV echo chamber could have told you years ago after applying even the smallest bit of thought.