The Chinese models are pretrained on large clusters just like OpenAI ones are. Yes, they use outputs of the frontier models to further improve the final model, but even without those outputs they'd still have very strong models.
It's not like in a world without distillation things would be much different as you claim.
Here is a project that guides you through it if you want to prove to yourself that it works https://github.com/arcee-ai/DistillKit
GPU kernel optimization is just the kind of well-bounded problem with clear success criteria that AI loves.
It's about evidence this is an active force in competition in LLMs.
[1] https://www.anthropic.com/news/detecting-and-preventing-dist...
It's also how providers build their smaller models out of their larger ones; they publicly talk about the process.
Make sure to stay updated!
That said, there are other moat factors like, a US company needing to use a US AI provider, sticky customers due to corporate onboarding friction, and others. Not nothing, but not as large a moat as some imagined.
There were somewhat good reasons to think it needed more than just this data-driven ML approach.
Then i think tool use became a priority or at lest a sibling priority to more data/more params. Along with multiple specialized models communicating with each other which is sort of a special case of tool use. That pretty much brings us to today.
Distillation was big news a year or even 6 months ago, but as far as we can tell it's not really a moat anymore. Now that multiple players have trillion+ parameter models and the capacity to post-train them, there's no putting the genie back in the lamp.
Besides, identity verification that actually works at scale is a much harder problem than identity verification which is good enough to satisfy your compliance people and regulators. Especially if the fraudsters have a major world government standing behind them, and if their aim is to be identified as a real customer, not one customer in particular.
And with the sheer volume of data created from that, coupled with benign-seeming prompts like "plan out your reasoning in a document before implementing" that could never be patched without breaking existing customer workflows... there's more than enough for someone to distill on. Even if that only gets them to not-quite-frontier, if you're pushing the frontier every few months, they're only ever a few months behind you.
The primary resource you need to train LLMs is money and China has plenty of that.
If I run out of tokens on ChatGPT of course I will try Claude. I never ran out of Google searches so no reason to try Bing
At some point Google gets suspicious of your persistent searches and makes you solve captchas and puts cooldowns on your searches.
More like the other way around - Claude burns tokens faster than any other LLM.
ChatGPT is AI for the average non-techie the world over, but the average non-techie isn't eager to pay for it. The more progress that's made, the less incentive to pay - most people are happy with the total garbage spewed by google AI overview. They'd be happy with google's 30b MoE gemma, whose performance will likely be squeezed down to something that can run on a phone in 2-3 years. Why would they pay $20 a month?
It's why OpenAI is pushing a variety of things such as ads and offer a more polished ui/ux than the competition, I think. The models are already good enough for people who just want to know how much sugar to add to their cake or when's the next basketball match their team plays - it's OpenAI's game to lose those people, by annoying UX and whatnot. If they can make a few bucks off of every one of their non-paying users it'll stretch their runway immensely. Those users will never go to Antrophic or some cheap Chinese model, but they might defect to Google because a popup on Android / in Chrome told them to.
I had a discovery call last week with someone who did not realize he could use ChatGPT for work. It was a revelation that he could drag a PDF into ChatGPT and it could summarize it for him.
FWIW, guy in his late-30s in a pretty senior sales role.
Wouldn't that just be price fixing? If they arrive at their prices independently and they all happen to be similar, fine. But if they're all "smart" and coordinate so none of them undercuts the other, that's probably illegal.
Personally, I came to this conclusion early this year. To acquire the data that AI Companies are using to train their models is low cost and once they have it, they can refine and store it. Creating the LLM takes a bit of money but it is not a serious blocker. Clearly, the Chinese companies can make AI so they will drive down costs. There is a need for good AI (Not just Great AI) and it is not cost prohibitive to make good AI (The same with specialized AI).
My prediction is that AI will spilt into two categories, Great AI (High Cost) and Good Enough AI (Low Cost). Which for the long run of AI and companies that use AI, this is good.
I live in Germany where people won't stop whining about electricity prices, and I pay 75€/mo.
He even raised money on that premise.
He is a pathological liar, so is Dario. Don’t rely on the benevolence or truthfulness of these people.
They will say whatever is beneficial to say in the moment.
Referring to a baseless prediction by Sam Altman that AI will become like electricity without any push-back? Who really thinks Sam is working toward that future?
He already worked to undo every early promise made (non-profit, open source models, strong governing board, strong ethics/alignment/security focus). He's flip-flopped on other things like first characterising Trump "an unprecedented threat to America", then contributing 1M USD to Trump's inaugural fund far exceeding his earlier political contributions. Lately OpenAI, under his supervision, has also been working with Anthropic to lobby regulators in Washington for restrictions on open weights models - why so if not to undermine a free market in favour of an oligopoly?
Beyond that, you have the simple fact that most of his personal wealth and very probably the fate of OpenAI hinges on AI inference NOT becoming an interchangeable commodity.
I mean.. Honestly. The naivete is downright astounding.
The quality of the harness UX, and random fun crap like Sora, it's a shame that OpenAI killed that so soon, and also Group Chats in ChatGPT.. they risk running a Googlelike reputation at this rate
Maybe ultimately whomever can be the "Apple of AI" will win
China or SpaceX seem like the 2 likely candidates in 5 years, but who knows.
If (a) demand for AI continues to increase, and (b) SpaceX can get to ~$100/kg to orbit, then they will have a ridiculously deep moat. Probably more like 10 years, though.
But as you said, who knows.
You can put AI chips in datacenters in the desert for far less than $100/kg. With lots of solar power available, the option to easily access your hardware and far less radiation issues.
The datacenter in space story really only exists to make it possible for Musk to sell X to SpaceX and make more money from the IPO. That's all. There is no engineering reason.
But anyone who thinks they can predict those prices in ten years is wildly overconfident.
Musk's bet is dysfunctional politics will make it impossible to build enough data centres and the energy needed to power them. There are many, many reasons that might be wrong. However, if the economics are even close to viable, they could start throwing up data centres quicker than anyone can build them terrestrially (at least in the democratic west).
GMAFB.
It's stock pump bullshit from a guy who has figured out how to extract the maximum from stock markets.
But yeah, I'm sure you people with Elon Derangement Syndrome actually have it all figured out /s
well, no, they announced the concept of a space-optimized Vera Rubin designed with SpaceX:
> NVIDIA and SpaceXAI are working to adapt that foundation to the requirements of orbital computing while preserving a common NVIDIA architecture and software ecosystem.
The press release is really announcing that SpaceX's terrestrial data centres are going to use Vera Rubin.
https://nvidianews.nvidia.com/news/spacexai-adopts-nvidia-ve...
if you can't put them outside of Amarillo Texas without people throwing a fit then you can't put them anywhere. I mean freaking Pantex is there ffs!
Cooling is probably the easiest problem to solve, easier than power. And in both cases, the problem is solved by mass to orbit. All you need for cooling is a big f-ing radiator. Solar panels are chips, and not trivial to manufacture. But a radiator is just a hunk of metal with some pipes.
That's why the cost of mass to orbit is the most important thing. You can solve almost any space problem by just throwing more mass at it.
If it doesn't work out, I think China's exponential terrestrial energy deployment will eventually give them the lead, IF they can get enough chips. Another big if.
The economy he and his ilk want to build is infinitely worse.
If investors start fleeing from senseless businesses in the AI sector, that does not mean that sensible businesses will be spared. These things follow herd mentality, and the primary drivers of the herd are greed and fear, not fundamentals or business logic.
A major correction would be a bummer but we were never entitled to these abnormal gains in the first place.