But that said: K3 is not a distilled version of Fable or Sol. Fable has been barely available and Sol was just released! Moreover, K3 is superior to both models in some domains, according to user scoring on the Arena.
API distillation can’t give you these results anyway. All it is useful for is bootstrapping RL in new domains to get past the “cold start” problem faster. By far, what matters more is the quality and variety of RL environments the model learns from.
People are going to be gobsmacked when, in our lifetime, China becomes a world power comparable to the U.S. Probably still poorer per capita, but at Spain/Italy levels, not third world country levels. And they’ll be shocked at the implications of that on the world economy, migration patterns, etc. There will be fields where China is a global leader, and Americans and Europeans will have to learn Chinese and move there, or else be stuck in some satellite office of a Chinese company. We’re all in Europe circa 1895 not realizing the behemoth America will become in WWI.
Also, yes, often.
Thus far the US has not really chosen to go the Chinese rare-earth method yet. The problem with distillation attacks is the end result is everyone who is not doing them is going to deal with some kind of regulation whether it's complete loss of access, or the amount of control you'll have to give up to access them will be ridiculous.
Sort of like the "stealing music is fine" but "lets freak out now that it's producing visual art", in the end the entire thing is a social construct. Whether this is treated as theft or "business as usual" is entirely societal.
Eventually the gap will close, unless there's a major breakthrough that hasn't been made yet.
Taxes on AI subscriptions or AI capable hardware, to financially compensate IP holders for (potential) IP theft, could very well arrive in the near future, once the industry is mature.
If this shocks you and sounds preposterous, I'll remind you that in several EU countries, we still pay extra taxes on any and all storage mediums and on devices with built-in storage (tapes, CDs, DVDs, HDDs, SSDs, tablets, phones, etc) simply because they can be used to store pirated content, decisions based on laws from 50-100 years ago, and the money goes to the national unions and associations of music and arts IP holders. It's basically a lobby pushed and government legalized extortion racket that no voter agrees with or can change but has no choice but to conform either way.
So I guarantee you in the future, it will be the same for AI subscriptions and hardware capable of running LLMs locally. Every time you purchase a Claude or ChatGPT subscription, an Nvidia GPU, Intel/AMD SoC PC or an Apple/Qualcomm powered smartphone, you'll pay a government enforced tax to the likes of Sony, Axel Springer, etc. for licensing their IP, whether you want to or not. In the EU at least. US maybe not.
Under the settlement, Anthropic was forced to delete the pirated data they were training on.
Chinese labs can still train on pirated data. I doubt the Chinese models operate under similar licensing agreements.
The payment was for illegally downloading copyrighted material, not training. Training was explicitly ruled to be fair use.
Training on legally acquired / licensed data is potentially fair use.
And other district courts don't agree on this. The US district court for Delaware recently rejected a fair use defense for the use of copyrighted works to train AI. https://www.reedsmith.com/articles/court-ai-fair-use-thomson...
There are more cases in the pipeline. The massive NYT vs OpenAI is still ongoing. Nothing will be "settled" until this makes its way to the Supreme Court or Congress steps in.
> I doubt the Chinese models operate under similar licensing agreements.
US corps likely pay licenses when afraid to be sued, or have troubles getting that data, otherwise they just take data, which was demonstrated many times. The same apply to Chinese corps, alibaba totally can be sued in US.
Chinese labs can freely train on pirated material, which is a structural advantage.
Meanwhile, Chinese labs are speeding in a different county. Everyone knows they are speeding, yet the sheriff won't pull them over, so they just keep doing it.
This lax enforcement gives Chinese labs a structural advantage over American ones.
Do you purport to know for a fact that they're no longer training on the data they'd pirated? Because I highly doubt that.
Destruction of Materials: In addition to the monetary compensation, Anthropic has agreed to destroy the two libraries that allegedly contain the pirated works, as well as any derivative copies originating from those sources. Anthropic must certify in writing to class counsel that the destruction has been completed and that the allegedly infringing materials are permanently removed from its systems.
The libraries in question were Library Genesis (LibGen) and Pirate Library Mirror (PiLiMi).
If Anthropic is training models on deleted data, I'd be quite impressed.
there is much less intellectual property in China so it’s not ‘theft’ (as you can’t put property on information)
I'm not aiming for a what about kickflip here: I'm saying we need to either agree on some rules or stop crying foul. Maybe the coherent legal theory is that neural networks and intellectual property don't interact. That would be weird but it would be consistent, a market could price it, I could do coding stuff and know if I was illegaling.
But this weird gerrymander that no judge will really rule on in an emphatic way is like, bad for the planet, bad for markets, bad business.
There are a lot of reasons to look forward to DeepSeek Huggingface drop kicking the unambiguous frontier weights in like, November, but I think my favorite one will be "who's distilling now bitch?"
But the American AI companies only let you query their models if you first sign a contract to not train on the output.
It's hypocrisy and unfair, but I think there's a strong legal argument for it.
Of course China can simply decline to assist in enforcing that contract... But I would expect US courts to do their best to.
Now THAT'S doing some heavy lifting lmao. The vast, vast, VAST majority of the original datasets were from pirated books and the like. Also, arguably a robots.txt is the exact mechanism to follow to do the mass GET-ing, yet the AI cos choose time and time and time again to simply ignore it and be as abusive as they possibly fucking can
And there's been significant legal consequences as a result
> Also, arguably a robots.txt is the exact mechanism to follow to do the mass GET-ing
You're free to argue this of course, but the courts have largely rejected it already pre LLMs. See for example hiQ Labs v. LinkedIn
Or do you just mean that US courts don't have enough teeth to prevent Chinese companies from violating contracts? On that I agree.
The US is already publicizing the way they are using Claude with Palantir for war gaming purposes. It’s a matter of national defense. Contract law has no meaning here.
That right there is the problem.
Maybe today. I doubt it tomorrow. Legal and not legal, largely, has to answer to the population sooner or later. Ultimately, humanity decides legality. And I don't think the frontier labs will get a pass from humanity in the midterm, let alone the long term. I think you'll see the rules change towards something more "intent" driven. And there's absolutely no difference in intent between Frontier labs and everyone chasing them.
Frontier labs just want the door closed behind them, as do their investors, because they know the money will never be recouped if others can do the same magic tricks.
Situation right now seems more like a fragile detente: if you got a Hill staffer drunk and hounded him long enough he'd probably be like "God damnit the market will fucking tank if we don't get these two IPOs out north of a trillion. And don't even get me started on how I'm going to sell Chinese AI to a Senate that still calls people Nipponesians when no one is looking. We're doing the best we can alright, get off my back man."
We have a situation, but it's not exactly A&M Records, Inc. v. Napster.
Oh it is, and at least anthropic has paid $1.5 billion and deleted there torrented copies and not released any models derived from them as a consequence.
The thing is it turns out to be not that expensive to just buy a copy of every book legally and scan them. And there's even precedent that this is legal predating LLMs (Google books)
I have a bridge to sell you
Facts are in fact knowable, and the US legal system is in fact not terrible at getting to them.
But we've gone through some pretty weird times too. Turn of the last century was pretty tech billionaire edits, reconstruction was uh, not smooth, it's a mixed bag.
And most takes I hear seem to acknowledge that this is one of those weirder times: serious election fraud rhetoric from most everybody from 2016 to the present, very politicized courts (on both sides to be clear), very soft on anti-trust, very soft on adventurous accounting. The Epstein files and like, no consequences (pretty much uniquely for a developed nation with Epstein people). It's weird right now.
And I think I would be hard pressed to think of a weirder part of this weird time than the rule of law meets AI. We can haggle on where laws end and norms begin (stare decis being maybe the midpoint), but in the 90s, the Justice Department got their brass knuckles on for a lot less.
I don't think it's a simple "the law works nothing to see here" story.
I can understand why as someone who didn't follow it and the more corrupt legal developments closely you wouldn't be confident in that.
Granting people some form of control over knowledge only serves the public interest inasmuch it provides incentive to create more of it. Mass media, effortless duplication, and copyright extensions had already broken this to the point where control of knowledge was suppressing creation of new knowledge more than it facilitated.
The world has changed, we need a mechanism that works for the public interest that applies to the facts as they now are.
> The problem with distillation attacks
Apologies for repeating myself here, but what you call "distillation" is function approximation.
I feel for the teams at Anthropic and Open AI, but unlike startups from prior eras; Anthropic and OpenAI have decided to be in the business of selling compute. Not creating a product that uses compute, but a product that's math running on compute. This is different from what Google is (or, rather was. As always, RIP Google 1998-2019).
Google's algorithm might be math, but Google search isn't. Google search is a process that's continuously operating in the background. Google crawls pages. Google stores and indexes what it finds. Google then exposes this to retrieval via its algorithm. User uses algorithm.
Now, let's compare that to AI models. When Anthropic serves Mythos / Opus etc, they're taking input or x from their user, doing compute, and then serving the result of the Mythos / Opus function, i.e.,
f(text) -> (text_transform)
According to Stone-Weierstrass, given enough values of y for f(x), anyone can approximate this function.
The fidelity and sophistication of this approximation definitely requires a lot of cleverness and effort, and it is arguably an imposition on Anthropic and OpenAI. But on a long-enough timeline, they don't even have to poll Anthropic or OpenAI. As the internet is flooded by PRs, content, emails written by Mythos / Claude, and just people otherwise sharing the results of Claude prompts, then there's an ever increasing set of data to approximate the f(x) that's f_Claude.
Eventually, in the future, anyone will be able to create a good enough approximation of the f_Mythos. Which is Anthropic's product.
Anthropic and OpenAI can now wage war on mathematics and the open-ended compute. Or, they can adapt and build a better product.
Choosing Option B was the Silicon Valley option / choice. I think the OG large-scale Valley lobbying effort, the Semiconductor Industry Association, was unique in that it prioritized and chose to do real research.
https://en.wikipedia.org/wiki/Semiconductor_Industry_Associa...
https://en.wikipedia.org/wiki/Semiconductor_Research_Corpora...
This helped the industry to survive and outcompete the pressure they were facing (at the time).
What a nice post hoc revision of history. Distillation is still an active area of research, that you can distill models as easily as you can it genuinely interesting and absolutely not something that was taken for granted even 12 months ago.
Even 6 months ago this idea that 'using model outputs as training examples' was listed as the reason that all models would fail in the near future due to some spooky circular training catastrophe.
Don't pretend like this was so obvious.
https://en.wikipedia.org/wiki/Feist_Publications,_Inc._v._Ru....
And then there's new updates related to AI that fully take out LLMs from protection.
yeah hardware companies make for nice stories or green numbers on Wall Street - but value will be captured by application layer.
look at history.
These Chinese labs are producing novel models, publishing their techniques and sharing their open weights and the first topic of conversation is how they stole from U.S. AI labs.
Setting aside the fact that it doesn’t make any feasible sense to do API distillation, these models are outperforming frontier models on a number of benchmarks, and often times run more efficiently by several orders of magnitude.
We have to stop crying distillation, it’s getting embarrassing and at this point feels even a bit delusional.
Anthropic stated in February that Moonshot AI (the creator of Kimi) distilled ~3.4 million exchanges from Claude models, as explained in their press release https://www.anthropic.com/news/detecting-and-preventing-dist...
DeepSeek and others like Minimax are publishing deep research on Multi-Head Latent Attention and Mixture of Experts, Multi-Token Prediction, novel Sparse Attention approaches, I mean they trained long context models on a fraction of the resources and gave everyone the recipe.
Chinese labs might not have the funding of labs like Anthropic, but at least they provide the receipts.
Assuming each session was 10,000 words each, that's 34 billion words; lets call it 50 billion tokens (0.05 trillion) unfairly pilfered from Claude. That left Moonshot needing to scrounge for the other 14.950 trillion training tokens required for a baseline frontier model.
They are used for post-training, i.e. calibrating the model to understand and use tools/command line more effectively.
That's an increase of only a single order of magnitude, increasing my estimate of exfiltrated tokens from 0.05 to 0.15 trillion - a far cry from the 15 trillion required.
> They are used for post-training
Possibly - it may be too much data for post-training, unless further curation was done. However, this is not distillation; you know it, I know it, Dario knows it, but "Distillation Attack" is a short, memorable, sciencey-sounding, political sound-bite with enough malevolence to be deployed on the floors of congress, or by the usual fear-mongering newstainment talking heads.
Nobody is suggesting Moonshot used 15 trillion tokens of Claude data to pre-train a base model from scratch. That would be impossible and nonsensical.
This is entirely about distillation, which happens during post-training (alignment and SFT). Here, datasets are measured in millions or billions of tokens, not trillions. 50 billion Claude tokens is far, far than enough to copy Claude's reasoning logic, writing style, and tool-use ability to the pre-trained base model.
> However, this is not distillation
I don't understand how you're so caught up on the term "distillation". Distillation is using a larger model's outputs to train a (weaker) student model. Which is exactly what's happening. It's a standardized term that has been in use for a decade.
It's a PR campaign - when they say its an "attack" they don't mean on Anthropic - but on America itself. What kind of American can let such a brazen attack go unanswered? At the very least, they ought to demand the dangerous, pinko, stolen models be banned in all 50 states, and pay whatever price demanded by the patriotic, freedom-loving, all-American AI labs that can never be accused of stealing.
what is the end game for this strategy?
if the frontier labs shut down, or stop releasing to the public, and there's noting left to distill, how will you progress?
I'm pretty sure that all labs are distilling each others' LLMs, maybe apart from Anthropic and OpenAI. It would be stupid not to do it, because it's cheap and effective. But that's not the only thing they're doing. If you think K3 and GLM-5.2 got this good only from distilling frontier models, you're not paying attention to Chinese labs' publications.
if this were not the case, then we would be observing chinese models that far surpass frontier models in capabilities, rather than "almost as good, but much cheaper", and we would be having a very different conversation. what happens to these efforts when the subsidy is cut off?
I see no evidence for that.
> if this were not the case, then we would be observing chinese models that far surpass frontier models
It's pretty clear that the primary reason for the difference is budget and compute availability. Chinese labs have at least an order of magnitude less money than Anthropic and OpenAI.
> what happens to these efforts when the subsidy is cut off?
They will continue making progress as they do now, minus the benefits of distillation.
Moonshot AI Scale: Over 3.4 million exchanges
The operation targeted:
Agentic reasoning and tool use Coding and data analysis Computer-use agent development Computer vision Moonshot (Kimi models) employed hundreds of fraudulent accounts spanning multiple access pathways. Varied account types made the campaign harder to detect as a coordinated operation. We attributed the campaign through request metadata, which matched the public profiles of senior Moonshot staff. In a later phase, Moonshot used a more targeted approach, attempting to extract and reconstruct Claude’s reasoning traces.
Translation: we have the machinery in place to identify our users, and actively do so.
Once a model becomes competent enough to perform complex reasoning, a teacher model is no longer necessary. The model can now reason about its own behavior and build a better version of itself through recursive self-improvement (RSI).
Kimi K3 is capable of RSI.
and what will fund these budgets exactly? inference is cheap, distillation is cheap, training is what's expensive.
Say it louder for the people in the back. All these complaints about "distillation" from frontier labs are bordering on felony contempt of business model at this point. It's great for us. Maybe it's bad for them but nobody other than shareholders really cares.
The optimal outcome for humanity is for oligarchs to spend trillions training a godlike AI, only for the precious weights to just leak. No "distillation" required.
If "distillation attacks" happen, we have to conclude there is some value add in what model labs do. Regardless of how we feel about using existing human knowledge in the way they currently do, it's simply impractical to infer that everything that happens downstream of LLMs can not be an attack on some IP because of it.
So both things can be true: a) People infringe on Anthropics IP and b) what Anthropic did to build their models is legally questionable (or might be ruled illegal, even though I doubt it).
No.
Authors do not infringe on IP when they read another's book, nor should the lumber company be able to dictate how I use planks and if I can resell them if i'm done with them.
You're framing it as if the added value of the author or lumber company, awards them consideration when somebody uses the products to create more value.
IP law was always a big mess, and these questions cross far into ideology instead of law; but I do not understand people who think we need an ideology where more IP-law is good for society.
I guess you could steal them but thats a whole other issue.
Are the distillers reading books or are they building models?
If anthropic is providing no value they can just build from scratch. But obviously distilling is easier. Hes saying thats the value they add.
This would demolish agent usage by corporations.
Unless someone literally stole the weights somehow (which is not out of the question, I doubt either oAI/Anthropic have the capabilities to prevent a state-level actor getting those weights), distillation from generations is not infringement on anyone's IP nor is it stealing nor is it an attack. It can't be. As long as you pay for tokens you get to do whatever you want with them. Someone saying you can't doesn't mean it's an attack or their IP or whatever. They either sell the tokens or not. They can decide to not sell them to anyone, but again that's not stealing.
And their ToS are a joke. Imagine how people would react if MS had ToS saying that you can't use MS software to develop solutions that compete with MS. They'd be laughed out of the room. Somehow it's ok for token sellers to decide what you do with the tokens? Why? If you pay for something you get to do whatever you want with that output. Train, distill, whatever.
How do things get "established" from someone's perspective, exactly?
By that logic it is established from my perspective that Anthropic has no right to train on anything I've written that is publicly available on the internet.
Of course, they don't care about my perspective, but then again I don't care about theirs.
Anthropic’s model outputs contain no IP. This is actually a simple legal proposition (rare in this field!) that derives from the fact that only specific classes of IP exist: copyrights, patents, trade secrets, and trademarks. Examining each, it is clear that API outputs do not qualify. Anthropic disclaims copyright in outputs; the outputs are not patented; the outputs are not secret (a prerequisite to having trade secrets); and trademarks are irrelevant in concept.
Anthropic, OpenAI, etc do not deserve legal protection.
that would be existential doom for them because then they have a case to claim ownership of their users' codebases
no corporation would sign off on that
Do you really think intellectual property laws will prevent this in practice? It’s like as if we said, “hey, USSR, you can’t make a nuke, too! We patented that already.”
Asking China to not distill our models down is equally as ridiculous.
How far are we willing to go as a nation (and as a species) to prove out the scaling laws? Are we willing to sacrifice our industrial base? Would we rather train models or smelt aluminum?
(I actually appreciated your analogy, despite my lark)
So why didnt we have these LLMs in 2005?
Don't forget to account for all the costs. It's not just that CPUs are X times slower. Memory is X times smaller, too, and networks are X time slower. And all this hardware is many times more expensive.
If I'm getting my mental estimation right, training a 2026-frontier-class LLM in 2005 would be somewhere on the order of all the computation power in the world at the time. It's not that many more factors of magnitude before you end up at "all the computation power in the world up to that point".
"Distillation" literally means to separate and take some components out of something. You can distill how a model works from a model. You cant distill a model from information because the information does not contain the model.
People are happy to conflate distilling with building because they dont like how the information was used. You distill how the model works from the model, and you build a model with information. Both could be morally good or bad but its not the same thing.
Not really, what the information actually is, matters a great deal. It's harder to get good results going from "nothing > model+weights" than "nothing + traces from known good sessions of other good model > model+weights", this is what the "distillation" part is referring to. If "information is information", you wouldn't even need to separate good from bad sessions while doing the training, which leads to somewhat obvious results if you don't.
To succinctly restate my point, you cannot distill a model from information because the model is not contained within that information. You can distill a model from another model.
1: That's the "T" in GPT fyi, even though Google is the author of the research paper that changed everything
So the initial models arent just distilled from information. We’ve always had the information.
The first "L" in LLM does the work. In 2005 you had no Github, Stackoverflow, Youtube, common crawl and no archive of digital ebooks.