It turns out you can train a 1b model at almost 1000 tokens/s on a m5 max laptop. As a personal experiment, I've been asking Sol for synthetic training data and synthetic agentic training data (model distillation in it's purest form), plus modified opencode, codex transcripts etc for training data, and nobody's even paying me to do it. If I'm doing it has a hobby, you can bet industrial users are doing it.
replyDistillation a pretty well documented technique that actually pre-dates LLMs https://arxiv.org/pdf/1503.02531
replyHere is a project that guides you through it if you want to prove to yourself that it works https://github.com/arcee-ai/DistillKit
I took GP as asking for evidence that the reduced-cost Sol is actually a distillation of the previous-cost Sol. AFAIK, providers distilling or quantising models and offering them as the same model have not been proven.
replyI doubt he was claiming that. He's probably saying that the ability of Chinese companies to be able to distill frontier US models has put downwards pressure on the price of all models.
replyI'm saying that it is unclear that without distillation this wouldn't still be happening. There is a massive narrative that no one but OpenAI, Anthropic, and Google can make a model without distilling. But there's basically no evidence of that.
replyAlternatively, modern AI is good enough at optimizing its own kernels that it just keeps pushing costs down. Unlike the semi-decentralized inference provider community, OpenAI has both the talent and the compute to throw at the problem of making their models much more efficient to run.
replyGPU kernel optimization is just the kind of well-bounded problem with clear success criteria that AI loves.
That distillation exists isn't the question.
replyIt's about evidence this is an active force in competition in LLMs.
I think the biggest actual piece of evidence is how hard the major players are trying to stop it
replyTo be precise, the distillation mentioned in this paper is not the distillation used by other model companies. In the one mentioned in the paper, your teacher and student model typically have similar architectures - and you typically need access to the full logits. What happens here instead is motivated by the fact that these companies don't have access to the training data and compute that anthropic/openai have. The distillation they do basically amounts to using traces from ant/oai models trained on much more data with a lot more compute (in many cases including the hidden intermediate tokens! turns out there were many ways to coax it out) and then either directly training on it or using it in many ways in post training pipelines. It falls under imitation learning, IMO.
replyBeen using a lot of Kimi K3 lately and the answers have been… „load-bearing“ to the point of hilariousness. It‘s obvious from where they distilled, even if sceptics rightly point out it can‘t have been the only source of their secret sauce, as it‘s been better than the current Opus 4.x at the time of release.
replyWhy does Kimi insist its name is Claude?
replyPlease elaborate the mechanisms by which a LLM would know what model it is.
replyBy being trained on text containing "I am X" in the model response section.
replydeleted
replyask it what type of model it is or what it's name is... it's weird that Kimi will say it's claude...
replyOkay, but I would also ask “why does Claude say that its name is Claude?”
replyThe evidence is Anthropic's own reporting [1]. You may doubt that they're telling the truth, but that's what they're reporting.
reply[1] https://www.anthropic.com/news/detecting-and-preventing-dist...
Musk confirmed in federal court that xAI does it:
https://techcrunch.com/2026/04/30/elon-musk-testifies-that-x...
replyIt's also how providers build their smaller models out of their larger ones; they publicly talk about the process.
Here's an example: https://github.com/microsoft/Build25-LAB329
reply[flagged]
replythere is no evidence. it shortcuts post training by a huge margin this is true. but that is all.
replyhttps://techcrunch.com/2026/04/30/elon-musk-testifies-that-x...
replyMake sure to stay updated!
Nothing against what I said in there. Distillation is far from enough to get to the frontier. Its at best a ramp (the most efficient one used by everyone ) that shortcut and saves millions of rl runs before a model moves.
replyYou said there isn't evidence of distillation. Yes there is
reply