Commenter below gets it absolutely correct: stockfish, which runs on your 5 year old phone, is dramatically better at chess than Fable. Like, so much better that it’s not even remotely comparable. The theory of the Bitter Lesson, and it’s only a theory, is that LLMs could eventually outperform stockfish. It’s not true today and it remains to be seen whether it will ever be true. For now, specialized models are absolutely better at specialized tasks.
I can't find the comment you're referring to, but the latest versions of stockfish are based on neural networks trained on millions of games, so if anything the Bitter Lesson turned out true here.
Not really, if anything it's closer to the opposite. The Bitter Lesson essay literally has this as an example:
> These researchers wanted methods based on human input to win and were disappointed when they did not.[1]
and
> Enormous initial efforts went into avoiding search by taking advantage of human knowledge, or of the special features of the game, but all those efforts proved irrelevant, or worse, once search was applied effectively at scale[1]
The actual bitter lesson is this:
> breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach.[1]
Applying to the "LLMs-for-chess" example the bitter lesson approach would be to put many, many more games into the LLM.
Does this work? People have trained fairly small LLMs that are competitive Stockfish at the ELO 1500-2000 level, eg: https://github.com/kinggongzilla/chess-bot-3000
This seems to be evidence that large LLMs probably don't have as much chess training data as Stockfish does.
[1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html
The bitter lesson is that simply scaling training on more games—including self-play—trumps any hand-crafted human input, whether that's fine-tuning on human commentary or clever engineering tricks.
Current models are just high-dimensional interpolation engines. The denser the data sampling, the more accurate the interpolation gets. Given a choice between denser sampling and anything else, denser sampling always wins. That is the bitter lesson.
Computer chess is the canonical example of this.
In the case of stockfish, the harness is a tree search around the neural network evaluations.
The common mistake is to think “maybe if we use a blend of raw data and hand-crafted heuristics, we’ll get the best of both worlds!” But the bitter lesson says no, beyond a certain point it’s better just to use the data.
Thinking that an LLM might be able to improve on purely “big data” machine learning seems to me to be the same incorrect idea. Its “intelligence” is no more useful than human intelligence. The LLM is based on a massive data corpus, sure, but the amount of data specifically about chess in there pales in comparison to just playing billions of games of chess.
> maybe if we use a blend of raw data and hand-crafted heuristics
I don't follow. They're suggesting giving raw chess data to the LLM, no heuristics involved.
Just kidding of course
It would be better to compare models at how well they can write the code for chess engines, otherwise it's just saying that Fable is not a good CPU emulator, which is obvious.
Stockfish is the best chess search engine we've got, and you can learn some good heuristics for chess search policy that will make time-limited chess search a lot more powerful. That's perfectly in line with the Bitter Lesson.
In contrast, LLMs playing chess are relying solely on learned behavior. The inference harnesses surrounding them aren't designed to do chess things, they're designed to do autoregressive token decoding, which isn't a search process. Reasoning traces can resemble a search process, but they're far less efficient - the LLM would have to work out each legal move, test each one, calculate a score, and simulate minimax over all of that. Assuming the LLM is smart enough to even do all that.
A hand-crafted approach can absolutely beat data if your approach unlocks more search and/or learning than the general solution.
Now let's look at the bitter lesson again. It says that general methods that leverage computation are ultimately the most effective, and by a large margin.
That's different from just saying to leverage computation (which is how I would interpret "unlocks more search/learning"). If the lesson is "more computation wins, when sufficiently channeled" you're basically looking at a truism. Of course more computation beats less when it's used right. The bitter lesson is about abandoning specialization in order to get more computation, and while there's a couple ways where that helps with chess, there's a lot more ways where it's counterproductive. It looks like it's more true for Go than it is for chess, and that it's not universally true. It probably correlates with the state space.
And perhaps at the end it all gets a single pass by a god-tier model for overall sanity and congruence, but the actual work, planning, coordination, and even user interaction was done by cheaper and faster agents of much more limited capability.
It really is absurd to ask programming questions to a model also trained about the lifecycle of a fruit fly.
Instead of building small models from scratch, we train an enormous model and use ridiculous amounts of GPU memory. In the end, the whole thing is shoved into RAM because we don’t know where the useful parts are…
We certainly would know where they were if they were just in smaller models in the first place!
Dumb AIs are needed for customer service. Most of that industry is still at "press 1 for sales, 2 for billing..." and needs something that will run locally on a 1U server.
GP isn’t suggesting that focused narrow model(s) will be more capable than large model, but that many small focused models can have sufficient capability while being more optimal.
Also, the bitter lesson is just wrong. The bitter lesson is about hand tuned AI vs computational general methods. However in truth today’s AI uses both. We have general compute heavy models which require narrow expert instructions (eg tools internet docs).
LLMs would not be as good without expertly written context, and expert context without LLMs aren’t as good either.
The models are not even really trained bitter lesson-style anymore. That concept peaked during the era of pre-train scaling, back when it was thought that making a bigger and bigger GPT-3 would automatically solve all problems through prompting. In 2026, the most important part of training is post-training, which uses vast quantities of niche, hand-curated data to fit the models for specific tasks in domains like tax law.
If you look at value as purely the LLM output, then there's a valid argument that the best frontier models will always be better than fine tuned specialists. (I'm not convinced personally, but it's a defensible claim)
But that misses two dimensions: 1. The cost of acquiring that output 2. What is actually "good enough" for that specialist domain
Not every output needs to be the best to produce value.
And as specialist models increase in cost, their cost/value proposition goes down.
At some point, there's a threshold where cheaper, fine tuned models are "good enough" at the task and also substantially cheaper than the expert models.
That's where fine tuning helps.
Personally, I became a believer in fine tuning after fine tuning a 1B Qwen model as a second pass over my local voice transcription app, achieving excellent accuracy at ~zero token cost and waaaay lower latency than if I'd invoked my Claude subscription under the hood.
Absolutely false. At least when it comes to multimodal inputs, even a simple classifier will outperform the largest LLMs who still hallucinate details or don’t describe audio and images accurately.
And there’s also the issue of cost/inference speed. Running a trillion parameter model for all tasks will be incredibly costly, require a cloud API, while a tiny CNN can be run locally or at a cost multiple orders of magnitude lower.
Bitter lesson #1: don't waste time optimizing code when a faster processor is around the corner.
What countered it: Moore's law stopped working.
Bitter lesson #2 similarly relies on scaling laws that might have diminishing returns wrt model runtime vs intelligence. Runtime matters for turnaround on the problem you're solving.
However, what most people think of as Moore's Law--CPU speed doubles every 18 months--broke somewhere between 90nm and 22nm.
And even the actual Moore's Law--2x the transistors every 18 months--doesn't hold for all types of chips anymore. Memory only gained 2x density over 10 years.
Secondly, the bitter lesson is predicated on compute being cheap. There was a period where a hand-tuned algorithm informed by human expertise would outperform a raw alpha-beta search at Chess. Then compute got cheaper, and DeepBlue ascended to the top. Compute is now expensive again relative to the tasks being performed. We are absolutely still in a period where human expertise in training LLMs will outperform a naive approach with more raw compute.
In the latter case, the chess example would tend to support the Bitter Lesson, rather than refute it.
I would also be VERY slow to claim that general-purpose models will never be competitive at chess. It wasn't so long ago that transformers couldn't add two-digit numbers reliably without resorting to tool use. They are now as good at "mental arithmetic" as any human savant. It wouldn't surprise me at all to see someone come up with a model that just happens to be really, really good at leveraging the portions of its general training data having to do with chess.
In fact you could argue that AGI demands such a model, if we are to assume that LLMs are a guidepost in that direction.
And in 2020 Stockfish 12 adding some NN evaluation. And then in 2023 Stockfish 16 entirely removing the classical position evaluation code.
> I would also be VERY slow to claim that general-purpose models will never be competitive at chess.
This is not the claim. The claim is that for the same amount of compute, a general-purpose language model will never beat a Chess model. I'm dubious, but allow for the possibility that a language model could eventually compete at a top level against humans with enough compute. However, it will never compete with a dedicated Chess model with similar resources. Training a model for a specific task with the same amount of compute will outperform training a general-purpose model with the same amount of compute. This should be common sense, right? The bitter lesson was only about compute over human algorithms, not at throwing compute at a generalised domain over throwing compute at a specific domain.
You made arguments against two claims that I did not make (that I was trying to refute the bitter lesson or that I claimed that LLMs could never be competitive against humans at Chess), so I'd like to ask you read my statements a little more carefully this time.
The actual argument of the Bitter Lesson essay is pretty limited but people's interpretation of it has gradually drifted until it's seen as prediction that current LLM will reach AGI at a large enough scale.
VibeThinker 3B constitutes extraordinary evidence, IMO. The first such evidence I've seen myself. Very small model, very low literacy, almost no world knowledge, but it is as good at math and logical reasoning as models a hundred times larger.
The Bitter Lesson is a valid and trenchant observation about how about we got here, but I think it's a mistake to assume it tells us very much about where we're going. Too much has changed recently and is still doing so.
Is this whole thing than maybe a read vs write optimisation again? Spent more time and effort training more knowledge into the model upfront and get it out in a single question instead of training a small model and needing more steps to answer the same question?
Any similar model aimed at coding?
A >10B model for mass spawning/swarming and reporting back to a larger model
I wouldn't use it for anything important without heavy supervision, as it's very weak outside its specialty. Not ideal for instruction-following tasks.