With that said, maybe he meant inevitable as N approaches infinity... So there's still time!
Aside: I think the biggest interesting prediction he had was that AI would obey a user driven network effect. It seems like a very clean prediction, and yet it has failed to materialize? I wonder if this has more to do with just how scrape-able the whole internet was or with distilation or what? There is certainly an advantage to having good user data, but it seems like it is largely something that can be worked around (by hook or crook)?
Possibly the best photography book I've come across. It's out of print, but I managed to snag a copy on eBay last year
> "By 2026, Google’s main product will not be search but AI."
Nah. Not by any common-sense measure at least. i.e. not if you ask a person on the street "what is Google's main product?", not if you look at where their revenue comes from. Perhaps if you look at where their staffing/spending/attention is going, but I don't think it's fair to say AI is their "main product" yet. (disclaimer: I work there, not directly on AI)
> "oligarchy of two or three large, general-purpose cloud-based commercial intelligences"
Directionally, yes. The number is too low, though, at least for now. As someone not even super closely following everything, I can rattle off more than 2 or 3. OpenAI, Anthropic, Google, xAI, Meta, Microsoft, DeepSeek + other Chinese players... Probably in the eyes of the public it really is just 2 or 3 (ChatGPT + something from work, perhaps).
> "The AI on the horizon looks more like Amazon Web Services"
Yeah.
> "The bulk of AI will be special purpose software brains"
Feels underspecified... I'd say this is a miss. "most of the commercial work completed by AI will be done by nonhuman-like programs" Most by what measure? Most invocations/day? Most users? Most economic output? Most public awareness? "AI" or "ML" in one form or another is ubiquitous and indeed if you were to make a giant list of "AI" or "ML" applications it would probably be dominated by hyper-specialized non-human examples, e.g. the image processing in your phone's camera, your Spotify recommendation algo, image recognition on factory floors. But in the public consciousness, the term "AI" means know-everything, do-everything human-like chat/assistant systems.
"in conversational tones, any question at all"
Yeah.
People sort of generally refer to the Bitter Lesson as if its key argument is "Large Transformer models are best", but this is directly from the essay:
"One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning."
The second paragraph of the essay is specifically about chess:
"In computer chess, the methods that defeated the world champion, Kasparov, in 1997, were based on massive, deep search. At the time, this was looked upon with dismay by the majority of computer-chess researchers who had pursued methods that leveraged human understanding of the special structure of chess. When a simpler, search-based approach with special hardware and software proved vastly more effective, these human-knowledge-based chess researchers were not good losers. They said that ``brute force" search may have won this time, but it was not a general strategy, and anyway it was not how people played chess. These researchers wanted methods based on human input to win and were disappointed when they did not."
The "brute force search" he's talking about (Deep Blue) used tree search with alpha-beta pruning on top of a ton of human crafted heuristics and a database of game-specific information it could access. Stockfish has historically been an even more "brute force" approach in that it is much more focused on optimizing search--which is one of "arbitrarily scalable" techniques listed in the Bitter Lesson essay--and it has increasingly replaced handwritten human heuristics with neural networks over the life of its development.
At no point in the Bitter Lesson is there a mention of LLMs driving cars, or a suggestion that large transformer models will be the dominant class of model for every single task. It is specifically reflecting on the fact that approaches that leverage search and learning, combined with large amounts of compute, consistently prove to be most effective. Stockfish would qualify as evidence of that, under his criteria.
At the most abstract level, you can observe patterns in a model's activations and theorize that particular collections of components contribute to particular kinds of operations. There are many flavors of this intuition. In that sense, you can think of there being "submodels" within a model, insofar as different parts of the network can participate in specialized computations. Some reading that might interest you in this area would include things like mechanistic interpretability.
There are also Mixture of Expert models, which are a dominant architecture for LLMs now, in which you essentially have a bunch of smaller FFNs that get routed over/aggregated together when the LLM performs a forward pass. It would be a mistake to treat any single expert as a better task specific model though. They are trained as part of an ensemble, and they're almost always being combined with other experts in the forward pass, so they're more like (to use a loose analogy) regions of the brain that tend to correspond more towards particular actions than others. You couldn't just extract one and say "This is now our Chess model that performs as well as the larger LLM we took it from". You could potentially use the relevant experts as a basis for training a smaller task-specific model, and there is work around expert pruning and other distillation techniques, but it's not as simple as "This smaller inner model is a complete task-specific model as is". And more importantly, that ensemble of experts *is* the LLM you're comparing it to, at least in major part. It doesn't really make sense to treat the LLM as some foreign model that calls the expert. The experts themselves don't often correlate to easily interpretable task domains, as they've been learned via joint computation with the other experts and broader LLM.
Then there's tool calling. Today LLMs are trained to make tool calls (basically execute functions) for particular tasks. Those tools can be smaller specialized models, or deterministic code, or anything really that can be fired from a host environment.
The most striking thing is that almost all of his missed predictions were missed not because the technology was not available, but because society rejected or didn’t adopt that technology
OK, so imagine that in 1980, I predicted that we'll colonize Mars by the year 2020. Today, I'm arguing that I was right in my prediction - in broad strokes, we have the technology! Nothing is stopping us. It's just that politicians have different priorities.
Of course, that's a load of bull. You can't decouple predictions about the future of mankind from what said mankind actually decides.
I looked up the predictions on Wikipedia and I'd say that most of them are general enough that they'd hard to disagree with back in 1999 (there will be bioengineering, there will be portable computers, there will be robots, there will be AI), but the timing or the specifics are off on almost every single one of them.
For instance, by his prediction, we're already many years past a 100 year average life expectancy (with 120 years expected by the end of the decade).
I think his mistake was tightly coupling exponential compute with exponential progress in biology - which has so far not panned out.
I would argue that society would likely have been glad to adopt, but the profit-seeking motive got in the way (again).
I was focused on augmented reality largely in part because it was such a big enabler of all other AI technologies
However people have almost uniformly rejected AR glasses my entire career for social reasons and that is only increasing
To the point where in 2016 I was debating with Jessica Alba and she was firm that they won’t take off cause most people don’t want to wear glasses