Bitter lesson wildly overstated in this context.
(had to look it up)
That may not be the intent of the original article, but over the past few years that’s what the phrase turned into.
As GP said. More RLHF is in fact the bitter lesson.
My sense of the Sutton Dwarkesh interview was that he was calling out that he didn't mean just longer datasets, but rather learning through exploration and that's exactly RL.
I agree with the sibling comment, effiency is probably the more important component at this point. We are hitting not just a practical engineering roadblock for scaling with current technology, I think we have definitely hit a financial and logistical roadblock for up scaling with the number of GPUs (on an immediate basis)
I'd imagine they're going to 10x this, maybe 100x this.
Right now, we have models that are statistical models of language, with a world model and reasoning "falling out" of a lot of effort.
It's like we've made something that's a little bit intelligent, and now we're trying to amplify that trick to create something that's quite intelligent. And - don't get me wrong - it works.
But it's also super, super inefficient. We're having machines "think out loud" to compensate for the quality of their thought processes. We elongate the path to make up for the progress made on a given step.
I tink there's probably a much smarter way of doing things that will require qualitative architectural (and quite possibly hardware) innovations. Right now we're on the path to a Dyson sphere: that's probably not going to be necessary once we figure out a smarter way to think.
It would be nice to see on which categories of problems the extra thinking makes it better and on which it makes it worse.
Richard Sutton specifically states that the search has to be smart. We know that the brain uses recurrent connections and is shallow. I think a lot more money has to go into architecture. Feed Forward transformers can only scale so far
When Mythos was announced after that, I was pleasantly surprised to hear about it. But when it turned out to be only two times bigger, I was a little disappointed!
(I am even more disappointed with the safety filters, but that's kind of a separate discussion... "Fortunately" I find that I can usually edit my prompt by single character and get through...)
Isn't this just the difference between getting 0 right and getting 1 right?
Or a breakthrough in algorithms etc.
The human brain, heck all bio brains, are proof that you don't need a lot of power or size for intelligence.
The real message of the last 15 years has actually been the opposite: if you throw enough processing power at it, intelligence emerges.
[1] https://www.sciencedirect.com/science/article/pii/S193459091... [2] https://pmc.ncbi.nlm.nih.gov/articles/PMC5063692/
Yeah, people might object, but it can be argued that we are already subjecting scores of animals to horrors beyond comprehension just to get a bucket of chicken wings. And even if we manage to get silicon to do what brains do, it will likely cost 1000x as much and consume 1000x the power like you said.
It's hell of an economic incentive.
Replay of Sol attempting the game: https://arcprize.org/replay/83543d22-8e1e-439a-8809-129ff1d9...
It seems a weird and arbitrary challenge for a language model to be expected to perform. It also seems like there are some harness/visual issues even in the first few steps, where it states that it hasn't moved when it clearly has.
I think it is historical name. At some point when benchmarking was very undeveloped, this was targeting abstract reasoning and generalization, hence AGI.
I was thinking about those species earlier in the context of, what does intelligence mean outside of language.
The benchmark appears to be testing the same thing. Although I don't know how much transfer there would be between this data set and the kind of situations a crow or an octopus would encounter.
Edit: Huh, it's just a Game boy game? I just did a couple of the tasks. It looks like C64 era game to me. Navigating levels. A lot of overlap with animal intelligence then.
Fable's main advantage is that its average solution size is smaller. However, GPT 5.6 Sol is a substantial improvement from GPT 5.4/5.5 which would write verbose, defensive code. 31KB for GPT 5.4/5.5 down to 26KB for GPT 5.6 Sol, with better performance for Sol.
Fable scores slightly lower, but with an average solution size of 12.2 KB.
I wonder why nobody has tried to optimize for actual code size or complexity metric, or at least why I haven’t seen more benchmarks that display this. GPT5.5 just keeps pushing more and more pointless indirection into every function it writes in my main project, it’s borderline negative productivity.
P.S. I’d be curious to see Cursor’s composer models in there, they seem to be among the best performing low cost models: https://artificialanalysis.ai/articles/cursor-composer-2-5-c...
For my usage, I would very much prefer if those $/task were being spent in thinking and experimenting, and the actual output would be as short and maintainable as possible. “maintainability” is a vague target of course, but it’s at least somewhat correlated with code size.
Yikes
Also what does this tell about Yann LeCuns whole world model theory? Bro has been going on and on about it. He has made multiple wrong predictions on the trajectory of LLMs.
At some point his claim should be fully falsified no?
And yeah.. Reality has not been kind to LeCun.
JEPA is just getting started
There's many ways to skin a cat so you can probably do something with a JEPA approach as well, but I doubt he actually catches up to having agents on the level of where Anthropic/OpenAI will be at any point.
What exactly was he dead wrong about that is proven by any of this?
GPT getting better has absolutely nothing to do with completely disproving anything LeCun has been saying.
He never said LLMs couldn't get better. He never said they couldn't score 7.6% on Arc AGI 3.
He's merely said they don't think, and you probably want something that actually thinks if you want a model that can be trained cheaply on a small amount of data and provide a ton of value.
Spending $5B to train a model that scores better than an older model does not disprove any of that in any way.
He said years ago even 'GPT 5000' couldnt do things that they ended up doing fine a month later, let alone by 5000. His later predictions are just moving that goal post including towards them not being able to do more general, harder problems of which Arc AGI is a counter-example.
What things specifically and when?
You probably wont like the edit but I dont have the timestamp of the original on hand, you can find it.
LeCun's ideas cannot be reduced to a 6 second clip...
You're missing the forrest for the trees, taking a singular example of a problem and thinking that if an LLM can solve the singular example it completely disproves LeCun is comical...
Ive read and watched more of his interviews and lectures it seems, it feels like you just have a rosier idea of his views than the views he repeatedly presents.
That LLMs don't have common sense and don't have good physical reasoning abilities; that you can't scale LLMs all the way to AGI; or that they can't predict the consequences of their actions which is the foundation of agentic behavior all seem like still (mostly) accurate predictions to me.
While LeCun has his share of problems, I think largely his criticisms on LLMs are more right than wrong. What remains to see is how good JEPA can be at filling in the gaps left behind from the brittleness of LLMS.
The human brain manages to self-organize with only a fraction of the information that LLMs get trained on. To train an LLM you need a lot of high-quality data.
There are two threads in history: firstly the compute thread leading to GPUs and AlexNet in 2012. Secondly the model architecture thread that started long before we had the compute and lead to transformers in 2017.
If the compute thread had been 30 years behind then we might be spending this century coming up with better architectures to make do with the more limited compute. However since the compute came first, we settled on the first thing that worked (transformers) and all effort went into polishing that.
There's something wrong with transformers though. No matter how many trillions of dollars get thrown at the problem, they still don't learn like humans do.
So? The question isnt can we get to ASI as efficiently as a brain, the question is can we get there, which we likely can. The inefficiencies can also be fixed after that.
>No matter how many trillions of dollars get thrown at the problem, they still don't learn like humans do.
Again, so? Humans are efficient but also bad at many things that transformers are already better at because of it. You are looking at the wrong thing if you think it needs to be like humans.
To me LLMs have gotten better since 2024, but their fundamental flaws still seem there.
They hallucinate when it comes to really challenging tasks such as math proofs. They still do not reuse code well and will rewrite functions instead of perusing the standard library.
But this is good news. LLMs are awesome and they are only the first step towards AI being applied everywhere. They are a Model T
He said as you need more and more tokens models will fall apart because each additional token is a chance for a mistake and they will just exponentially fall apart. But in practice models have learned to identify and self-correct mistakes and if you look at the graphs more inference reasoning tokens almost always give far better accuracy.
His other criticism of LLMs that I like better is that they try to predict tokens instead of learned embeddings. Tokens are arbitrary and in order to decode LLMs you need technical analysis (see mechanistic interpretability).
With JEPA models so far, it seems that PCA on latent vectors suffices.
tldr: embeddings have a lot more room for improvement
The difficulty in predicting a latent is so called "collapse"; the embedding neutral network can always output the zero vector and this would predict the output correctly.
There are different ways to solve this, DINO uses two different models - a teacher and a student and LeCunn uses an explicit term against collapsing to a single output.
Yann mentions DINO in his talks
"we use a rough estimate of a total 9M GPU hours"
From CoreWeave, at current prices (~$2.46/hr spot to ~$6.16/hr on demand) would correspond to $22M–$55M.
The dataset is really where the cost is though - they used LVD-1689M - 1.6B images of curated web data from roughly 17B instagram images. This probably cost a huge amount of hours in human annotation, compute for algorithmic filtering, etc and not to mention probably a 20-50 person team working on this model.
You might want to change assumptions about how expensive these models are.
The 9M GPU hours includes the DINO v2 inference used in order to curate the data set.
The final training run used like 300000 dollars of compute.
Unfortunately we don't know how much RLVR + Agent training costs these companies. I'm just gonna say it's in the hundreds of millions, because they are supposedly making billions of profit on inference yet making billion dollar losses
Mythos can do some amazing things (I'm assuming, I've never seen it). A young child can learn to control its body without reading any books on dynamical systems and kinematics. Mythos cannot learn to control a humanoid robot after sucking in every piece of data Anthropic can get their hands on.
I’d not wager against him having at one one more break though architecture before he retires.
It's telling
As far as the lack of shipping, they're scientists and what we're doing now with LLMs is more "engineering."