This is an amazingly ignorant thing to say given the current pace of progress.
Even if both aren't true, your evidence was people saying two opposing things. The truth (if there is a single objective truth on a given thing) has little bearing on whether or not different people agree on it.
And there are benchmarks that cleanly separate the SOTA models:
Saturation of benchmarks is a property of benchmarks just as much as of the models.
We absolutely don't know enough about LLMs and intelligence to make such a bold (and ridiculous) claim. If anything, all evidence point to the contrary, with new scientific breakthrough achieved across a variety of fields via LLMs.
I've been really struggling to understand how the HN community can so boldly claim that LLMs are going to stop improving or not really smart. I just read it as the "denial" stage of the stages of grief that a good portion of this community is in right now (which is understandable).
We know exactly how attention layers work and how they produce the next word as well as draw them from larger feature spaces.
If we would know that, there would be no need for interpretability research.
I tried to generate the next word to the best of my ability, starting with a mathematical problem, but I did not create a valid proof. How do these LLMs work when they create math proofs to problems not yet solved?
> maths are internally coherent and entirely theoretical
Nope. This kind of wish-washy thinking is not what we mean by understanding.
This is like saying we don‘t know how a car works because a car can beat the best human athletes in 100 meter dash.
But I would bet that pretty much all of the improvement we’ve seen over the last year with coding has come from RL, not from the models becoming particularly stronger. And this makes sense, if models grow sublinearly with compute. And it seems like they do.
As a civilization, we are currently brushing up against the physics of efficiency. In many areas we have achieved close to what is theoretically possible, based on physics.
Such was not the case for the majority of human existence.
The body of research a.k.a. "writings of the ancients" is now insurmountably higher than it would have been during the time of ibn al-Haytham, when any kind of writing at all was scarce and literacy was low.
It's just inadequate benchmarks. Anyone who has used Fable for anything particularly difficult will have seen that it's miles ahead of Opus 5.0, yet the majority of benchmarks are completely unable to capture this.
I have seen people claim the exact opposite. If there was so much progress, then you wouldn't have endless disagreements with people championing their own favourite model as the strongest.
We already know from testing humans that test scores don't correlate that well with how effective a person is at work. Same applies here, we just aren't that great at making good tests.
What were seeing is all models failing to ace these tests.
"Benchmark Saturation" is term that promotes lowering the bar.
I think Epoch has the best analysis I’ve seen on evaluation trends; they use IRT to basically model a variety of benchmark difficulties, and then model a capability parameter for each model. This is as robust a sort of “meta-study” of evaluations as I’ve seen and the trend in capabilities show no sign of slowing down.
So I think people’s feelings clash with reality, and that’s because releases are more frequent and the jumps between releases are smaller, but the growth in capabilities _over time_ has not changed for the better or worse over a very very long period of time.
This is not "Acing" a test, this is hitting a wall.
If models can't get more than 90% of the benchmark right I think it's a strong indication that they were not trained on the answers and that benchmark itself is messy enough that <10% desired answers might be wrong or misleading.
Also to respond to the parent comment: benchmarks have a variety of difficulty levels. Humanity’s Last Exam, though now hitting the beginning of a saturation phase with Fable, was long unsaturated while other benchmarks saturated awhile ago. So that’s what I meant by Epoch capability index: using IRT models this effect so that you gather robust signals from variety of benchmark difficulties and can track progress over time as model capabilities have evolved (and so benchmarks have had to evolve to keep up).
But yes like I was saying: all benchmarks are problematic, some are useful. Benchmark quality problems abound, so 90% being the true ceiling is not surprising. There may be other factors at play here too, I haven’t studied this problem that deeply to have a good thorough answer to this. But keep in mind there are probably 50,000 benchmarks in the literature and that is not a joke number. A crapload of noise in that signal but it’s not all noise.
Most "AI" is really an optimization algorithm in software tools, same as its always been. This really isnt anything new, aside from adding a chatbot / MCP interface to the same tools.
Frontier labs have categorically different & better set ups for evaluation, they're fine. It's work but it's not a crisis.