This lasted for 2005–2014. By 2015, the advantage humans added was negative.
I imagine this will be the case here as well, but I highly doubt it'll take that long.
The genius here is Claude.
It doesn't make sense to compare the quantity of work. Humans are significantly more efficient than AI, so little work can go a lot further.
I’m sure for every incredible insight which Levant credits to Claude, there’s 100x+ more bad insights that Levant had to filter out.
This is certainly my experience of using Claude every day, in a much less cutting edge domain.
And Claude itself cannot tell the difference between the two. The best it can do is use some sort of external constraint (e.g. verification) to filter them.
And then they are simply boldly wrong at times. About mathematical facts. That’s why we need formalisation when working with them. They are still stupid and can’t be trusted. But theyvare also smart and can come with amazing insight. It’s a tool, but definitely one which cuts both ways.
Do we even know what Alpöge had at his disposal? I think most people underestimate what working with an agent swarm even with the current public technology would be like. We can't assume he didn't have something more advanced either.
I don't feel confident my usage of AI on a personal level reflects any of what has been done here.
A non math person may not be able to comprehend the output, but even professional mathematicians have trouble reading through the huge “alien” AI generated LEAN formalised proofs.
On that topic, I'm surprised the usual anti-ai trolls aren't posting low-effort drive-by "ai slop", "stochastic parrot", "autocomplete on steroids" comments to this very discussion.
Edit: I stand corrected: RunSet, ToucanLoucan, luka598, and others have checked in.
We know the architectures of the open weight models as they include architecture diagrams, support in projects like llama.cpp, etc.
Companies like Anthropic and OpenAI don't release information about their models (aside from general things like context window size, whether they support reasoning, etc.). A lot of what we know is speculation. From press releases Anthropic have stated that they are getting Claude to assist with designing new iterations of the models, but no specific details.
The open weight models and other arXiv papers/research are iterating on and experimenting with the base/core transformer architecture:
- RoPE/NoPE/etc. for context window scaling;
- dense vs mixture of experts models;
- Mamba-based State Space Models (falcon models, etc.);
- reasoning, combining transformers with a residual stream, Attention Residuals (Kimi), etc.
- activation functions: ReLU; SwiGLU; etc.
- etc.
it does seem like (IMO) the biggest leaps have basically been perf & scaling + RL on high quality data + harnesses.
He just did the math and realized you just needed more and more compute to get more intelligence. People keep doubting it but it's showing no sign of being incorrect.
ChatGPT was the turning point of realizing we needed just more compute
Inference can be increased vertically (a single model thinking for longer), or horizontally (NS was solved by a 10,000-agent swarm).
And pretraining has two inputs that can be scaled independently - training data and model size. (Things like the Chinchilla 20:1 law are meant to establish the optimal way to train with a limited budget - you could just keep making either term bigger and bigger on its own if you had infinite money.)
I think creating and curating high-quality datasets is also part of scaling. Datasets like FineWeb are quite a bit better than the barely-filtered CommonCrawl dumps GPT-3 trained on, but someone had to invest in making them. (To say nothing of whatever exotic bespoke robotics/physics/etc datasets OpenAI and Anthropic are investing in.)
Ilya Sutskever has been saying more compute = better perf since at least 2012.
... the real conclusion now is that if you have a large big data set and you train a very large deep neural network success is guaranteed ...
https://www.youtube.com/watch?v=-uyXE7dY5H0&t=1185 In the end, the network's size is limited mainly by the amount of memory available on current GPUs and by the amount of training time that we are willing to tolerate ... All of our experiments suggest that our results can be improved simply by waiting for faster GPUs and bigger datasets ...
https://proceedings.neurips.cc/paper_files/paper/2012/file/c...As if drawing a line through few points on a chart is a mystic art.