It's unfortunate we're always focused on humans getting usurped by an "AI", instead of the reality that these models only succeed when we're there to show them what success looks like.
What if "superintelligence" is really a hybrid result of humans using and guiding these large pattern interpolators?
What if we're not the specialist snowflake?
But the human brain for now still has 100x the parameter count as the largest models. And they don’t have any circuitry for all the “keep yourself alive” work our brains are doing all the time, which may be the majority.
As with chess, the reign of the human-machine "centaur" as the greatest mathematician in the world will likely be brief. I'd guess within 5 years the humans won't be adding anything. I hope I'm wrong.
Obviously, humans with AIs are clearly more capable at a number of things than humans without but I don't think you can get out of the quandary of superintelligence that way.
Humans in the loop will always offer the opportunity of training and putting humans out of the loop.
And LLMs, if nothing else, have proven to be much faster than humans in the tasks they do master.
The main characteristic of our physiology is that of adaption, and we have fairly grounded research on progressive overload as a means to guide adaption to whichever direction we want. Should we then call humans RSI machines if they go work out or read a book, or study to get better at the activities they carry out?
Define an objective, choose an optimization algorithm, bring a loss down. If it keeps going down, do I call that RSI because the loss descended from the step that came before it?
Now, I see the nuance that the optimization algorithm changes over time, leading to interesting higher order effects the likes of which Hoefstadter describes in the GEB, but fundamentally, the idea is the same.
I also don't see what an intelligence explosion is supposed to mean. If we increase the models efficiency by 10x, and reduce costs by 10x, so we have 100x more software and math proofs being shipped by unit of time in say 2028, does that really solve all of our problems?
Say that gets distributed over the world. Do we get every human to be computer literate? Say that gets concentrated. Do we solve problems no one has the means to even care about?
I would consider an intelligence explosion when the models tell us whether our efforts are well or misdirected as looking back from the hindsight of 10 years ahead, and capable of explaining why in simple English.
Yeah, you can achieve something similar if you scale the compute infinitely and intelligently search the exponentially large search space.
Why would implementation matter? The thought experiment is that you have a black box that can do X. What is X that validates RSI/super-intelligence/etc? It's meant to clarify definitions and end the cycle of goal post moving.
(I would also say that computers are not good at math in the sense that LLMs are proving to be good at it: reliable symbolic manipulation is important for correctness but it's no use for actual progress in mathematics without some very good informal thinking/creativity as well)