upvote
Agreed. The fact is, Claude didn't get here on its own. It took someone who has been working with the RH since they were a student in college; some 40 odd years of experience.

It's unfortunate we're always focused on humans getting usurped by an "AI", instead of the reality that these models only succeed when we're there to show them what success looks like.

What if "superintelligence" is really a hybrid result of humans using and guiding these large pattern interpolators?

reply
What if "anthrocentricity" is really humans being unwilling to acknowledge that they are made of the same vibrating condensed energy as everything else, such that other matter can just as easily be moved into an apparently self-determining pattern of behavior as we are.

What if we're not the specialist snowflake?

reply
I’m on board with this, and have been since I first had a hallucinogen.

But the human brain for now still has 100x the parameter count as the largest models. And they don’t have any circuitry for all the “keep yourself alive” work our brains are doing all the time, which may be the majority.

reply
how profound of you
reply
> It's unfortunate we're always focused on humans getting usurped by an "AI", instead of the reality that these models only succeed when we're there to show them what success looks like.

As with chess, the reign of the human-machine "centaur" as the greatest mathematician in the world will likely be brief. I'd guess within 5 years the humans won't be adding anything. I hope I'm wrong.

reply
This is incorrect because you can always record the human prompts guiding the model and train it on them. This is how alot of progress in software was done and the success there shows that this generalizes beyond the specific prompts. You could probably prompt Fable 7 "Solve <famous hard math problem>" and it could do so without an expert mathematician guiding it.
reply
What if "superintelligence" is really a hybrid result of humans using and guiding these large pattern interpolators?

Obviously, humans with AIs are clearly more capable at a number of things than humans without but I don't think you can get out of the quandary of superintelligence that way.

Humans in the loop will always offer the opportunity of training and putting humans out of the loop.

And LLMs, if nothing else, have proven to be much faster than humans in the tasks they do master.

reply
Can you point to a task which you think would be an indicator of RSI or an intelligence explosion?
reply
I'm not sure I really buy into the way these concepts are currently defined. What progress to date was not RSI and did not build on top of earlier progress?

The main characteristic of our physiology is that of adaption, and we have fairly grounded research on progressive overload as a means to guide adaption to whichever direction we want. Should we then call humans RSI machines if they go work out or read a book, or study to get better at the activities they carry out?

Define an objective, choose an optimization algorithm, bring a loss down. If it keeps going down, do I call that RSI because the loss descended from the step that came before it?

Now, I see the nuance that the optimization algorithm changes over time, leading to interesting higher order effects the likes of which Hoefstadter describes in the GEB, but fundamentally, the idea is the same.

I also don't see what an intelligence explosion is supposed to mean. If we increase the models efficiency by 10x, and reduce costs by 10x, so we have 100x more software and math proofs being shipped by unit of time in say 2028, does that really solve all of our problems?

Say that gets distributed over the world. Do we get every human to be computer literate? Say that gets concentrated. Do we solve problems no one has the means to even care about?

I would consider an intelligence explosion when the models tell us whether our efforts are well or misdirected as looking back from the hindsight of 10 years ahead, and capable of explaining why in simple English.

reply
If you ask a layman, the answer would probably be a model that zero-shots any problem in any reasonable domain. E.g. zero shot Riemann hypothesis. No tricks like swarms of agents, no multiple agents correcting each other. No high, xhigh "thinking" levels. Simply provide an input, feed the input through the layers, answer out.

Yeah, you can achieve something similar if you scale the compute infinitely and intelligently search the exponentially large search space.

reply
> No tricks like swarms of agents, no multiple agents correcting each other. No high, xhigh "thinking" levels.

Why would implementation matter? The thought experiment is that you have a black box that can do X. What is X that validates RSI/super-intelligence/etc? It's meant to clarify definitions and end the cycle of goal post moving.

reply
It is impressive because of how large the search space is not entirely because of the level of abstract reasoning required to solve it. The reasoning trace produced that lead to this result could be a part of training that teaches the next version of Claude how to navigate incredibly hard problems making it more efficient.
reply
People suggested that language models are incapable of intelligence because they don't do symbolic manipulation. I think it's fair to call a vector with ten thousand numbers some kind of symbol, but then what's the state in your brain?
reply
Yeah, I think it's not so much "math is symbolic manipulation so computers and by extension LLMs are going to be good at it". LLMs are not by construction so good at it because ML models are far more statistical approximators than formal logic systems. I think the main thing that math has going for it is sufficient formalisation through tools like Lean that it's very verifiable, and the latests hotness in improving models is RLVR which seems to really work very well when the verification is strong and relevant to the task (probably also why they're getting very good at writing code and especially finding exploits).

(I would also say that computers are not good at math in the sense that LLMs are proving to be good at it: reliable symbolic manipulation is important for correctness but it's no use for actual progress in mathematics without some very good informal thinking/creativity as well)

reply
The reliable symbolic manipulation done here is in the form of Lean proofs, an external tool used by the LLM. One has to wonder how far an LLM alone would have gotten if formal proof systems didn’t exist they can verify their reasoning with.
reply