upvote
Yeah, a lot of the RSI claims seem dependent on us having very specific knowledge about how much intelligence is required to build superintelligence that I haven't seen anyone claim to have. Maybe you have to be, I don't know, 100x smarter than humans before the recursive part of RSI becomes tractable. Or maybe there are fundamental limits you can do with the hardware that exists that we don't know about, which means building superintelligence will take more time. I'm not saying the 2027-2030 people are wrong, I just think it depends on a bunch of things we can't credibly claim to know at this point.
reply
I'm not a fan of Dwarkesh, but even he laid out that machine learning research progress is very much verifiable. At least in the RSI sense, that our current approach can be drastically optimized.
reply
I won't trust that source at all, it's all about hype and trust me bro considerations
reply
The models are improving at certain fields, and one of those fields is creating new models, which are verified by applying them to fields where you can prove that they're better.

I agree they don't transfer to all domains, but we do not know how far the theoretical will overlap with practical science. It could be "not all that much" and we hit roadblocks quite soon, or it could be that as the models become better and better at reasoning and research within their theoretical domains, it turns out to translate really well to the real world.

We don't know, but my bet is that we will continue to see rapid progress and impacts on the real world, until we reach a point where sceptics will have to throw in the towel.

reply
deleted
reply
LLMs don't produce Lean proofs directly, they produce the proofs in natural language, and only translate them afterward. The fact that we have Lean has nothing to do with LLMs being good at theorem proving.
reply
(I think we mostly agree.) From a previos post about Navier-Stokes, the English version may have small holes and the AI make a few changes and fixups when translating to Lean.
reply
This is simply not true that they only invoke lean at the end. They invoke it multiple times in between to reduce the search space like GP said. See how e.g the prove2me harness (which anthropic used for FLT) works for more details.
reply
What kind of a thing do you think AI won't be able to do in 2030? A specific thing please.
reply
Love.

Throw a football at a wide receiver running in chaotic patterns while avoiding oncoming attackers.

Use chopsticks to pick up a single grain of rice.

Olympic level gymnastics via neural-control of muscle fibers in an extremely precise signaling orchestration.

You may say the above are robotics and not “the model can scale to this” domain but I’d counter an animal can do amazing things exactly because of its generalizable and plastic neural architecture and that is what’s in scope.

reply
Have you seen the progress in robotics lately? They aren't yet at a level you described, but quite close - so I'd say it would be possible in 3 years even without intelligence explosion. (E.g. if all frontier AI research stops we'll still have improvements in robotics.)

There are already Unitree robots which are much closer to Olympic level than an ordinary dude: https://www.youtube.com/watch?v=Y3zb4o47yqA You might argue but it's kinda at a level you described - I think if Unitree wanted to make a football-thrower they could just do it now.

reply
Shoot I can’t do those things either!
reply
I’ll grant you love, and I’ll also grant you 2030 on these, especially the last one. I suspect the football and chopsticks ones are very achievable by 2030 if anyone cared enough to focus on them.
reply
Fully autonomously run a business that generates $10+ million in ARR.
reply
What about one that generates $1+ million ARR?
reply
I find that unlikely as well. I should note though, that I incorrectly wrote "running a business", when I meant to say "scaling to $10+ million in ARR". I don't think there's necessarily that much difference in running a business of these two sizes, but scaling to $10m ARR is much harder than to $1m ARR.
reply
Sure.

If $1MM is also unlikely, how about $100k? Do you think that's unlikely as well?

The point is: what is the smallest amount that, if surpassed, would surprise you?

reply
It would surprise me if a fully autonomous AI company could even stay afloat over multiple months. Agents are getting (a lot) better at task completion over longer time horizons, but still occasionally tend to make really bad mistakes. What are the time horizons right now? Something like 40 hours with a 50% success rate? This is on benchmark tasks! The real world is completely different, I mean companies are scaling back efforts to even let agents handle things such as customer service. A fully autonomous business without any human oversight is on a completely different level to that. 2030 is not that far away, and I just don't buy that these models are exponentially improving in anything but a small subset of domains.
reply
Either AI won't be able to do this: - Make a version of itself 100x more efficient that runs on commodity hardware and undercuts athropic.

Or, alternatively, it will be able to do that. Either way there's no world where one company bottlenecks and holds the keys to AGI, and frankly that's good.

reply
- Keep electricity costs down

- Stop lying

- Design personal quantum computers

- Save fanatics from the delusion that AI will solve their life

reply
I mean something a human can do, but AI can't.
reply
Lay brick? Install pipe? Wire cable? Plaster wall?
reply
Here's a brick laying robot from 2017...

https://www.youtube.com/watch?v=6s17IAj-XpU

so yea....

reply
Try telling it to stop or move to a different wall in natural language. I mean, you might as well call a train AI with definitions this generous.
reply
Show me the car in which the robot drove itself there
reply
Self-reproducing physical agents (robots with onboard AI and ability to change their surroundings/build physical things) with human or higher intelligence which don't need a global supply chain of microchips, electro motors, etc. but could live off the land. That's basically the human concept, right? Totally untenable for AI/robotics of the current generation. Would require something like 3D-printing new chips but also 3D-printing new printers. Probably never happening. AI is fundamentally different, reliant on data centers, which are immovable and more like organs than organisms. Robots only come out of factories, not out of other robots or eggs layed by robots. So as Jack of all trades in the game of survival on Earth, humans win for the foreseeable future. Also, AI currently can't exist without humans, whereas humans have thrived without AI. Whether a silicon based ecosystem can ever survive on its own is an interesting question in and of itself. Because no matter how smart data centers become, they degrade fast, and then what? A data center needs maintenance. But if robots maintain the data center, who will maintain the robots? Does this equation ever solve? Humans have bacteria, a complex microcosm of life maintaining them. I don't think such a dependency-hierarchy of "life forms" is feasible for AI/robots, and humans will always have to be in the loop. That's one answer to your question, and not just for the next 3 years.

What stands axiomatically are human-to-human connections that we're wired to crave. Not to mention human reproduction.

Other points: - Editing a live action movie that will get 8.0 or more on imdb (AI sucks at reading emotions and won't be able to tell apart good takes from bad ones) - Reliably making good jokes (many humans can do it, AI weirdly sucks at it) - Accountability (contradictory because of its undead nature) - Reliably not lying/cheating (with no accountability, incentives will always be misaligned)

reply