upvote
My thoughts remain the same since I've been hearing "months or a year" for the past 5 years. Maybe this time truly is different, but unless acceptance of super-intelligence is required to take some action (most people I've seen who hold this view also don't think it should be seriously regulated so I'm not sure what that would be), why not just wait until it happens rather than making more predictions? The only thing I've ever seen come out of these predictions is willing me to "stop resisting", but there's no reason to do that until its here.

If you do want to see something done about it and this isn't just another prediction or a vague warning, more power to you, but otherwise, I'll wait and see

reply
> My thoughts remain the same since I've been hearing "months or a year" for the past 5 years.

No one said that. If you look at e.g. the Metaculus predictions, you can see AGI has been hovering around 2027-2030 for 5 years.

https://www.metaculus.com/questions/3479/date-weakly-general...

https://www.metaculus.com/questions/5121/date-of-artificial-...

reply
There are certainly a lot of other people making all kinds of predictions. Some of them will eventually be correct but which one?

Additionally even with all the breakthroughs I am still doubting RSI. Up till now all the advancements heavily rely on RL over Lean. But how do model formalize it's goal as in RSI and RL over it?

reply
No one said [AGI in months or years, five years ago] that. If you look at e.g. the Metaculus predictions, you can see AGI has been hovering around 2027-2030 for 5 years.

That statement just contracts itself - if the average prediction has been 2027-30, someone, a significant percentage of someones, said earlier since many have said later.

reply
Will OPs prediction be counted on Metaculus? If not, it's clearly not an accurate source for "No one said that".
reply
2027-2030 is months to years away.
reply
It is, but if OPs prediction isn't counted on there, it's not counting every prediction made by any individual, probably just the major ones that get covered, and so therefore "No one said that" would become "No one significant said that", which I'd imagine it probably true but, unless I'm mistaken, OPs prediction would sit in the same pool as all those other ones, they hold equal validity.
reply
If you zoom out, you will see that the median for that estimate a few years ago was two thousand fifty. While certainly some people might have thought it happens the next day, the average person thought it's many years away.
reply
You're mis-remembering things. Nobody in 2021 claimed that superintelligence is likely to arrive in 2022. That just did not happen.

People were talking about superintelligence in 2021, but "by the end of the decade", not "next year".

reply
Not splitting hairs at all, but I recall a lot of mid-2030s estimates around that time period. The horizon has drawn in a lot in a few years.
reply
Yeah, I remember that last decade the prediction was the 2040s, but then LLMs happened and it became 2030s.
reply
Mid-2030s estimates aren't uncommon even this year.

And then there's this gem: https://www.dwarkesh.com/p/ege-tamay "AGI is still 30 years away — Ege Erdil & Tamay Besiroglu"

reply
I was _hoping that maybe_ 2050 and other software engineers were considering me a crazy. GPT-2 was out by then.
reply
As I said to someone else, this wasn't by major figures, this was in conversation with ordinary people. As I said to them, if you want to dismiss this you'd have to dismiss OPs prediction too, and probably cease making your own unless you can get a news-outlet to cover it
reply
Why do you believe that "OP" is making a prediction on his own?

There are some well-known experts saying this, like Paul Christiano, Daniel Kokotajlo, etc. Both are ex-OpenAI and are advocating pause/slowdown.

reply
That kind of feels like moving the goalposts. Nobody becomes Nobody of significance becomes Nobody of significance without others of significance making similar claims, all 3 of these falling under the banner of "Nobody".

But at this point its a matter of semantics, and wasn't really the point I was trying to make. And as I said to someone else, if the reason for making these predictions is to encourage a slowdown I'm much more sympathetic to them (since there a real reason to act of them)

reply
> My thoughts remain the same since I've been hearing "months or a year" for the past 5 years.

If you zoom into any curve it tends to look linear. I think that is what you keep doing. But if you look at the world as how it was 5 years ago compared to now, there is no denying there is a massive change.

reply
> But if you look at the world as how it was 5 years ago compared to now, there is no denying there is a massive change.

Is there? Yes, LLMs are integrated into a bunch of software, but practically speaking life is pretty much the same as it was five years ago.

reply
> LLMs are integrated into a bunch of software

I should have restricted it to the fields that have been impacted and have seen significant adoption: art for commercial purposes, writing, software development and mathematics primarily, and those adjacent to those fields. It is true that most other fields have not really felt that many effects.

reply
AI videos everywhere, software developers no longer writing any code and it being much harder to find a job, and people largely no longer using search and instead switched to chatbots is “same as it was in covid”?
reply
I'm not saying LLMs aren't an absolutely impressive technology, especially for software related tasks, but in practical terms, life remains the same as it was five years ago.
reply
People are really really bad at seeing incremental change as it occurs around them.

It's like watching your kids grow. Your there everyday and you typically miss a lot, but then your cousin shows up for a holiday after not seeing your kid for 4 months and they are like "holy shit they've grown".

If you made a SOTA LLM enabled computer show up 10 years ago people would be fucking panicked at the crap we're messing with a decade later. But because we mess with it a day at a time we're like, eh, whatever.

reply
Tell that to the SWEs getting laid off left and right all over Silicon Valley!
reply
Once again though, is me accepting this a pretext for something? If not, why not just wait and see? Unless there's something that needs to be done before it's too late, it seems an odd choice to push for belief when there's no reason we can't just wait a while and have actual proof
reply
There are many things we can do and many people are talking about it.

One such coalescion is: https://pauseai.info/

reply
Absolutely, the thing is that much of the time when I hear these predictions it's not to encourage participation in the movement like this. Many people who I speak to about it are even against such movements, which is what leads me to question why I need to accept their predictions.

If someone were to come to me with something like this as their reasoning, I wouldn't feel the same way.

reply
Heard. Then:

Exponential progress has been the repeated truth, mirroring predictions.

The result of exponential progress into 2027 is a complete de-mapping of what we know to be human, whatever that may look like exactly at this point is hard to say.

We need your help, and everyone's help, to put a pause on this such that we are able to ground and find a healthy way forward together for as many human beings as possible.

Please join us.

reply
Gotta say I really appreciate this page, I'm happy to donate but providing tangible actions is a really solid step

https://pauseai.info/action

I'm still not sure I believe it, but in my view its a bit like global warming, the worst(or best) case scenario of it being wasted time and money is far outweighed by the inverse, so it only seems pragmatic to help

reply
I don't know about others in the thread think there definitely needs to be something done before it's too late.

I understand that many people don't think it will happen, but I can't really understand the position that you think there is a real chance superintelligence happens, but aren't concerned about it and are happy to wait passively to find out.

If you care about humanity at all it should be concerning because superint (if real) could be heavily destructive to our society.

reply
I agree and I'm happy to hear out this kind of thing, my "I'll wait and see" attitude comes when the expected response is that I accept that the AI has "won". Thats pretty vague because the expected response, if there even is one, is pretty vague, but it generally amounts to something like that.
reply
[dead]
reply
> we can't just wait a while and have actual proof

I am not pushing you to do anything. I am not asking you to "believe" or anything. But we are, in my and many other's estimations, in the midst of what was called the "singularity" in AI communities, although it isn't a true mathematical singularity, it is just a strong acceleration.

reply
> I am not pushing you to do anything. I am not asking you to "believe" or anything

Fantastic, I'll remain at my previous position then, I'll believe it when I see it.

(As a very off-topic sidenote, the term singularity never made much sense to me. I believe it was originally used to describe the sci-fi concept of a human gestalt consciousness, but has shifted to being about super-intelligence which doesn't seem to have much to do with "singular" at all. But, language changes I suppose)

reply
> I believe it was originally used to describe the sci-fi concept of a human gestalt consciousness,

Huh? The term itself was introduced by Vernor Vinge to refer to the accelerated rate of change as a result of the introduction of super-intelligence. He said it would make things harder to predict in the future. It is part of this article:

https://en.wikipedia.org/wiki/Technological_singularity

It was never about "consciousness".

reply
Huh, looks like you're right, though I would argue with "it was never about consciousness", but yeh it certainly wasn't originally. I think I got my meaning from a sci-fi short story were a unified collective called themselves "The Singularity", and I guess that was my first exposure to the term as a kid.
reply
The biggest risk of the singularity is incoherence, and it's actually looking like what is happening now.

Like AI video/audio of high quality makes you have to question and doubt any media you see.

AI generated text has become the vast majority of material that you see showing up on the internet.

Things are moving from centralized pieces of software used by lots of people to vibe coded apps.

Reality tunnels and almost fully close and you can live in an existence that seems 'normal' but is a completely simulated experience with no interaction with other humans.

reply
> If not, why not just wait and see?

Curious what you need to see. Some physical impact to your way of life? Are you not able to extrapolate the obvious?

reply
So you think it's already here?
reply
5 years ago, what concretely did you expect 5 years later? One comment from then https://news.ycombinator.com/item?id=29020401 falling way short of today, treated then as wild near-sci-fi.
reply
Pretty much, a pragmatic attitude at its finest
reply
Yet AI has changed the world incredibly fast.
reply
Why try to predict the most important single event to happen in human history?

Surely you recognize how absurd it is you’ve gone to such lengths to make your priors palatable. Please Look Up.

reply
> Why try to predict the most important single event to happen in human history?

You're free to try to do so, but why are you pushing it so heavily? To what end?

> Surely you recognize how absurd it is you’ve gone to such lengths to make your priors palatable

And there is the answer to my above question. I like to think it's not most people, even of those who are staunchly pro-AI, but to some previous doubts are at best a great shame (assuming prior is short for prior doubts) or at worst a strange kind of heresy (assuming prior isn't short for anything). Either one seems anti-intellectual though, and encourages just the kind of endless hype we've seen so many times before.

reply
Who said "recursive self-improvement / intelligence explosion / superintelligence" would happen in "months or a year" before 2026?

I can't think of anyone except maybe Elon Musk.

reply
It was in conversations just like this, so people you wouldn't know and I didn't know outside of that conversation. You might be tempted to dismiss those, but then you'd also have to dismiss OPs for the same reason.
reply
Okay can you provide links to these HN comments?

There might be, but I'm curious about the volume (I suspect very low).

reply
I think it was on Reddit more-so if memory serves. Maybe it was on HN too, but obviously it was years ago now, I don't have a strong memory of the specifics. I'm not going to go digging through years of comments though, if you don't want to believe me thats fine, it seems a bit absurd to think absolutely nobody believed that but it wasn't really my point to begin with
reply
> it seems a bit absurd to think absolutely nobody believed that but it wasn't really my point to begin with

Okay. If your point isn't that people believed superintelligence was months to a year away years ago, what is your point?

reply
All due respect, I wrote a paragraph and that was the first sentence.

My point was that I'd rather wait for proof than accept a prediction.

reply
great demonstration of the mindset OP was talking about
reply
I'm always shocked by the number of smart, technical people who seem to think that AI is going to crash and burn and go away. In 2025 AI got a gold at the IMO for the first time, a competition for high school kids. Now it's likely solved the biggest problem in math. There's no reason to expect progress to stop here, especially on domains highly amenable to RL.
reply
> I'm always shocked by the number of smart, technical people who seem to think that AI is going to crash and burn and go away.

I suspect that these "smart, technical people" you speak of have more nuanced views on AI than you realize. Myself, for example, a self-styled smartypants:

Like the dot-com bubble, I suspect that certain parts of the current AI landscape are unsustainable, and I suspect/hope that those specific features of AI will crash and burn, while other useful bits remain.

AI usage like we see in this post here--AI that assists with discoveries in math, medicine, genetics, astronomy--I think they're great, and I hope they will continue to advance human knowledge for the foreseeable future. They assist experts and professionals rather than replacing them.

In contrast, generative AI ("creative" AI writing, illustrating, music, etc) is a pox on civilization. It is industrialized low-effort plagiarism that harms artists' livelihoods, discourages new art, makes everything bland, and pollutes the world in multiple senses of the word. I hope it dies from a combination of negative sentiments, and too much real-world cost once token subsidies go away.

Then there are the AIs that do things like voice reproduction. They have some arguable positive uses (e.g., assistive technologies), but many negative ones (e.g., phone scams, replacing human narrators with copies of their own voices). I think we're better off without these apart from niche accessibility uses. I think it should be legal peril to reproduce a voice without permission.

Apart from helping with some cinema CGI, deepfakes and deepnudes seem like a net negative for civilization.

Knowledge-based AI like Google's AI search summaries might just have built-in expirations. I think people will burn out on being vigilant about hallucinations. And when the bulk of their training data is made by AIs, it'll be hard to prevent model collapse. The cost of human intervention might make it too expensive for the wide use we see today.

Coding AIs are a grayer area. I think it's highly problematic that they are basically open source plagiarism machines, but I begrudgingly believe that society will decide that the upsides outweigh the downsides in the long term. But I choose not to use AI in my own coding projects, and respect others who do the same.

reply
My instincts are very similar on all points.

To push back, or share the self-pushbacks I grapple with...

> In contrast, generative AI ("creative" AI writing, illustrating, music, etc) is a pox on civilization. It is industrialized low-effort plagiarism that harms artists' livelihoods, discourages new art, makes everything bland, and pollutes the world in multiple senses of the word. I hope it dies from a combination of negative sentiments, and too much real-world cost once token subsidies go away.

While this summarizes my feelings on the current versions (which, say, are roughly at or below the math olympiad stage in writing, video, etc), what happens when they actually become good? Meaning, an AI show that was actually as good as The Wire? To the point you couldn't tell the difference without being told?

I really want to say that even then it won't be the same, because the human provenance is essential to the art, but if I truly couldn't tell the difference...

reply
> While this summarizes my feelings on the current versions (which, say, are roughly at or below the math olympiad stage in writing, video, etc), what happens when they actually become good? Meaning, an AI show that was actually as good as The Wire? To the point you couldn't tell the difference without being told?

But here we're dipping into science fiction again. This is not currently the case, and we seem a long way from it.

reply
> and we seem a long way from it.

We're dipping into prediction. But we have the (seemingly relevant, at least) data point that we're in the same situation w.r.t. writing/art as we were w.r.t. to math a couple years ago...

reply
To an extent, sure, but:

- There was progress on maths that led to this point, whereas it's been stalled on art for the same reasons since the beginning.

- Art doesn't lend itself as well to the 'add more agents/complexity' strategy; a 250,000 word limerick is a terrible limerick.

reply
> what happens when they actually become good? Meaning, an AI show that was actually as good as The Wire? To the point you couldn't tell the difference without being told?

Perhaps this comment will make me look silly one day, but my guess is that we'll eventually get there, and a handful of AI-generated productions will truly shine. But "as good as" is an important phrase. Because current-generation AI is trained on existing art, it won't be pushing many envelopes, it'll mostly be remixing what we have, and that will grow stale.

After a few breakout hits with broad shared appeal, I think the novelty will wear off, and AI produced TV and cinema will be too resource-intensive to remain as a profitable going enterprise. There will likely be a short window of hyper-personalized media that will seem very exciting, until we grow weary of the isolation of it. For many people, half the fun of watching a thing is discussing it with other humans, and today's streaming media landscape already leaves many people unmoored from that shared experience.

There is also the question of intentionality. When AI creates that show as good as The Wire, is that because AI has something to say about the human condition? If not, what's the point? If so, should we be paying attention to what AI wants us to pay attention to?

reply
So what if AI slop wasn't slop? If you care about the details and the values behind your work enough to make it actually good, you won't use gen AI or use it invisibly because it concedes control of the details.

If you believe that using the authenticity of your own experience in your art will become vanish from how people value art then yeah we're headed for a future where AI rules. I really hope that that isn't the case.

reply
I like your framework of separating use cases, mostly to evaluate how it will change in those specific areas, but I think you can evaluate capabilities fairly simply: I like to separate AI that depends on giant data centers and huge power draw from the kind that doesn't (not always local)—basically expensive vs cheap.

The expensive stuff isn't sustainable. Not yet. It probably will be, who knows when. I suspect it will be eventually because there's good money in it.

Here are the claims I find highly suspect based on the above:

- Anyone making claims that require expensive AI and attaching it to a short-term timeframe

- Anyone saying highly capable AI is impossible in the long-term

- Anyone saying these specific companies will be the winners—there is absolutely no moat for these models

I personally think anyone claiming something like super-intelligence or something approaching what we might consider sentience even in the next 5 years is suspect as well. I would not take the bet myself, but I would also not bet against it. The main reason is power: there's literally only so much power we can generate, so much societal disruption due to creating the infrastructure we can absorb (not to mention the societal disruption due to the tech itself), so its capability to self-improve is going to be highly constrained.

So far, we see increases in capabilities also increase power usage, I believe linearly. So, we're not seeing increases in efficiency that will make this easier to absorb. There's a ceiling, it won't go up quickly, so I think actual sentient-like AI capabilities are quite farther out than promoters think. Not impossible! I think it's more like a decade+ of maturation of the software and hardware stack—which isn't really that long in the grand scheme of things but longer than most people are betting.

reply
They may have more nuanced views for sure, but nuance is famously lacking from most online discussions unfortunately. I would love to see more of it surface!
reply
> Like the dot-com bubble, I suspect that certain parts of the current AI landscape are unsustainable

It's possible to believe that _almost all_ AI companies will go bankrupt while still believing it is a massively important world changing technology.

reply
My experience on teams is there are the high agency AI adopters. Management likes this group. Then there’s what management would characterize as those not adapting (or adapting enough). This is the group getting laid off / not advancing in the environment. And this group is particularly bitter about AI disrupting an otherwise stable career, with wishful thinking they’ll be able to go back.
reply
I've seen the first group vibe up garbage and dump maintenance on the second group. It may be unfair to blame AI for that when management is ultimately responsible, but that's what a lot of people are doing. In their defense, management is being mislead by AI hype men who are saying AI is so intelligent it shouldn't be making dumb mistakes.
reply
Agreed, its more complicated than GP describes. There is clearly lots of value, but there are also lots of poorly led organizations creating a mountain of problems for themselves. We are all still figuring out what AI really means just for software development, which is the most useful economic activity it can do by an enormous margin.

Human minds are still needed to understand many things related to developing correct software, especially in business domains. There is still no AI running a profitable business, because that is not a verifiable task that can be scaled in RL like math or coding. So extrapolating from a math proof that no one can understand to a economically useful agents taking over most jobs is quite suspicious.

What is changing now is RSI truly beginning to happen. Much of ML is a verifiable task that can be scaled, and a lot transfers from general coding as well. Leading researchers are predicting their own obsolescence as early as next year, and that is something I take seriously, though I cannot judge the likelihood myself.

As improvement accelerates, we may see generalization happen at a scale that means we don't need RLVR for every profession. We've been surprised by it before.

reply
> because that is not a verifiable task that can be scaled in RL like math or coding

Is it not, or are the feedback loops just longer? I could imagine being able to build a pretty good simulator of business operation for various kinds of businesses. (Some decent ones already exist as games.) From there, if you can build a good enough simulation, I can't see why you couldn't RL it.

Of course it's easier said than done to generalize it to the real world, where the possibility space is large. But I don't necessarily see an impediment to getting there eventually.

reply
The feedback loops are quarters and years, so no you cannot scale it like you can scale math or coding. This is the reason we are seeing the most progress in those domains. Even ML is harder to scale because test runs are expensive and consume significant calendar time, so research "taste" is still important.
reply
A full simulation would be, but I suspect even a partial simulation of evolving a software project though multiple "checkpoints" over a quarter would improve model performance
reply
"Running a business" includes lots of real-world interactions that would be hard to model. An AI trained on an accurate simulation would probably get torn to shreds by scammers in the first six months. The models that exist today have enormous blind spots.
reply
Honestly, the second group kind of brought this upon themselves. You cannot survive in tech without having an open mind and moving fast.
reply
[dead]
reply
I actually asked a guy who got an IMO gold what he thought about it, and he's happy to concede that AI works. He used to be my go-to guy for those kinds of problems when I came across them. Now you can actually get it to solve those things on your standard plan.
reply
Level of intelligence doesn’t matter if it is not applied to truth searching and instead to reasoning backwards from personal wishes.
reply
Isn't this post an example of the former?
reply
> I'm always shocked by the number of smart, technical people who seem to think that AI is going to crash and burn and go away

That’s what I am responding to.

reply
deleted
reply
Define truth
reply
42
reply
Parts are already disappearing if not gone already. If not by lack of profit, it will be by liability, overuse and moderation. Filters are being put up, bans in place, services restricted.

Just the other day a model helped me root an old android tablet. Will that still work in 5 years? I don't really have great confidence.

reply
Yes, as long as it’s legal for you to buy a GPU.
reply
>I'm always shocked by the number of smart, technical people who seem to think that AI is going to crash and burn and go away.

The tech industry cultivated that reputation though. People were just acting rationally by being more skeptical of even larger claims than they were making before.

reply
Because they're _desperate_ to keep their social standing. That's literally all there is to it. The same occurred during Enlightenment and the industrial revolution. A small upper echelon of society (the church or aristocrats respectively) were deemed worthy enough to possess knowledge and thus to belong to an upper strata. The same story can be seen pretty much in every civilization btw (Greeks/India/China/Japan etc). When the masses started gaining access to this knowledge there was _massive_ pushback, even armed revolutions in certain cases.

What's happening today is that anyone and everyone can have access to superintelligence. Anyone with access to an LLM effectively is, simultaneously, a world-class diagnostician, engineer, architect, musician, mathematician... you get the point. This will necessarily lead to a post-scarcity society, since everyone knows everything and will be capable of producing anything.

reply
> This will necessarily lead to a post-scarcity society

I wish I shared this optimism, unfortunately that upper echelon of society is the one which currently controls the superintelligence. Everyone may have access to it, but they own it.

reply
I would share your concern if there wasn't such a thriving ecosystem of open weight models. And they're usually about a 6 months to a year behind.

Still, as of now, while I use AI for everything I do (mostly Claude but also ChatGPT), I don't use any open weight models (probably more of a mental block at this point), but I do see them as a necessary counterweight to this potential power imbalance.

reply
Where are you going to run these open models? Natural resources are controlled by this upper echelon too.
reply
Most households own a car which even if you go used has a price tag of around $20k if you want something average. It wouldn't be much of a stretch to see a future where every household owns a $20k AI computer. Or two. If you are fancy you own a $100k AI computer.

However, since there is an ongoing war on general computing it wouldn't surprise me if instead people get locked into subscriptions.

reply
Shit people be telling ChatGPT isn't protected by law. You get into any trouble, your chat history gonna be gone over with a fine tooth comb. Some of the masses gonna realize that and gonna want a sever rack at home instead.
reply
I bought a 2000 Honda Accord V6 from a friend for $3k a long time ago. I drove it for several years, adding probably another 90,000mi to it. It didn't need much maintenance during that. It's still kicking from what I understand. An incredible car, I'm a little sad I parted with it.

I still think of the economic benefit I got from that car. It drove me to university classes. It drove me to work. It took me to see a total solar eclipse. It brought me to see distant family regularly. I went on a lot of road trips on it. For roughly $3k of capital costs.

I look at most expensive things and size them up as "how many Accords is this?"

Almost 7 Accords is a lot.

reply
deleted
reply
For now. Open weight models are effectively only lagging ~6 months behind the frontier.
reply
> This will necessarily lead to a post-scarcity society, since everyone knows everything and will be capable of producing anything.

A post-scarcity society requires MATERIAL abundance in the hands of everyone.

All the material is controlled by ~100 mentally ill people who would rather kill us all than share.

reply
I don't get why academics get the shaft every time. These labs are worth trillions of dollars while your average professor has to deal with 18 year old shitheads and gets paid 50k a year. Like what?
reply
Anti-intellectualism. The median AI booster resents that there is a class of people that had to work to become proficient in something, and wants effort to become irrelevant so their lack of investment in it appears to no longer be a failing.
reply
AI boosters love to frame the trillion dollar corporations as the real victims. War is peace, freedom is slavery, scraping all known data and refusing the share the fruits ("distillation attacks") is democratization, people who are losing their jobs to megacorporations with lots of compute power are the real sheltered elites who are mad because they're hypocrites.
reply
I only hear this part of non-reasoning from people who have never managed to accomplish anything and simply want to blame AI for their personal failures.
reply
That professor is free and highly encouraged to navigate their web browser to Anthropic’s careers page.
reply
I don't get what you are trying to say. We will be all working for Anthropic?
reply
How would an LLM make me a world-class violin player? If scarcity is from needing real "stuff" (incl. time), then knowledge alone might not change that scarcity.
reply
Same way as a human would? By watching you (upload video) and listening to you (upload audio) and talking to you about your technique (text to speech). It's obviously no substitute for the real thing, but for $20/month, it's a lot more accessible to more people than a human instructor would be.
reply
Can anyone at any point in their life be turned into a world class violinist?
reply
Malcolm Gladwell popularized the notion that 10,000 hours of focused practice gets anyone to an expert level of something. I've not got the drive to be a violinist, but in my world view, yes. Excluding the infirm, if we focused all of society into turning a randomly chosen person into a world class violinist, including mind control such that they want real bad, then yes. Violin playing isn't like the castrati. You need arms, and practice. The vocals of the castrati are something that could not be replicated in anyone.

The world of violin playing isn't that large though, so the question is merely getting someone to be really really really good but not quite world class close enough, or not really?

reply
Using an llm would focus all of society on a person?
reply
Why do these goalposts seem so shifty?
reply
How is that shifting from the initial claim I replied to? Or how an llm would make me a world class violin player? It's not about becoming a reasonably good player or using some huge resources beyond an llm.
reply
We don’t have access to knowledge, we have access to information. Knowledge requires processing information and that just doesn’t happen any more.

A post-scarcity society is not possible without first wiping out society. If everyone has everything then everyone has nothing. And of course, natural resources are actually limited, and those power LLMs.

reply
I'm sick and tired of seeing people go "AI skeptics are just afraid of not being important". That's complete bullshit. We are skeptical because we can see, with our eyes, that the claims of AI boosters continue to be false time and time again. The simple empirical fact is that LLMs can't actually do a good job at the things people claim they are good at, and the industry lies through its teeth about said capabilities. So we are skeptical.
reply
deleted
reply
Yep, argument by psychoanalysis is a convenient way to disregard any stated disagreements as really just rationalization by weak/scared/lazy/depressed/etc people.
reply
Wild thread to be making an “LLMs can’t do what’s claimed!” last stand…
reply
AI still can't do a lot of what's claimed. This thread is an example of it being able to do one admittedly-impressive thing, but it's still nowhere near living up to all the promises.
reply
> Anyone with access to an LLM effectively is, simultaneously, a world-class diagnostician, engineer, architect, musician, mathematician... you get the point. This will necessarily lead to a post-scarcity society, since everyone knows everything and will be capable of producing anything.

It’s hard to take you AI-hype men seriously when you are this ravingly delusional.

reply
Where are these smart and technical people who said AI was going to "go away"? I honestly haven't met a single one this whole time.
reply
> There's no reason to expect progress to stop here

Well, the amount of money and resources it took to get here is a reason. I don't think the resources exist for another order of magnitude increase, given the strain on our economy and society we're already under due to AI.

reply
The frontier models are getting cheaper as they are getting more powerful [0]. It's really not going to crash. I wish it would, for the sake of humanity.

[0] https://epoch.ai/publications/the-plunging-price-of-thought

reply
As does every technology, but websites are not a few million times faster than they were when they were invented, and your computer does not boot a few billion times faster (quite the opposite...).

Reality is extremely chaotic.

reply
I was thinking more about the cost in time, power, hardware, and data to train the models, not the price charged to use the models. I don't think it's going to crash, either—but that doesn't mean that progress will increase forever.
reply
We are well past the point where massive changes are guaranteed even if LLMs immediately and permanently hit a wall in capabilities.
reply
"If you are smart, why can't you predict the future?"
reply
It's not an ability to predict the future, but the reluctance to step back and look at the trendline rather than saying "But it still can't do X, Y or Z!".
reply
These are extremely jaw dropping results, but I feel even though it is easy to conflate them with RSI / intelligence explosion, another simpler explanation is that symbolic manipulation is something humans are notoriously bad at, hence we prizing individuals that can rise above and dedicate their lives to it, and call them intelligent.

For computers though, symbolic manipulation is something they are extremely good at, and something that clearly scales well will memory and compute. Therefore I would argue that calling a computer superintelligent would, in analogy to do what we do with humans, require computers to rise above things that are extremely difficult for them. It is of course hard to judge what would be hard for them not being them for the introspection, but it seems the ability to operate outside the realm of symbolic or numeric manipulation would be precisely that.

It stands to reason that if some proofs require chaining 250k pages together, no human would ever be able to put it together. That doesn't mean something that is capable of doing so is necessarily super-intelligent.

Still very impressive, and if true, a wonderful result of the symbiotic interplay between a human and a machine, each at what they are good at.

reply
Agreed. The fact is, Claude didn't get here on its own. It took someone who has been working with the RH since they were a student in college; some 40 odd years of experience.

It's unfortunate we're always focused on humans getting usurped by an "AI", instead of the reality that these models only succeed when we're there to show them what success looks like.

What if "superintelligence" is really a hybrid result of humans using and guiding these large pattern interpolators?

reply
What if "anthrocentricity" is really humans being unwilling to acknowledge that they are made of the same vibrating condensed energy as everything else, such that other matter can just as easily be moved into an apparently self-determining pattern of behavior as we are.

What if we're not the specialist snowflake?

reply
I’m on board with this, and have been since I first had a hallucinogen.

But the human brain for now still has 100x the parameter count as the largest models. And they don’t have any circuitry for all the “keep yourself alive” work our brains are doing all the time, which may be the majority.

reply
how profound of you
reply
> It's unfortunate we're always focused on humans getting usurped by an "AI", instead of the reality that these models only succeed when we're there to show them what success looks like.

As with chess, the reign of the human-machine "centaur" as the greatest mathematician in the world will likely be brief. I'd guess within 5 years the humans won't be adding anything. I hope I'm wrong.

reply
This is incorrect because you can always record the human prompts guiding the model and train it on them. This is how alot of progress in software was done and the success there shows that this generalizes beyond the specific prompts. You could probably prompt Fable 7 "Solve <famous hard math problem>" and it could do so without an expert mathematician guiding it.
reply
What if "superintelligence" is really a hybrid result of humans using and guiding these large pattern interpolators?

Obviously, humans with AIs are clearly more capable at a number of things than humans without but I don't think you can get out of the quandary of superintelligence that way.

Humans in the loop will always offer the opportunity of training and putting humans out of the loop.

And LLMs, if nothing else, have proven to be much faster than humans in the tasks they do master.

reply
Can you point to a task which you think would be an indicator of RSI or an intelligence explosion?
reply
I'm not sure I really buy into the way these concepts are currently defined. What progress to date was not RSI and did not build on top of earlier progress?

The main characteristic of our physiology is that of adaption, and we have fairly grounded research on progressive overload as a means to guide adaption to whichever direction we want. Should we then call humans RSI machines if they go work out or read a book, or study to get better at the activities they carry out?

Define an objective, choose an optimization algorithm, bring a loss down. If it keeps going down, do I call that RSI because the loss descended from the step that came before it?

Now, I see the nuance that the optimization algorithm changes over time, leading to interesting higher order effects the likes of which Hoefstadter describes in the GEB, but fundamentally, the idea is the same.

I also don't see what an intelligence explosion is supposed to mean. If we increase the models efficiency by 10x, and reduce costs by 10x, so we have 100x more software and math proofs being shipped by unit of time in say 2028, does that really solve all of our problems?

Say that gets distributed over the world. Do we get every human to be computer literate? Say that gets concentrated. Do we solve problems no one has the means to even care about?

I would consider an intelligence explosion when the models tell us whether our efforts are well or misdirected as looking back from the hindsight of 10 years ahead, and capable of explaining why in simple English.

reply
If you ask a layman, the answer would probably be a model that zero-shots any problem in any reasonable domain. E.g. zero shot Riemann hypothesis. No tricks like swarms of agents, no multiple agents correcting each other. No high, xhigh "thinking" levels. Simply provide an input, feed the input through the layers, answer out.

Yeah, you can achieve something similar if you scale the compute infinitely and intelligently search the exponentially large search space.

reply
> No tricks like swarms of agents, no multiple agents correcting each other. No high, xhigh "thinking" levels.

Why would implementation matter? The thought experiment is that you have a black box that can do X. What is X that validates RSI/super-intelligence/etc? It's meant to clarify definitions and end the cycle of goal post moving.

reply
It is impressive because of how large the search space is not entirely because of the level of abstract reasoning required to solve it. The reasoning trace produced that lead to this result could be a part of training that teaches the next version of Claude how to navigate incredibly hard problems making it more efficient.
reply
People suggested that language models are incapable of intelligence because they don't do symbolic manipulation. I think it's fair to call a vector with ten thousand numbers some kind of symbol, but then what's the state in your brain?
reply
Yeah, I think it's not so much "math is symbolic manipulation so computers and by extension LLMs are going to be good at it". LLMs are not by construction so good at it because ML models are far more statistical approximators than formal logic systems. I think the main thing that math has going for it is sufficient formalisation through tools like Lean that it's very verifiable, and the latests hotness in improving models is RLVR which seems to really work very well when the verification is strong and relevant to the task (probably also why they're getting very good at writing code and especially finding exploits).

(I would also say that computers are not good at math in the sense that LLMs are proving to be good at it: reliable symbolic manipulation is important for correctness but it's no use for actual progress in mathematics without some very good informal thinking/creativity as well)

reply
The reliable symbolic manipulation done here is in the form of Lean proofs, an external tool used by the LLM. One has to wonder how far an LLM alone would have gotten if formal proof systems didn’t exist they can verify their reasoning with.
reply
When Atari games or Go were solved around 10 years ago, nobody was saying that superintelligence was within months just because reinforcement learning worked on a domain that allowed learning by self-play. Now we have a domain that is solvable by reinforcement learning with verifiable rewards. It's very impressive that these models are capable of doing the things they do, but the techniques aren't transferable to the vast majority of other domains. ML research also isn't verifiable in the sense that math is. People who claim it's verifiable, because there's a target that can directly be optimized don't understand why RL works for math. It's that lean is able to massively shrink the search space. This simply cannot be done in other domains in the same way.
reply
Yeah, a lot of the RSI claims seem dependent on us having very specific knowledge about how much intelligence is required to build superintelligence that I haven't seen anyone claim to have. Maybe you have to be, I don't know, 100x smarter than humans before the recursive part of RSI becomes tractable. Or maybe there are fundamental limits you can do with the hardware that exists that we don't know about, which means building superintelligence will take more time. I'm not saying the 2027-2030 people are wrong, I just think it depends on a bunch of things we can't credibly claim to know at this point.
reply
I'm not a fan of Dwarkesh, but even he laid out that machine learning research progress is very much verifiable. At least in the RSI sense, that our current approach can be drastically optimized.
reply
I won't trust that source at all, it's all about hype and trust me bro considerations
reply
The models are improving at certain fields, and one of those fields is creating new models, which are verified by applying them to fields where you can prove that they're better.

I agree they don't transfer to all domains, but we do not know how far the theoretical will overlap with practical science. It could be "not all that much" and we hit roadblocks quite soon, or it could be that as the models become better and better at reasoning and research within their theoretical domains, it turns out to translate really well to the real world.

We don't know, but my bet is that we will continue to see rapid progress and impacts on the real world, until we reach a point where sceptics will have to throw in the towel.

reply
deleted
reply
LLMs don't produce Lean proofs directly, they produce the proofs in natural language, and only translate them afterward. The fact that we have Lean has nothing to do with LLMs being good at theorem proving.
reply
(I think we mostly agree.) From a previos post about Navier-Stokes, the English version may have small holes and the AI make a few changes and fixups when translating to Lean.
reply
This is simply not true that they only invoke lean at the end. They invoke it multiple times in between to reduce the search space like GP said. See how e.g the prove2me harness (which anthropic used for FLT) works for more details.
reply
What kind of a thing do you think AI won't be able to do in 2030? A specific thing please.
reply
Love.

Throw a football at a wide receiver running in chaotic patterns while avoiding oncoming attackers.

Use chopsticks to pick up a single grain of rice.

Olympic level gymnastics via neural-control of muscle fibers in an extremely precise signaling orchestration.

You may say the above are robotics and not “the model can scale to this” domain but I’d counter an animal can do amazing things exactly because of its generalizable and plastic neural architecture and that is what’s in scope.

reply
Have you seen the progress in robotics lately? They aren't yet at a level you described, but quite close - so I'd say it would be possible in 3 years even without intelligence explosion. (E.g. if all frontier AI research stops we'll still have improvements in robotics.)

There are already Unitree robots which are much closer to Olympic level than an ordinary dude: https://www.youtube.com/watch?v=Y3zb4o47yqA You might argue but it's kinda at a level you described - I think if Unitree wanted to make a football-thrower they could just do it now.

reply
Shoot I can’t do those things either!
reply
I’ll grant you love, and I’ll also grant you 2030 on these, especially the last one. I suspect the football and chopsticks ones are very achievable by 2030 if anyone cared enough to focus on them.
reply
Fully autonomously run a business that generates $10+ million in ARR.
reply
What about one that generates $1+ million ARR?
reply
I find that unlikely as well. I should note though, that I incorrectly wrote "running a business", when I meant to say "scaling to $10+ million in ARR". I don't think there's necessarily that much difference in running a business of these two sizes, but scaling to $10m ARR is much harder than to $1m ARR.
reply
Sure.

If $1MM is also unlikely, how about $100k? Do you think that's unlikely as well?

The point is: what is the smallest amount that, if surpassed, would surprise you?

reply
It would surprise me if a fully autonomous AI company could even stay afloat over multiple months. Agents are getting (a lot) better at task completion over longer time horizons, but still occasionally tend to make really bad mistakes. What are the time horizons right now? Something like 40 hours with a 50% success rate? This is on benchmark tasks! The real world is completely different, I mean companies are scaling back efforts to even let agents handle things such as customer service. A fully autonomous business without any human oversight is on a completely different level to that. 2030 is not that far away, and I just don't buy that these models are exponentially improving in anything but a small subset of domains.
reply
Either AI won't be able to do this: - Make a version of itself 100x more efficient that runs on commodity hardware and undercuts athropic.

Or, alternatively, it will be able to do that. Either way there's no world where one company bottlenecks and holds the keys to AGI, and frankly that's good.

reply
- Keep electricity costs down

- Stop lying

- Design personal quantum computers

- Save fanatics from the delusion that AI will solve their life

reply
I mean something a human can do, but AI can't.
reply
Lay brick? Install pipe? Wire cable? Plaster wall?
reply
Here's a brick laying robot from 2017...

https://www.youtube.com/watch?v=6s17IAj-XpU

so yea....

reply
Try telling it to stop or move to a different wall in natural language. I mean, you might as well call a train AI with definitions this generous.
reply
Show me the car in which the robot drove itself there
reply
Self-reproducing physical agents (robots with onboard AI and ability to change their surroundings/build physical things) with human or higher intelligence which don't need a global supply chain of microchips, electro motors, etc. but could live off the land. That's basically the human concept, right? Totally untenable for AI/robotics of the current generation. Would require something like 3D-printing new chips but also 3D-printing new printers. Probably never happening. AI is fundamentally different, reliant on data centers, which are immovable and more like organs than organisms. Robots only come out of factories, not out of other robots or eggs layed by robots. So as Jack of all trades in the game of survival on Earth, humans win for the foreseeable future. Also, AI currently can't exist without humans, whereas humans have thrived without AI. Whether a silicon based ecosystem can ever survive on its own is an interesting question in and of itself. Because no matter how smart data centers become, they degrade fast, and then what? A data center needs maintenance. But if robots maintain the data center, who will maintain the robots? Does this equation ever solve? Humans have bacteria, a complex microcosm of life maintaining them. I don't think such a dependency-hierarchy of "life forms" is feasible for AI/robots, and humans will always have to be in the loop. That's one answer to your question, and not just for the next 3 years.

What stands axiomatically are human-to-human connections that we're wired to crave. Not to mention human reproduction.

Other points: - Editing a live action movie that will get 8.0 or more on imdb (AI sucks at reading emotions and won't be able to tell apart good takes from bad ones) - Reliably making good jokes (many humans can do it, AI weirdly sucks at it) - Accountability (contradictory because of its undead nature) - Reliably not lying/cheating (with no accountability, incentives will always be misaligned)

reply
The nature of mathematical proofs requires us to stay vigilant and careful. OpenAI pulled some of the research results from their famous repo due to simple mistakes that where easily spotted by human experts. There have been infamous mass of incorrect Fermi's theorem proofs.

LLM-assisted research is proving to be useful, rewarding and joyful, but we should not treat LLMs as a perfect machine, incapable of making mistakes. Especially in such massive proofs as this one.

I won't say that it "doesn't count". I will simply ask for staying rational.

reply
Open AI mixed some Lean verified proof with English-only proof. It's important to read the fine print of the anouncements.
reply
What's so surprising about this? A lot of these problems can only be solved using a significant amount of capital. At home or work, using a few-hundred dollar token budget, it's mostly a slog and requires handholding for the majority of open-ended problems.

This technology won't be available to the broad public in the scale that it's available to a select group. If it can only be deployed to a limited amount, to most it remains a magical inaccessible box. And that majority should naturally be incentivized to look for cheaper alternatives, and to "keep moving the goalposts", or in simpler terms, looking where they can provide value, of which there is plenty!

reply
> A lot of these problems can only be solved using a significant amount of capital. At home or work, using a few-hundred dollar token budget, it's mostly a slog and requires handholding for the majority of open-ended problems.

Consider just how much more inference capability one can buy per dollar spent today as compared to a year ago. I can get tasks done quickly for essentially no cost that, early last year, might have cost a frontier lab hundreds or thousands of dollars and would have been unavailable to the general public.

In my mind, the bigger unknown is whether actual profitable use cases will show up quickly enough to justify all the (enormous) investment made in the technology.

reply
By now, many open math problems have been solved with a cheap subscription. Not of this magnitude, of course.
reply
> I know this thread will be full of people in denial about [my prediction]

You can only be in denial about the past or the present. The future has a chance of being different from your predictions.

Personally, regardless of whether you want to call it "denial" or something else, I'm fervently hoping the AI never achieves superintelligence, but I would settle for it not achieving it until we can do our best to ensure that the superintelligence will be benevolent towards us.

Call me selfish, call me petty, call me a human supremacist, call me anything you want, but I prefer humans being the pinnacle of evolution. I like being able to point to our civilization, flawed as it is, and say "No other species we know of has achieved the same."

reply
I'm one of those people who's skeptical of recursive self-improvement and superintelligence. My rationale for my skepticism is that the algorithms get too much credit for the results. The algorithms ultimately run on hardware built by humans and learn from data generated by humans. In the same vein, I think knowledge work gets too much credit for building things and generating high quality data. The output of knowledge work is only useful if it can be applied to the physical world e.g. a CAD file is useful when the actual thing it describes is made in the real world. Similarly, high quality data comes from several feedback cycles with the physical world. I think people generally underestimate how much of human capability is driven by our sensorimotor skills. I think the idea of recursive self-improvement / superintelligence is just an extension of this attitude.

Humanity as collective does not buy anything, or send the design to the plant to be made. As a collective, humanity transforms the natural world into things, for example, people take fibres from plants, turn them into yarn, weave yarn into fabric, and then turn fabric into clothes. Similarly, humanity as collective does not "buy" chips, people turn quartz, bauxite, copper ore, petroleum, etc into the components of computer hardware.

There's a common adage about how everyone should work as a service person to learn patience and empathy. I think everyone should also work in an assembly line and a mine to learn about where the things they use come from.

reply
I don’t understand the technical basis for belief in continuous recursive self-improvement. Frontier models are increasingly impressive at complex math and coding, but the math and code underlying the transformer architecture are not complex. Major advancements in capability have come from scale: more data and more processing power. How does solving the Riemann Hypothesis (or similar achievements) further increase those? Aside from supplying fodder for investment-seeking hype.
reply
Below is just a hand-wavy argument.

Neural networks are function approximators. They're great because they can be trained to approximate almost any function.

What if, somewhere out there in platonic space, there's a complicated function that represents what a philosopher might call true intelligence. Perhaps it's structured totally differently than a neural network and shares no common architecture with a transformer.

Transformer-based LLMs are likely completely different in their fundamental structure. But perhaps the neural networks we have today are succeeding in intellectual tasks because they are beginning to approximate it. Like a bitmap approximates a vector image, it's not the same; the neural network is merely made of a bunch of repetitive simple elements, tuned to imitate the real thing. A higher resolution bitmap image will be a better approximation but still never become a vector image no matter the pixel count.

But at some point, the neural network might be good enough at intellectual tasks that when tasked with searching function space for an algorithm that does the job its neural network does but better, it finds the real thing it's been approximating all along.

reply
There are many big technical question-marks on if RSI is even possible. However, the advances in LLMs that we've seen is definitely not just limited to data and scale.

First: modern LLM architectures are quite far from the GPT-2 era (delta/linear attention, GQA, sparse attention methods, looped transformers, mixture of experts, engrams, etc).

Second: RL environments are largely autonomously generated at this point, as models get smarter they can generate even more complex environments to train in. While this does go into the "data" category, it's not human-generated or even gathered data. This gets particularly spooky when you think about things we cannot yet simulate, that models can develop mathematical methods & simulators for. For example: better/faster physics simulations for robotics RL, better biological simulation environments etc.

reply
My understanding is that LLMs can be used to make small improvements on top the transformer architecture, to make LLMs that are even more intelligent and so are more capable of making more improvements. At some point we might reach the practical limit of intelligence using transformers, and if the LLMs of that time aren't able to make the leap to discover a new architecture, then the improvement stalls. I don't think it will be possible
reply
I don't think you need to be in the football team mindset of us vs them, we're all seeing the progress but most of it is still built on top of what humans have put together over decades and not completely new inventions, which is probably what would convince most that we've achieved "superintelligence" or whatever you want to call it.
reply
LLMs are jagged in their capabilities. Opus 4.8 was already an excellent programmer, but it played Pokémon Red far worse than a human child, even if it did, eventually, manage to beat it.

I note that the dazzling events that have happened have all been in coding, cybersec, and math, that is, domains where the correctness of the work can be verified entirely inside the computer. It is unclear whether further progress along these lines leads to crazy scientific breakthroughs too, which is something people are taking as a given when it comes to superintelligence.

reply
The pace of progress is incredible, but the end result is hardly unexpected. We're talking about a theoretical domain that's easily verifiable and has no side effects.

You have the input (hypothesis), the output (true/false) and the lean kernel. There exists some path from the input to one of the outputs, you "just" need to find it. You use AI to search this exponential space, occasionally prunning ideas that are dead ends, and throw as many agents as possible to scale it up.

reply
In order to keep my mind safe from hype waves, I try to set my expectations periodically and see which of them happened in retrospect, updating my beliefs accordingly. At 2026-09-22 I made a personal 10 statements "prediction" to happen by 2027-03-22, which one of the statements was:

AI will solve exactly one or two millennium problems.

So I will become impressed by something, but only this is not enough. If AI solve three or more problems, I will become impressed and believe that some serious RSI stuff might be happening. If AI couldn't solve any problem (which it seems it did) I would become impressed in the other direction, and believe AI got stuck in a way that I didn't imagined (together with other 9 statements).

There may exist people (like me) who don't believe superintelligence is happening with current set of evidences, but believe it with some stronger evidences.

reply
Whether AI with it's math achievements is extremely cool, whether it's intelligent in a philosophical sense and whether it's capable of RSI are 3 distinct topics.
reply
These disagreements are natural and expected if most people are not properly defining their terms. What do you mean by artificial / super intelligence (explosion)?

I think it's a perfectly sane position to acknowledge amazing progress with very large neural networks, expect further progress, and also not buy into the specific mythology around the singularity or extropianism.

These things come bundled together quite often, and that's what I see people (reasonably, imo) resisting.

Doesn't mean there isn't unfair dismissal too, but I feel that's gotten much more rare.

reply
> self-improvement / intelligence explosion / superintelligence

These terms mean different things to different people. What are the particular implications of this that you're suggesting are imminent?

reply
I think this is all real and counts. Where I struggle is explaining to non-technical people how their everyday lives are going to change directly or indirectly. I hope these discoveries are the first in a sequence that will culminate in or surpass a cure for cancer and diabetes, cheaper and cleaner energy and water, and other tangible everyday life improvements that influence millions of people.
reply
What is the action item exactly for non-technical people? How do you prepare for “things will be different in unpredictable ways at an unpredictable time”? Even if that time moves sooner, still feels like very little info to truly act on.

If you watch movies or read fiction, we’ve expected for years that life in the future will be very different.

reply
Support building more datacenters, promote and integrate SI at your workplace and in your personal life as much as possible, stop resisting, stop criticizing Anthropic and OpenAI, purchase their subscriptions and stock, or be left behind to dig around in the dust
reply
> SI

lmfao, buddy

reply
How does one fail to see sarcasm this obvious?
reply
Oh that’s easy, you scroll HN seconds after waking up and still in bed, that’s my bad hahah
reply
Yeah honestly I don't blame you for it, I feel like it's at most half a step up from what you'd see normally
reply
Fair enough lol, I just thought it might have even been too direct at first
reply
Seen enough freaks on twitter use SI unironically that my eyes glazed over the instant I saw it here and my brain turned off
reply
I think there's a key distinction missing in a lot of these techno-optimist takes.

Abundance of productive capacity does not automatically equal abundant access to its products.

If all of this happens in a way where a few people control these new, automated means of production, all of these wonderful things do not automatically follow.

Why would they share? And worse yet, if they have this automated production capacity without reliance on labor, why would they interact positively with society at all? What's their incentive? And no, you can't say markets, because once you have a machine that builds you whatever you want, you don't need to trade anymore, money and markets become meaningless.

You're basically reduced to hoping for charity from a group of people that have repeatedly proven their lack thereof.

reply
A cure for cancer and disease will only be available to the oligarchy, we the peasants will be killed off since we're no longer needed.
reply
This is conspiracy theory level thinking. "The oligarchy" consists of humans just like you, who also want cancer and disease to be cured for everyone just as much as you do.
reply
Human history is full examples where the significantly more powerful and wealthy relish in stamping out the rodents
reply
Yes, and that were the limits of "AI", "SuperIntellingence", etc become clear.

The world already produces enough resources to feed and cloth and house every single person. Many pf mankind's fundamental problems are no due to lack of technology.

Heck, why can't I have a proper OS, if AI is working wonders?

reply
RL is not scalable to super intelligence. Super intelligence should be able to learn continuosly without verifiable rewards.
reply
You don't need RL all the way to ASI. You just need RL far enough that the model becomes good enough at AI research that it discovers something better than RL.
reply
> becomes good enough at AI research that it discovers something better than RL

That might require continuous learning without verifiable rewards. RL models might help humans join the dots faster, but we don't know how many years that will take.

If its not possible with neural nets as we know them now, and requires a number of completely novel breakthroughs, there could be far from exponential progress even with the best RL models.

reply
Yep, I agree. Improving AI might get harder, faster than the AI gets better.

But it also might not, which is the scary part.

reply
I cannot really comment on RSI, but I think it is important to make the point that an LLM doing large, even crucial parts in a proof of RH, it cannot be viewed in an anthropocentric light.

"If a human proved RH, they would clearly be extremely smart, so imagine how smart the LLM is!"

It is not surprising to me that LLMs show extreme aptitude for math while still having clear deficits in more real world applications. Handling stuff like super difficult math is precisely where AI has its greatest advantage over humans. Perfect recall, never tiring, never lose focus, attention to detail, etc.

If there is one area where AI can be pushed extremely hard ut is math. So the most amazing progress should be found there.

reply
The terms "super intelligence" or "intelligence explosion" are imprecise to the point of being useless. We have "super speed" in the form of automobiles. We have "super strength" in the form of excavators or bull dozers. We have recursive self improvement of tools by having better automation of machining. Intelligence != human ability. The intelligence demonstrated in the proof of RH is breathtaking, but narrow and brittle. It doesn't generalize to having grocery shopping bots, or automated psychotherapists.
reply
I am not impressed because I understand how it works. Not because what things it can or cannot do..

We can start talking when we can train these things without having to beat on its head a million times in a million different ways to teach it one little thing...

reply
How do you think our species was trained?
reply
our species was not “trained”
reply
This is such a substanceless argument, it’s pure vibes. It just says “Anyone who doesn’t believe in imminent intelligence explosion is an annoying curmudgeon.”

There are very solid arguments about why excellence at proving mathematical theorems does not necessarily imply general intelligence of the sort that people associate with ASI.

reply
We’re not arguing with you, we’re pointing to the giant fucking meteor above our heads and laughing/lamenting that you still refuse to see it. It’s not an exchange between balanced discursive peers, not really. It’s an astonished exchange between those of us who have Looked Up about those of us who have yet to, and maybe never will, not really.

I encourage you to try, tho. For one thing, we need good people like you with us in reality. For another, you might die in some horrific catastrophe before you get a chance to, if you wait much longer.

reply
I mean, you're barely arguing, you just sound a tad psychotic.
reply
Being called psychotic by someone with less than one sentence of reading comprehension doesn't really sting like it should, sorry. Doesn't clear the general omnipresent sting threshold, at least.
reply
If people kept applying the same progress expectations to cpu frequency since 20001 we'd have 400ghz cpus by now

It's OK to admit we just don't know when or where it'll srop

Also virtually no one argues llms will go away, but a lot of people argue the current companies might, it's completely different.

reply
There are a ton of people who think LLMs are completely useless, the whole thing is some combination of scam and delusion, and it will all come crumbling down once the music stops.
reply
There is also a ton of people who think llms are Gods and will cure every disease including aging

You need retards on both sides of the spectrum for a balanced society

reply
> superintelligence within months or a year

Didn’t we already achieve super intelligence?

reply
No, we achieved artificial general intelligence. The fact that you mixed up these two wildly distinct terms should be a sign that you should not trust your intuitions about those topic.
reply
Yes, 10 times already. You're supposed to drink a shot of vodka each time you read the term in an article or comment.
reply
I love hn
reply
Both the deniers and claqueurs are from the same ilk: Silly adults who don't have the intellectual capacity to create or even fully understand the subject, yet possess an almost vulgar self-confidence to constantly opine about its future.
reply
Good observation. Perhaps the noble path is to chop wood and carry water, after all.
reply
A slight nitpick: the hacker news equivalent of chop wood, carry water would be something like “charge the car, write a CRUD app.” Ordinary, not mystical powers doing magic. not high faulting thoughts but the basic simplicity of living in each moment and letting the necessary actions happen freely.

Seems the CRUD app may not need writing soon, and the car may charge itself.

reply
Neither of us know what the growth curve looks like, so focusing on just calling one side naive is ignoring the naivety of the other side as well. If you want to extrapolate, go ahead, but that doesn't make you some oracle, nor does it make what you're suggesting factual. There are many potential scenarios and we should consider them, sure.
reply
But it's not clear how well superhuman ability in math generalizes to other domains.
reply
You're listing poorly defined words that mean different things to different people and wonder why people disagree with you...? My guess is that a lot of people you perceive this "denial" from are simply realists who are tired of seeing blind hype.

Note that every technology exhibits extreme growth until it... Doesn't. You can't predict the future. Whether any kind of RSI will happen depends on how hard of a problem intelligence is, and the precise point is that no person on earth knows, otherwise we wouldn't need to dedicate an entire research field to it.

Anyone who is sure that X will happen or not happen within Y years is just lying to themselves. For example, if you're so sure RSI will happen, why not invest all the money you have that isn't strictly necessary for your survival into this? Surely it will pay off in Y few years...

And note that I'm not saying it won't happen! I'm saying it's stupid to make a prediction of either case.

reply
Your exact argument applies in full to climate change. No one can prove that the physical processes we know to be underway today will continue.
reply
Yes, we're kinda guessing there too and I would expect every scientist to have a reasonable view to account for these uncertainties. I'm not sure if you meant to disagree with me, because I agree with you ^^

I also expect climate change to still be a process that is easier to research than intelligence. With climate change you can at least kinda look and measure and apply general principles of geology and physics. With intelligence, we've been at it for ~60 years and don't even have a definition yet, don't know what causes it, have no instruments, next to no introspection capabilities that neurology provides, etc.

reply
I fear this is the result of a massive disconnect based on what you were told to expect. I read Kurzweil when that first came out, and maybe I'm remembering wrong, but I remember the singularity being the rate of technological progress in all domains will be so fast as to be asymptotic to an infinite slope and anything that will ever be possible will happen more or less instantly. Scott Alexander told me in a comments argument on his blog ten years ago that encryption is pointless because an AI will turn Jupiter into computronium and break it. Zvi was bandying about with the "take over its light cone" and "erase all value from the universe" phrases.

If that's the standard, then sorry, I'm not impressed by proving the Riemann hypothesis. If the standard is rapidly improving software for generating other software (which may or may not itself be any better than existing software) and math proofs (that may or may not have any meaningful real-world impact), then sure, I'm impressed. It doesn't mean I expect the daily lived experience of more than a few hundred thousand people globally to change much in the next decade, and most of the ones whose lives will change are software developers and mathematicians.

On that note, I have to imagine you have something different in mind when you talk about the "original meaning" of superintelligence, as you seem to not be talking about the Kurzweil/Bostrom/Yukowsky type of being effectively indistinguishable from God or at least the ancient alien civilizations from the Space Odyssey or Expanse novels, which again, I'm not seeing much of a hint toward because we have very good automated theorem provers. Proving Riemann doesn't mean we can now figure out zero point energy and turn planets into stars.

reply
people always under emphasize too that the gains of tomorrow aren’t made with the technology/tools of today (I think Kurzweil is where I heard this first). RSI, if landed, seems like that on steroids
reply
> people in denial about superintelligence

As long as I get to keep my house...

reply
I won't believe it until I get a robot to put my dishes into the dish washer, put copper meshes and squeeze spray foam under my siding panels.
reply
No offense, but have your heard about the infinite monkey theorem?
reply
> Experience has shown that, even now, there will still be people explaining in patronizing tones why [...]

This is true no matter which side of the debate you're on. See [1] for an example.

[1] https://news.ycombinator.com/item?id=50575797

reply
deleted
reply
I would say, seriously, that this is bounded superintelligence. That superintelligence needed top human talent to achieve the goal. It was not a prompt like: "solve the Riemann hypothesis".

Coincidentally, yesterday I did a small presentation on cybersecurity and AI where I made a similar point: scarce top talent in cybersecurity (e.g. Calif) gets superpowers with AI, while AI replaces a lot of the skills of average cybersecurity professionals using simple prompts.

I think the breaking point is when AI can solve complex problems on its own that even top talent can't achieve. IMHO, I've seen some of this in the latest Google models (especially Carbon, and then Argon), where they can do more research independently.

reply
"Human-assisted superintelligence" (HASI). ;)
reply
I think you can be extraordinarily impressed and believe (or expect) that there significant limits to this technology.

I mean, AI could make very impressive 20 second videos two years ago. Progress has been slow and incremental since; Ten minute video still look far away.

I'd acknowledge is making quick progress on things you can find massive and reliable data on but even in it's good mode it seems to have palpable weaknesses. Someone was claiming Claude has become more and more persuasive over time but all the generators "still sound like slop" to me and many people.

reply
Still, I have hope.

It’s not too late to Look Up for you. Yes, I’m talking to you — the person reading this with a vague grimace of disbelief and discomfort and disdain. It’s not too late for you to swallow some pride and reassess. Everyone has to eventually, especially me; why not now, for you?

So please. I’m begging you. LOOK. UP.

reply
I looked up, it said 'naive'

:/

reply
Stochastic parrot truthers in shambles.
reply
It’s a bait and switch to use a harvard fellow of mathematics solving a problem using AI with “The AI is super intelligent”. People are getting really tired of the constant bait and switches and subterfuge in press releases so unsurprisingly they just start rejecting it by default.
reply
I don't see anybody claiming that solving the Riemann hypothesis would be proof that current models are superintelligent. Reminder that the definition of the term is "any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest". I do sometimes see people arguing that current models are AGI, but I don't think even this is true - current models are slightly superhuman in some narrow domains like math, human-level at a bunch of them, and still pretty stupid in some ways, most notably agency.

The reason people worry about superintelligence is that one of the "domains of interest" is AI research, and once you have an AI that's as good at AI research as your human researchers, your AI progress shoots up, probably exponentially. So the delay between automated AI research and superintelligence might be tiny.

reply
You can believe what you want I probably agree with your nuanced view. But I would invite you to control+F this thread for “superintelligence” there are plenty of aggrandizing statements about what fools everyone are for being dubious about this being superintelligence.

(e.g)[https://news.ycombinator.com/reply?id=50575797&goto=item%3Fi...]

People are just tired of listening to these statements and being called idiots constantly with no evidence other than other press releases and disingenuous framing getting echoed constantly in volume across corporate media and all social media that allows LLMs.

reply
I don’t think like that, the most important question is: could he have solved the problem without AI assistance, and in the same time frame? I’d guess no.
reply
A few years ago if someone had said “we used an HPC cluster and Pandas to build some tools that in turn solved this problem” no-one would’ve batted an eyelid and no-one would’ve been making grandiose statements about the end of the world due to HPC computers and the power of Dataframes. Personally I think If someone had had a few hundred billion dollars of investment pumped into a bunch of mathemeticians to solve this problem then yes they probably would have solved it too and maybe that might have raised some eyebrows. Everyone of these conversations just feels like an extremely highly charged argument about Vim vs VSCode.
reply
We're really using a term invented by Trump yesterday for something we've been working on for 80 years?
reply
The term was not invented by Trump. For example, the (in)famous book Superintelligence by Nick Bostrom was published in 2014[1]. It has a long been used to refer to a theoretical general AI that far surpasses humans.

The Trump thing is a insane newspeak attempt to forcibly replace the use of "artificial intelligence" with "super intelligence" in a general sense. It is ridiculous, obviously, but the best thing we can do is to continue using "superintelligence" in its original, narrower sense.

[1] https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dang...

reply
Actually, I think the best thing we can do is take Trump round the back of the woodshed.
reply
Superintelligence is an old term with an established meaning. What Trump does is to use it for AI in general.
reply
[flagged]
reply
> regime

Likewise, this is the term you use when you want to align yourself with

reply
....unless you literally do have a new silicon god (in which case Trump is irrelevant).
reply