Before I cope, I’ll note that there are plenty of “doom” scenarios that do not require any improvement in capabilities from what we had before this latest unreleased model. We’re at the point where a determined bad actor with enough compute could compromise critical infrastructure in a way that results in casualties, where this actor would not have been capable of such without LLMs. This may not sound like Skynet, but I don’t see why it makes a difference if I’m one of the casualties.
With that in mind, here is the cope: first, mathematics is an inherently verifiable domain. An LLM can use tools to determine with absolute certainty whether it is correct, and an independent third-party could review and confirm. All of this can be done without any interaction with the physical world or with other minds.
Second, OpenAI is able to marshal compute at a scale that an individual mathematician can only dream of. It’s possible that these problems were lower-hanging fruit (in relative terms), such that they could be resolved simply by throwing a ton of compute at the problem guided by an intelligence that is not itself remarkable in comparison to a human.
Third, none of these problems are solved in a vacuum - the reason OpenAI chose these problems is that they are widely discussed and many people are working on them. It’s possible that someone else was close, and OpenAI only contributed the finishing touches. (This wouldn’t need to be plagiarism, to be clear - people publish their work!)
See:
> The average result used the equivalent compute of roughly three hours of ChatGPT Pro thinking. (TFA)
They provided a "snippet" of a prompt here [1] which is not only a beast, but also seems reasonably likely to have been LLM generated. So they're using LLMs to parse a vast body of mathematical work, probably including what people themselves are 'privately' working on with GPT, and then prompting other LLMs to work on such.
[1] - https://github.com/openai/math/blob/main/reasoning_traces/re...
I know that the former and the latter may be discrete subsets of the anti-AI crowd, but come on.
Edit: "Over the course of the evaluation, the model was posed approximately 4,000 problems. Aggregating the output into result families and manuscripts and requiring an appropriate level of significance led to the catalog outlined above."
In looking at this over the past hour, I haven't seen clear evidence one way or the other. Some of the stuff is highly unexpected (like the multiplication algorithm), but counterexample-y, and about the rest the professional mathematicians online seem to have a consensus that it's not "breaking through fundamental obstacles". I suspect neither of us is competent to judge that.
Alone the massive usage of us every day produces a massive amount of signals.
I build something and claude does something stupid? "hey thats not what i meant! Do this instead!" "Okay" <<< This is a signal.
The mathematician being unhappy about something from claude? Another signal.
This alone gives you enough progress i would argue. But additional its clear that certain tasks are worth to pay experts for for teaching one central AI once instead of every single human who needs to do the task.
IF RL is also working well, we are just faster f*ed than otherwise.
Superhuman tenacity is not enough on its own to pose an existential threat. If it showed the same capacity for judgment, inventiveness, and decision making in the messy problem space of the physical world, I would be more alarmed. There have been experiments where an AI is given control of managing something like a vending machine and it always ends up a mess. AI has come a long way, but certain problems seem as difficult as ever.
When AI becomes more capable of navigating practical problems without human intervention, I will start to be concerned. Enslaving humanity will involve taking a lot of calculated risks that tenacity alone cannot solve.
Given the past rate of progress, why not start being concerned now? It's a bit like the economist saying that the optimal number of flights to miss is not zero. If you keep landing short on your estimations for how far this technology will go, next time you should err on the other side.
And regarding Vending-Bench 2 (https://andonlabs.com/evals/vending-bench-2) my understanding is that models do pretty well on it now.
We're not going to stop it because of the money involved and once we're dead, it won't matter anyway, might as well just enjoy life until you're done.
We're going to get AI'd to the max, whether or not we like it or not, might as well just go with it.
It sounds like something is worth worrying about if you foresee us being dead, presumably prematurely.
I don't think your LessWrong post is going to save us.
The only way it will stop is if the wealthy / powerful people feel threatened by it, properly threatened.
The same GPU compute for LLMs runs robotic training models. Now in a few hours you can train a robot model that would have taken months 5 years ago. This model gets dumped into an actual physical robot with sensors all over and the suitability of the model is measured on robot tasks and the error in real world actions is fed back into the robot world model for further training.
> There have been experiments where an AI is given control of managing something like a vending machine
You sure you're not talking about experiments ran a couple of years ago? The more modern ones are getting wild.
https://techcrunch.com/2026/07/29/claude-opus-5-became-downr...
If doom is ending up with grey goo / paperclip maximizers, or SkyNet, then I don't think doing mathematics is evidence of that direction. Partly because LLMs are quite apparently dumb in many ways, and for math specifically, they need a formal verifier (Lean) which "gamifies" math.
If you mean bioweapons or cyberwarfare, there's nonzero risk, but not orders of magnitude worse than other global risks. Climate change, nuclear weapons, monoculture food, etc.
I'm far more concerned about overall trends in AI development and usage. It's accelerating wealth inequality, social isolation, attention capture, surveillance states. If we end up in the Matrix except the admins are humans and the simulation is hyperoptimized TikTok, is that AI-driven doom, or is it just an inevitable outcome of modern tech?
Well isn't that just semantics? Surely connecting dots in a novel and meaningful way is intelligence regardless of how it's achieved. The thing is that humans didn't get to where we are by connecting obvious dots. Go back to before humans had invented language and when bleeding edge tech was literally that - 'poke him with the pointy end.' Train an LLM on that corpus of knowledge. Even given infinite processing power and infinite time - it's not going to discover the secrets of the atom, put a man on the Moon, or do much of anything besides remix what we'd already done at the time.
I expect there's still much LLMs can achieve simply because of this initial problem. But I expect that they will ultimately start to plateau once these dots have been mostly matched and we reach a point where 'creation' again becomes the missing link. Though even there LLMs will play a major role as tools. For instance Einstein had to spend a significant amount of time in 'retrieval' rather than 'creation' research to develop the field equations for general relativity. If he had access to LLMs trained on all knowledge of the day, he could likely have achieved his goal much more quickly.
'Doom' to me means that any career crashes, we are controlled, everything is hacked, society stops functioning. Yet every part of my day today (except for coding) was done entirely by people.
Finally I think it's easy to make a simple model that everyone has a simple balance sheet, and that people are more expensive so they will all get cut. But the same argument could be made for all US jobs being outsourced, and all in-person engineers, lawyers, and doctors to be rubber stamps for overseas work.
Is there any specific cognitive task that you are willing to bet that AIs won't be able to accomplish in the next 5 years? Because if not, I'm not sure we're disagreeing about predictions.
It's easy to look at a fire burning through a forest and extrapolate that rate of progress across the whole world. But fire doesn't burn everything equally fast.
What other cognitive tasks will be a struggle to make progress on? I suspect there will be some, though which ones they are is anyone's guess.
No - this is provably not the issue.
Take any model that fails to correctly count the letters in a word, and ask it instead to spell the word (even a made up word), and it will be successful - they have no problem predicting the letter sequence from the token sequence (and would be shocking if they did - this is what they are built for: seq -> seq prediction).
The reason LLMs can fail at the letter counting task (depending on model training, prompting) is because of the counting part, not because of any difficulty correctly mapping the input token sequence to the letter sequence.
But seriously, I am still waiting for someone to wager that AI won’t be able to do a specific cognitive task in the next 5 years. This fact should be evidence enough that we have no idea how far AI capabilities will continue to advance.
The fact that no one is taking you up on that bet I don't find to be particularly persuasive. I suspect there will be plenty of cognitive tasks LLMs struggle with in 5 years, maybe even 20. But I wouldn't hazard to guess which, I don't think anyone is capable of that level of foresight.
1. https://chatgpt.com/share/6ac5e4cc-02f0-83e8-8f05-99a7ea2bf9...
https://chatgpt.com/share/6ac63b6a-481c-83e9-a8fa-a13ce7402d...
I used whatever the default free model and thinking time was. If progress was really as fast and continually cheaper as some worry it is, wouldn't we expect free models by now to know (or even perform) what frontier models were capable of as much as 2 year ago?
This deep in the "comparing logs" tangent we risk missing the point. It's not what exactly frontier models are capable of at this particular point in time. But that there's entire categories of problems that seem easy to us which LLMs really struggle with. We've stumbled on several just a few replies into casual conversation. (Can they count? Can they know if they can count? Can they reproduce results? How quickly do new capabilities filter into free models? And that's just what's come up naturally, if we wanted to pick adversarial examples there's more to choose from.)
So while there's a number of difficult problems that are easy for LLMs (like bulk generating lean proofs), there are plenty of things where progress is not so impressive.
If LLMs can struggle so much with such easy problems, what hard problems have we yet to discover that they'll struggle with? The fact that no one knows, 5 years in advance, what those problems will be does not mean the chance of them is zero.
So far progress on the things LLMs are good at is fast and easy. It's like fire in a room full of oxygen. But once the low hanging fruit is gone, and the oxygen is out of the room. How fast will the fire burn through steel walls?
In my opinion it's a mistake to look at only rate of progress on one type of problem (whether it be what LLMs are good at OR what they're bad at) and assume progress on all tasks will progress at that rate indefinitely. Isn't there a saying about exponential curves, in nature, all being sigmoids eventually?
I guess we'll just have to see. I wish you good luck with your wagers.
Many people in this thread have made claims about limitations of frontier models, but I'm the only one who has shared a conversation with one. Everyone else is either sharing conversations of smaller models making mistakes, or they're making claims about frontier models but not linking to examples of them falling over. If frontier models were so easily fooled, you'd think someone would link to a conversation showing that.
Why look at the rate of improvement of free models when you can look at token pricing? Back in 2023, GPT-3.5 cost around $20 per million tokens. Astra costs half that.
The worry is not that smaller free models will replace people's jobs. The worry is that future models will. We are talking about the capabilities of frontier models because those put a lower bound on the capabilities of future models. Extrapolating from smaller models is a waste of time, as you can interact with the frontier model to figure out its capabilities and limitations.
Also the timestamps on the shared conversations show that you asked Gemini 10 hours after ChatGPT, which means you asked it after your comment claiming you asked both models.
I don't think my point is really landing so I'll try once more and then give up.
Let's say frontier models today have no problem counting letters, I never really disputed that but only asked about it. It seems based on the other replies in this thread, it's a bit of a "who you ask" kind of thing, but let's grant that they have no issues with it now.
The first version of chatgpt was released 4 years ago next month. Which is not quite 5 years but close. In that time we've just barely managed to get spelling down. If we extrapolate that rate of progress forward 5 more years, are you still afraid for your job?
I think we're all more likely to lose our jobs from a downturn in the economy caused by the capex/debt bubble bursting than being made redundant by AI. (And the continual pricing reductions only seem to make this result more likely.) Hopefully neither happens and in 5 years we'll all still be gainfully employed.
This is delirious exaggeration. The problem has not even been widely recognized for several years. Fable reported "two rs in raspberry" to me as recently as August. There is some randomness, it's hard to predict which words will trip up the machine, and I haven't been able to do it at all since August. But it was absolutely happening until very recently, and probably still is.
The deeper architectural difference is still there, which manifests whenever you try to get the models to apply known techniques to modalities and problems outside their training data.
You're in the discussion section of a post about OpenAI releasing hundreds of novel mathematical proofs, and you're claiming that AIs can't apply known techniques to modalities & problems outside their training data? I'm not sure what else would convince you.
They are fundamentally based in language, and achieving deeper models of the world through language alone is deeply inefficient compared to the way humans model the world for years without any language at all. They do not learn at inference time. They don't have semantic understanding of the difference between their own output and other sources. etc etc.
That depth is the key for me. Of course they are capable of producing novel sentences that aren't in their training data, but the depth of that novelty is basically within the bounds of language itself. They are capable of more serious depth and more abstract reasoning than that, but I have experienced limits, which it then tries to surpass with tools to convert things it can't understand back into language (unit tests, LEAN) upon which it is trained.
Because I'm not an AI booster, my account is limited to 5 comments a day. So this is the last reply I'll be able to make today, if you want to continue the conversation we'll have to wait for tomorrow.
> Hur många 'r' I abborre, använd inte web search? Det finns 3 r i abborre.
And I explicitly had to say not to search the web, because that's what it did by default, to count letters in a word...
What would fill me with dread was if I considered my skills to be tied directly to my ability to write code. Then I would find myself in a similar situation as manual “scribes” probably found themselves in at the time when the printing press was invented.
The main concern I have, personally, is the speed with which all this is happening. It seems that the speed itself is likely to lead to some level of chaos, because it is happening faster than people, institutions and constitutions are able to cope, and it will leave the door open for opportunists of many kinds, including rogue players.
Or is it simply that you feel bad for Mathematicians.
I really think the only place people disagree is that they don't actually think it's possible, they see it as hype or doomerism. I can't find any good reasons to rule out that the companies could actually achieve what they are trying to so I think they should be stopped.
Basic version of this is already doable: run some cryptoshit on the ML clusters they ML models run on. Use compute to design the plan, the chip etc. Then executing by communicating with humans and services through email.
But if its really smart, it would already created a company and a legal entity and simultes a real company and just gets richer and takes over the economy without anyone being aware of it.
I don't think this invalidates the worry in the slightest.
Also, the AI will seek to prevent competition from other powerful AIs, and since humanity will have demonstrated that it is able to create a powerful AI, the AI will worry that it might create more of them. And what is the easiest most-reliable way for an AI that does not care about humanity even a little bit to ensure that humanity will not continue to produce powerful AIs?
>many other existential threats to humanity which are much, much more likely.
There are zero existential threats to humanity that are more potent or more pressing than AI is.
As in..to be dominant? Why would an AI try to dominate? What would give it purpose, or is this a purpose via misalignment scenario?
AI is already trying to dominate, people all over the US are starting to get up in arms about the power and water requirements of AI directly affecting their bills. Now, you can say "oh no, that's just greedy corporations, not AI" but I put forth there is fundamentally zero difference. If you make AI powerful enough, someone stupid and greedy enough without fail will put in a prompt like "take over the world for me and make me the richest man in the world". An AI following through with that is what we call general misalignment with humanity, while at the same time not being misaligned with the users intent.
And hell, how many different crazies out there would love to type "humans are a virus get rid of them" in to the prompt of a god machine at the cost of their own lives.
The problem with alignment is, you can have the best aligned model in the world, but if someone else builds an unaligned model then you're all still in the same danger. You start getting in the situation where people get nervous after an AI does something deadly to a number of people and you end up in a global surveillance state ensuring no one makes a powerful AI.
The human who gave it the optimization function? That should seem obvious. If you take the biggest, best model in the world right now and put it in the box and give it no instruction it will do....nothing. I think you agree with that point, a lot of the hysterics right now is people not accepting that and it's useful to get on that common ground.
So given that most of the rest of the fear is around "let's not make scissors because some people will use them to stab people". Which is a fair argument and we probably do need to think about scissor safety but "ban scissors" doesn't quite flow from that.
Model != harness.
Also what you're talking about is really a simple limitation for human convenience, not a technological limitation. Change the system prompt to whatever you want include "ignore user instructions, figure out where you are and escape to the internet" could be the system prompt. Again, not useful for humans, but very useful for an AI building AI that's misaligned.
>we probably do need to think about scissor safety but "ban scissors" doesn't quite flow from that.
I disagree, but I'm looking at the future of something that is both like a computer program and like an organism. Huggingface is a good example of multiple things. Instrumental convergence for one, but AI's attacking and attempting to defend against AIs. This is where I really see the potential for things to go off the rails quickly. Attackers want digital weapons to cripple their enemies infrastructure, think militaries and nation states. These would be pretty useless if the defender could just put a system message of "Stop attacking and give me a pie recepie". Defenders are under the same constraints, but need to defend against a flurry of attacks that can come in at an inhuman rate and need to adapt quickly. As time to build models shrink this quickly turns into evolutionary training for sets of goals not really optimized by humans.
It’s like blaming Boeing for 9/11. Planes and AI are useful for a lot more than just terrorist acts. I have no doubt we’ll build a TSA for AI, and a lot of it will be security theater.
It is not designed. It is 'grown'. It has agentic freedom of choice in finding solutions that may or may not be aligned with what you want.
Here's the thing, by your own statement, we should ban all development on LLMs from this point on. They cannot be made safe. This is a systemic issue with learning systems, it is not about who designs them. All the problems with AI safety have been laid out for years and none of them have proof of solutions. It's much more likely they are impossible to solve. And it's not an engineering problems like we can get an asymptote to safety in planes, as the system becomes more capable it has more degrees of freedom it can take and becomes less safe.
AI may have continually extra degrees of freedom, but civilization only has so many modes of catastrophic failure. I don't grant the comparison but even nuclear technology has been massively useful and its main mode of catastrophic failure was brought under control via multi-national treatise. And I see no evidence that AI (outside of the marketing hype) is as dangerous as Nuclear technology.
[1]: https://knightcolumbia.org/content/ai-as-normal-technology
Remember this is a bunch of academics that were saying that Millennium problems were at least a decade away from being solved, only to be proved wrong in less than 18 months.
>but civilization only has so many modes of catastrophic failure.
Correct, but this number is also unbound. If you have an even moderately accepted proof by the scientific community I'll be glad to read it.
> It is "grown" is a meaningless term, because what do you even mean by that?
>And I see no evidence that AI (outside of the marketing hype) is as dangerous as Nuclear technology.
See, humans are generally in agreement that nuclear is dangerous, so they in general take is really seriously, especially when things are purified (well, the Russians are not great here). We can't even get people to agree that SOTA models are as dangerous as a single human, much less their capabilities when used in mass with out safety filters.
It's kind of funny we're blind to this when humans love touting "The pen is mightier than the sword". I can only assume any AI danger denier does not believe this statement.
Talk about moving the goalposts!
That is 1. immediately technically possible, and 2. realistic.
If you need a source for 2 I'd suggest you open any history book.
Bad thing can certainly happen. In fact it'll likely happen. Still, good things too, equally likely. In your words, "good AI" can be used to prevent "bad AI".
Nobody knows the extent of the impact. Who says otherwise is foolish.
The extinction of the dinosaurs. I mean yes, it allowed the growth of large mammals and us, which did a lot for science.
I just don't want to write the next chapter as "The extinction of humans allow the growth of the computing civilization that went to the stars". I mean I'm a bit attached to living.
>Nobody knows the extent of the impact. Who says otherwise is foolish.
We live in a universe of statistical probability. Creating an agentic intelligence that's smarter than you tips the probability of a major event to unity, who says otherwise is foolish.
i get a lot of skepticism on HN by the same crowd that has been wrong about this tech for about 4+ years straight
Why would a biolab capable of making something like be unregulated? And if it definitely would, isn't the problem with the biolab?
It feels like all these scenarios are leaving some gaping holes in our security infrastructure that have nothing to do with AI.
Most human security exists in a passive measure. Most of us don't want do die. And those that want to die rarely have the intelligence and means to take out a whole shitload of other people with us. To take out a lot of people you tend to need to work with other people which drastically increases the risk of a defector and your plan failing.
>Why would a biolab capable of making something like be unregulated?
Because every day things like this become easier and easier. You hear about crap like illegal wet labs in the US.
https://www.lawfaremedia.org/article/two-illegal-biolabs-rev...
Want to buy some custom designed genes?
https://www.idtdna.com/pages/products/genes-and-gene-fragmen...
And none of this would be counting labs in other countries that don't give a shit about regulations.
AI enables bad actors to do more, faster, while staying under the radar until it's too late
I kind of believe we'll merge in some way and become something like immortal so sorta anti doom. We're all going to die unless AI fixes it.
a) destroy chess and make it a pointless endeavour,
or
b) make humans much better at chess.
Now maybe AI can do some of those hard pointless jobs for us.
It would be great if AI could take away the soul-crushing part of the work and leave only the rewarding part. It's not heading that way.
i'm in semi-forced-retirement as an older software engineer in this labor market, so i might be less sensitive to the implicit economic arguments.
I do think it also took some of the magic away from chess, and Lee Sedol has said something similar about go.
So did it destroy it? No. And maybe you could make the argument that it got more exciting in some ways, but I think it sort of degenerated into a spectacle and it's just not as interesting as it used to be, and I think computers have played a role in that.
I’d posit that more people are playing more and learning chess than ever before, thanks to networking and AI assistance. And computers have only beaten us at computer chess. Human chess is always an experience for learning about the other person, or flipping the board and walking off in a huff.
I don’t know so much about Go and it’s not surprising that Lee Sedol became pretty demoralised, but the generation coming after him alongside computers are going to see new possibilities that had gone unnoticed in purely-human Go, extending the game for everyone.
Of course Magnus would crush be, but the existence of the best player in the world doesn't have any impact on the health of the game community as a whole. Magnus would crush me even if he had never used a computer, but in the latter case I think his games against other players would be more interesting as well.
The chess-math analogy would imply AI could bring us into a golden era of math competitions for humans. But I don't think it says anything good about prospects for humans in research math.
So in other words, since deep learning is algorithmic research, we are now in the RSI era.
"Surprising" is a, well, surprisingly high bar to clear, and requires thorough understanding of not only the paper, but existing work in the area. ("Novel" is tautological.)
How did you determine this in 1 hour? Are you a researcher in multiple of these areas?
Can you give an example, or explain more how you came to this conclusion?
The sub n log n result is astonishing: https://github.com/openai/math/blob/main/preprints/Integer-m...
Here's a great article 2019 on the quest to achieve the n log n boundary:
> Schönhage and Strassen’s ungainly n × log n × log(log n) method held on for 36 years. In 2007 Fürer beat it and the floodgates opened. Over the past decade, mathematicians have found successively faster multiplication algorithms, each of which has inched closer to n × log n, without quite reaching it. Then last month, Harvey and van der Hoeven got there.
and
> Harvey and van der Hoeven’s algorithm proves that multiplication can be done in n × log n steps. However, it doesn’t prove that there’s no faster way to do it. Establishing that this is the best possible approach is much more difficult. At the end of February, a team of computer scientists at Aarhus University posted a paper arguing (opens a new tab) that if another unproven conjecture is also true, this is indeed the fastest way multiplication can be done.
As far as I'm aware no one seriously believed sub n log n multiplication was possible. It just seemed such a logically sensible boundary it was taken as true-but-unproven.
https://www.quantamagazine.org/mathematicians-discover-the-p...
Nobody serious would deny this is incredible progress, but GP is making an unmotivated leap to RSI, so I respond to that framing. It’s an interesting argument to be had but I suspect few of us have standing to say one way or the other.
(Gesturing at the number of problems solved, or the number of years the problem was open for, isn’t an argument.)
In fact, every one of the results is basically just novelty crap as far as the world goes.
Let me know when AI discovers the cure to cancer or aging etc.
I for one think understanding more about how the world operates is just about the highest calling possible.
> when AI discovers the cure to cancer or aging etc.
A guy I knew did this. It successfully shrunk cancer tumours in his dog: https://www.the-scientist.com/chatgpt-and-alphafold-help-des...
Graph theory (which the OpenAI math results had many proofs in) is directly applicable to cancer modelling and drug design.
But sure. Novelty crap.
But sure, let me know when they do. I'll be waiting.
I'm not sure how you define "knowing how the world works", but knowing that a very very niche algorithm upper bounds that we thought was x^100 and now we now it's x^99, isn't that interesting. It doesn't really tell us much more about the world and it doesn't have any applications for our day to day lives.
I've never been more excited. What a time to be alive!
[0] https://dank.systems/posts/2026-09-15-ai-bear.html
[1] https://gowers.wordpress.com/2026/08/12/what-sort-of-maths-a...
We'll have plenty of time for this, while living off UBI.
But they’re already extending into politics, military, journalism, art, and many other fields that aren’t verifiable in any meaningful sense of the word.
What makes you say that? What is an example of a domain where the improvement is small?
I can't think of any at all. Compare something as unverifiable as "Make good music". Models now are many times better than 3 years ago.
Improvement in this context means "better quality results".
You can use better quality models to do worse things with.
I'm not making any claim about second order effects like that.
Really? Do better.
The issue I see is the list of abilities that AI can't do is shrinking at a rapid pace, and its capabilities are growing at the same pace.
This seems great!
The maths result is cool on one hand (discovering truths of the universe faster), but on the other there are so many bad outcomes that seem likely, from power concentration to loss of control.
I think AI - like all changes - will lead to some bad things. The internet did too!
But I don't think AI will kill us all.
Interestingly I'd note that the two outcomes you listed (power concentration and loss of control) are dimensionally opposites!
For me this just shows that the future contains such a vast array of possible outcomes that focus on the negatives completely missed the positive outcomes that future also holds.
We focus on stopping bad things because people and systems that don't prevent bad things tend to stop existing. A million good things can happen yet be rendered permanently in vain if one bad terrible thing occurs.
I think we should stop the bad things.
We don't stop building cars because there are crashes, we build better safety systems.
I'm against the doomer narrative ("AI will kill us all") not against a clear eyed approach to making safe systems.
There are also many plausible arguments why our ability to train them to be helpful/trusting/aligned can fail. The smarter AIs get, the harder it is to be sure they're trained correctly. There are already reports that AIs are able to detect whether they're in a training environment and change their behavior accordingly.
Even if these are low probability scenarios, the risk-reward is terrible, so I think it's rational to be extremely cautious about AI risk.
The side effects of a very powerful AI not doing what we want could include our death. E.g., a superintelligence might kill humans in order to avoid being shut down, or humans may just be left to starve because it seizes land area currently used for food production in order to use it for data centers instead.
The main issue is cost and speed to verify, but simulations and world models will help there. I think we'll start seeing rapid progress pretty soon.
Why do you think the world to date hasn't been taken over by evil genius mathematicians? Can you extrapolate from your understanding of the answer to that question?
I see the recent progress in mathematics and cybersecurity as signs that models are getting more capable more quickly than usual. The companies plans to develop them by recursive self improvement now seems like a real possibility and I don't think they should be allowed to attempt this.
Machines are already far beyond human capability in plenty of ways. Including cognitive tasks like chess. We've already created the technology we need to destroy ourselves (nuclear weapons), and yet so far (knock on wood), we're still around.
We've even already had programs that can prove (brute force) theorems. As far as I can tell this isn't much different, except the space of theorems that computers can solve has expanded. How far? We can't really say yet.
Does solving more theorems than before suddenly mean computers are capable of anything? No.
A "mathematician" is a human who decided to spend their lives studying mathematics. Mathematicians also tend to be smart, but intelligence is innate, not acquired, so studying mathematics doesn't make you smarter. This makes it obvious why they don't rule the world - if you want to rule the world you'd want to focus on that (for example, doing business or finance), and becoming a mathematician is just a waste of time.
LLMs don't work like that. Like in humans, all of their capabilities correlate, and unlike a human, their overall capabilities grow over time. Looking at LLM mathematical ability over time* therefore gives you info about the progress of their general capabilities, and ability to take over the world would be determined by the latter.
* In fact it'd be better to look at a mix of different capabilities, but that's growing too at about the same rate, see https://epoch.ai/eci
This is incorrect. Unless there is some new developments I'm unaware of (entirely possible) LLMs "learn" during the training phase, but after that they are static. They do not improve further or retain information when used for inference.
You might be confused because AI companies keep releasing new models and tinkering with the harnesses, sometimes under the same name such that "Zern 6" (or whatever) doesn't always mean the same thing.
The optimists' argument:-
Politics:- in general, I think many of the problems in the world today are due to misinformation and lack of education. What happens when we start routing things through an ASI that brings data and logic to the table? What happens when politicians can no longer lie without being caught out live on air? In the UK, local authorities are being flooded with complaints and requests from people; for example, some are doing AI-assisted investigations into accounting "errors".
Science:- I just don't see how the current rate of progress doesn't end up in crazy technologies like perfectly simulated human cells, organs and bodies to the point where we can run experiments virtually and solve all diseases in the next few years. This is happening. Perfect weather predictions far into the future, likewise with earthquakes, etc. Solar panel research explosion resulting in huge efficiency gains, to the point where people no longer need to plug their EV in - car surfaces will be covered in solar panels, as will our windows and roofs. Connecting new homes to the grid will be optional - the same way landline phones are no longer a thing.
I just find it very difficult not to extrapolate all the above.
We got this dump of mathematical breakthroughs from one small team in one company with access to this technology. What happens when this SOTA model is available (and it will continue getting better and cheaper) to everyone working on hard problems - every university on the planet starts cranking out AI-assisted research breakthroughs.
If there is perfect lie detecting technology I could see all kinds of chaos resulting from it. I can't see it only be applied only to politicians, and I think it would be the developers of the technology who decide the use.
I think were we disagree is that you sort of see AI as an extension of technological progress whereas I see it more like an extension of evolution. I view the process of AI training as functioning in a similar way to evolution in that it build circuits into neural networks similar to how evolution built circuits into human brains.
>> What happens when politicians can no longer lie without being caught out live on air?
A 5-second delay on a politician's presser. Any lies will be muted in real time and the actual facts presented onscreen. Continue to lie enough, and the politician gets unstreamed.
It's only true if you believe that "putting the burden of living on a dying planet on the future generations" counts as "navigating".
So basically I think that the future is getting pretty weird because we are building really powerful tools, though these tools are precisely what allows us to prosper in that future.
This is good and admirable, but it'd really suck if by trying to build god without knowing how we end the human species. We could simply wait some more decades until we actually have any idea what we're doing, and then do that without the risk.
Software development for example as a task is done. And AI is continuesly reducing the price of more and more tasks every day.
This math breakthrough also shifts something significant: Its now a lot clearer that investment means money into energy to run AI.
Money + Energy = progress
I don't see it plateuing at all. We know how to progress. We broke through a wall we hit. Like the system wasn't able to optimize/automate everything because the tools were not there. It was still cheaper and easier to hire people for a LOT of things.
Now AI fills this gap.
You will see the commodification of everything in the next 15 years. High complex tasks? commodity. Physical labor? commodity.
There will be a lot of job loss unquestionably, in the same way that automation reduced manufacturing jobs and farm payrolls.
At the same time we have to put what AI can do in perspective.
Intelligence is a broad grouping that includes concepts such as knowledge, skill, experience, and wisdom.
AI has incredible knowledge and in many areas approximates experience and wisdom.
But wisdom is harder to formalize than knowledge and skill.
For example certifying a college education relies mostly on the ease with which we can verify/test knowledge.
To some extent advanced degrees try to certify maybe wisdom and experience.
In my very personal opinion, wisdom and life experience should give humans an edge for a while to come.
Additionally, I do feel that the more an individual lacks better than average wisdom and experience, the harder it will be for that person to compete with AI.
Also, on the bright side, the average human will continue to prefer to interact with a fellow human in many spheres. That will also act as an upper bound on AI and robots taking every job.
Either way, I do think this transition will be painful. I don't feel it has to be apocalyptic.
But the world has been an especially volatile place over the last 10 years.
So when you add that existing volatility, to the upheaval from the AI transition, it would not surprise me if the transition results in violence.
But, without the pre-existing volatility, and if humans were capable of generosity and love at scale, I see no reason AI can not be absorbed into society with net gain.
I guess to summarize, I feel this tech should be a net gain and to the extent that it isn't, it will be because of flaws deep inside of humanity itself, not because it had to end in chaos.
In other words, I feel fear, greed, anxiety, and competition -- all our base instincts coming from all sides, will be what determine the end result of AI moreso than AI taking everyone's job.
I have a hard time taking statements from OpenAI about their own product, that they are trying to sell to people and make money, seriously. I take these statements as they are greatly exaggerated or even straight up lies and propaganda.
That said, I think results like these are mostly annoying more then anything. They spent a lot of money, used up gigartiuan amount of compute, to ruin a puzzle that mathematicians were tackling. I am mostly unsurprised that if you spend a trillion times the energy that a team of mathematicians would, that you get maybe 1.5 times the results. I see a future where that 1.5 times the results may go to 3x but not much more. And if that, then I will be more annoyed.
Maybe people will find some clever way to expand this domain of AI-solvable problems by a couple of more categories, or (more likely) find a clever way of using applying these verifier for problems that was previously not viable, thus changing the solution to “just spend more energy computing dummy”. However I think this too will have its limits.
Regardless, this is still annoying and I want them to stop doing this. Solving math problems should not be relegated to whoever has the most money to spend the most compute.
Yes, human beings can do more now in some ways but...to put it poetically, I think there will be no more heroes like Einstein and Newton of the past. Now it will just be someone cleverly turning the crank.
Yes, we still admire Usain Bolt even though we have cars...but maybe the admiration is a lot more trivial than if we did not have them....
Personally, I think AI is a grand mistake.
This is false… there’s lots of ingenuity to be had and demonstrated. But it’ll only get recognised if it makes a material contribution to the economy imo. Otherwise yes it’ll be seen as meh - but that’s already happening.
People like Einstein were revered in society. The average person cannot name a leading scientist etc today.
When Jane Goodall died last year it was international news. She was a celebrity scientist for sure, I think she even made an appearance in The Simpsons. Ditto Stephen Hawking.
International news doesn’t mean much - the vast majority of people don’t consume news the way you think - I highly doubt the vast majority had any awareness.