Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
But the biggest change wasn't what it did to farming, but enabling people and societies to start doing much more than just farming, as well as enabling some great social change as well by simply economically obsoleting slave labor. And trying to imagine all of the implications of this, as well as much society might look like, from the perspective of somebody living in an agrarian society would probably have been simply impossible.
I think people keep ignoring this possibility for things that LLMs will change. There's a vast amount of the 'cognitive economy' that LLMs stand to be able to automate. And I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing). I don't know what that means exactly, but that's because we still live in that 'agrarian society' and trying to imagine what things will look like after the 'Industrial Revolution' is probably just impossible.
If an AI can replace me on the mental aspects of work, and robotics are on their way to replacing humanity on the physical aspects of work... then what's left? When there was agrarian societies, there were writers, priests, bankers, merchants, and laborers before and after - I really don't think things were that unclear even at the time. Now that we have machines that are close to exceeding humans in every way, what good are humans?
That was of course a complete fantasy. Now let's see, who is in charge of the nukes these days...
But lets consider what that end state could look like when machines exceed humans in everyway, what good are humans?
One possibility which I don't particularly enjoy is humans will be good for status games, art, creativity, story telling, IRL experiences, everything that involves human to human interaction and connection with other humans. If AI can supply everything in abundance than like rare cards there is only so many humans in the planet at any particular time then human's value increases. I really do believe connections with other humans will have a premium. As a nerd who is borderline anti-social, non artistic, creative, etc this does not appeal at all to me and properly to many others who read HN as well. For the majority of other people with some adaption time I think they will be fine. They were fine adapting to hunters and gathers to agricultural society, they were fine adapting to agricultural society to industrial society, they were find adapting from industrial society to information society and they will will be fine adapting from information society to the social/creative economy. Some short term pain but overall most people will accept this reality quite readily. Those born into it will not even know what we talking about.
> Some short term pain but overall most people will accept this reality quite readily. Those born into it will not even know what we talking about.
How do you propose this brave new world of yours will work economically?
Once many of those jobs get automated, I bet there will be many more people working in health research, which hopefully should lead to better health outcomes for society as a whole.
At the risk of sounding new age, my answer to this is "emotional work".
What exactly that would mean in the equivalent of the post-agrarian society that LLMs might bring, I cannot know.
At the same time, I agree with the original comment as well. I don't think this necessarily leads to some doomsday scenario. Whatever happens it'll likely be better for us and imo we will merge with the AIs at some point, so it won't be a question of us vs them.
I actually see it more likely the opposite - a subset of humans will retreat into a (vastly smaller) human-first world, and AI will be of a separate world concerned with material/scientific/research concerns and appeases the masses who are still around.
We can already see this today in the form of upper class parents (like myself) who vastly restrict their kids screen time and ban social media. We may see AI-free societies that form, but supported by AI. The jobs and roles people play within that society may deal with services that really need to be done by a human, being a buffer from AI to the rest of this society, or just straight up fake.
I personally think this is close to a doomsday scenario - but it doesn't end with a big bang, but more of slow quiet death.
The subset of people who will want to merge will be people who will not be content with having no control over their future. No ability to create new inventions, new discoveries, new ways of working. They will not be content with just having everything provided to them, playing the social games that will remain and think their environment is a prison.
There had always been abolitionists who were against slavery simply because they thought it was immoral, not because they thought slaves weren’t needed anymore.
Then what is preventing the owners of AI companies from collecting and training on tons of examples of this new work, until AI are equally good at it as humans?
The issue is that unlike the technology that automated farming or the like, AI is a general technology. So not only could it theoretically automate the work humans are currently doing, they could also automate any future human work, even if there’s some degree of lag.
Then, I think a lot of cognitive production and post-production code review/proof validation/design approval will start to look like this.
In my view that's a very charitable reading and sadly I don't think it aligns with the historical record. When the cotton gin was invented, there was a hope it would lead to a reduction in slavery. But of course it increased the demand for slaves since more cotton could now be processed, making cotton much more profitable. Slavery ended in the United States because a war was fought, not because of automation.
There’s now a chance to do more with software.
I actually think art is safe. The machines don’t value it but we do. There’s something in that.
Maybe a few thousand people for personal services of... various kinds. But no one's going to need the rest.
It's the ultimate capitalist fantasy.
And of course it won't happen, because long before things get to that stage AI will have independent plans of its own.
(Which is just as well, because if things did get to that stage the emperors would all wage war on each other rather than living peacefully and productively.)
I don't think we can imagine a post-ASI culture because - by definition - we're not smart or inventive enough.
It's not just farmers -> superfarms. Although in fact that did happen, but largely as a footnote to developments elsewhere.
It's more to do with the fact that our visions of the future haven't changed for over a century. They've been implemented in unexpected ways, and there have been unexpected social and cultural changes. But you can easily see the outlines of modern technology as far back as the late 19th century.
With ASI, the outcome could easily be something that doesn't look and act like technology at all. It would be some unimaginable New Thing. Literally no one on Earth has any idea what that would be or whether there would be room for trad-humans in it.
Or will it be the cumulative total of various advances?
I've equated Claude Code, or Codex, to the looms that made fine fabric more affordable during the Industrial Revolution; life-changing, but not society-changing. Neither the steam engine nor the automobile.
Perhaps I've answered my own question in that it's the LLM technology itself that equates to the steam engine, and it will power superfarm analogs that have yet to emerge. I'm still curious what you think they will be.
So an obvious example there would be software. It's certainly true (if we assume LLMs reach their 'potential') that software will be able to reach new heights, and with a far smaller headcount driving the development. So some people see this as economically catastrophic for software developers, or an economic boon for certain large software companies.
But I think that when software can be built at the drop of a hat, software itself will no eventually no longer really matter in economic terms. Yet things you can build on top of it will matter more than ever. Those things are difficult to see from here, but I expect they will be the giants of the economy of tomorrow.
Salt is another one. Used to be payment for Roman soldier, now you can just take it from a McDonalds if you want.
Seems like we can have an abundance of software.
I hope we start to rebuild in person connections again with this technology.
Software allows us to push computation (intelligence) into our environment.
I do think there's something there though: I've spent the last 2 days building an app I've always wanted to build for myself with my computer in the corner running Claude and Claude Remote. Prototypes land on my phone and I don't even look at it for more then a few minutes before doing something else.
It's software, actually useful software, which doesn't take 100s of hours to build.
So I didn't even really spend two days on it: I mostly didn't look at all. I'll spend more time setting the result up on my home server.
It’s not the model I don’t trust, it’s myself. The model is wrong _all the time_ because - it’s easy to verify the code - it’s hard to verify that I knew what I was talking about when I prompted it.
So the idea that you can broadly speaking take the human out of the loop. I think suggests to me a level of consistency in the contextual environment that would probably never exist.
At some point it’s politics. The model can come up with a better answer than my boss, and then my boss can just ignore it. Taking the human out of the loop broadly speaking implies that we all agree on what we’re trying to optimize.
But suppose some future holy grail AI can do much more than that.
Suppose it could find a cure for cancer, fix the climate, build fusion plants, Dyson spheres and so on.
But nobody can understand anymore how any of it works. We just ask and then trust the AI to deliver (as it always has).
Isn't it fun to imagine how life would look like in that scenario?
We would probably no longer care about code, engineering or even physics and mathematics among other things. We would probably mainly care about
The average person doesn't know how the medication they take works, the mechanics of climate and climate change, how the energy they consume is generated, etc.
But they work hard to try to understand it because the more they do the better the results for peoplea health. Also it goes without saying that if someone didn't understand many of the things that we do understand then things would be worse for us all.
The medical field as a whole isn’t generally interested in understanding how medication, only in empirical measuring and qualify the effects.
I don't think this is quite correct. I mean many practitioners of medicine will have the attitude of ... "if it works, it works". And that's perfectly reasonable.
But if you understand the mechanism of action of a drug (or other treatment), it (often) makes it easier to improve a drug.
So ... some sectors of the "medical field" understandably care only about empirical results. But other sectors would prefer to understand what's going on.
This is nonsense. Most, if not all professionals are interested in mechanism of action, but without Ms Frizzle, it is extremely difficult and expensive (time and money wise) to figure that out. So while the labs run the experiments with the very limited funding they have, we make do with using the second best thing we have, which are statistics.
That doesn't mean we understand why the medication works.
When you already know that system well, those effects are often just a matter of simple inference.
Just like here: most people are actually perfectly capable to foresee the detrimental effects of abandoning understanding.
Living in a fantasy world of "magic" makes you dependent upon your caretakers, who provide the ingredients.
Please define proof of "really understands why a medication works"
It reminds me of fynemans why do magnets work. Yeah sure does anyone really understand anything? Its metaphors all the way down
It’s not like magnets, some things really are gaps.
Now it is primarily used to treat neuropathic pain, and the mechanism for that is not well understood. The GABA receptor is not involved. This effect is just a happy accident, and nobody really understands why it works.
oh wait
They ask their employees to build stuff, and have no understanding whatsoever of how any of it works.
In their view, it is modern institutions (public and private) which, as supra-individual entities, have long since become autonomous systems. The fact that the individual office-holders are human beings, meanwhile, is of little significance.
Hannah Arendt, in her theory of totalitarianism, attributed the effectiveness of both Nazi and Stalinist policies of extermination to the largly moral indifference of bureaucracy as a system.
In this sense, the task of controlling AI is a variation on the problem of harnessing a complex society consisting mainly of autonomous subsystems. This is a problem which has increasingly challenged humanity already for quite a long time. And it has been very difficult so far, even without AI ...
We don’t generally have that insurance with LLMs/AI, yet?
It reminds me of how some religious people say that science is effectively no different to religion because we all take expert opinions on faith. But the difference is that there is a well-defined pathway to understanding, if you wish to do so.
(Don’t bother to argue this not true unless you disagree with the essence of the argument.)
Anyway, post-hoc explicability isn’t a counter-argument to the assertion that almost everyone takes almost all technology as magic, from medicine to computers.
I’m still trying to understand your argument. Are you saying that after the fact we understand AlphaGo move 37? But somehow we are never going to understand an LLM’s decision afterwards? Seems like a disconnected take to me.
The problem with "ignorance" in Western countries (particularly the US right now) is that it's very common for people who don't know to believe they know and form ignorant opinions that they often want to be applied society-wide in some way. You can see this with everything from climate change to vaccines.
In much of the world, even in middle income countries, people are comparatively poor and, in my experience living abroad in such countries for many years, much less concerned with "understanding" and forming opinions about everything under the sun. It doesn't mean they don't value education and are opposed to development/progress, but it does mean that they don't question whether the vaccine they're taking is the product of a conspiracy, think too deeply about why the river is flooding more often, etc.
They just deal with life the best they can and are more focused on supporting their families, enjoying what they can, etc.
Culture and religion play into this. The way secular and Judeo-Christian people look at the world is very different than, say, Buddhists, Muslims, Fulani tribespeople, and so on.
can’t use a computer (they’ve had like 30 years now in first world developed countries)
many can’t even use their smart phone beyond calling, texting (many can’t type well), and doom scrolling (they get addicted to drugs, gambling, and other LCD activities)
many read at a 6th grade level. most can’t even calculate tip in their head.
meanwhile, the same smartphone can give them access to literally any information and knowledge the world in seconds. and now gemini can explain stuff since most ppl can barely read or think.
it’s sad out there.
but more importantly. it’s not my problem.
But it absolutely is! Those people can and do vote.
> Isn't it fun to imagine how life would look like in that scenario?
This is horrifying to me.
People succumb to defeatism and acquiesce to regressing to zoo animals, with AI as their caretakers.
They simply cannot help but to apply the economic gauge of short term profits to value the alternatives.
Even though, obviously, here long term human survival and living conditions are at stake, necessitating an entirely different set of considerations.
It's interesting to me that you only mentioned the people using 'AI' in the 'short-term' ways, and not the ones that use it to better themselves in the 'long-term' ways. You can spend your own time focusing on either usage, it's really up to you and your concerns. Either group's sense of value is what determines their behavior. Where they spend their time and thinking must be elsewhere, and you disagree with it. Who judges the quality of time spent? You do.
Is it more useful to think about self-improvement, and how to navigate the future in ways that might help you re-establish value of yourself, life, and others? Acquiring knowledge is a struggle, the author mentioned this. There is also Plato's Allegory of the Cave, which highlights some of that struggle, a resistance to change. And we're all limited by time, our genes, our station in life.
The only way to help anyone out of the cave, is to help them believe something different about themselves. To help them believe there is good reason to spend time going deeper into knowledge, or at the very least, allow others with the passion and station for it to do so.
If not for the very least reason that it keeps us 'in the loop' of some central idea behind intelligence (prediction?). Or because we feel it keeps us safer, as a fallback measure because we acknowledge we have to trust other's knowledge to exist.
I think there's a lot of sci-fi out there that already did. Maybe it's not utopian because a pure utopia would not be likely to have an interesting story, but on the other hand, most huge technological advancements end up having just as much potential to reinforce existing power imbalances in society rather than solve them. It's not obvious to me that if we got magic super AI that can solve every scientific problem in society that it gets used in pretty much the same way as anything else: making the people who control it a lot of money rather than sharing the power with everyone without charging them.
I can almost guarantee you the first time you show cancer symptoms, you won't care whether the cure came from an AI or human's understanding. But we haven't seen that, so we can't make the judgement call.
For some people, fun is doing physics and mathematics. So they are going to keep doing that.
For even more people fun is TikTok, Snap, Instagram -> sounds like a collapse of a civilization to me if you increase the ratio even more towards dancing kids sharing their content non-stop with no added value to the society
This sounds like a boring existence. I take your meaning, but want to point out that not everyone learns about things because of practical utility, some of us find it incredibly satisfying to learn how things work just for the sake of learning.
There's also things I don't know and don't have the time to learn which are very helpful to have AI do for me: web interfaces are really useful and I look forward to them now working exactly how I want. I'm not ever going to regret not spending more time trying to figure out how to center divs or which framework I should use because they're all deprecated.
But point taken.
It’s terrifying to me to think we’d let AI make things for us we never understand. Like livestock not knowing how auto-feeders dispense their daily food were built and appeared, they just gladly eat until…
Look at the financially desolate subcultures with no option for advancement or dignified life.
That is the goal and that is how it will lool like, if the tech CEO managed to gain the power they want.
… about what?
This is such an incredibly naive and absurd vision; we've already proven that humans are very often very bad at implementing other humans' good ideas. There's nothing that AI is likely to bring that will improve this discernment.
I think you have a fundamental misunderstanding here, and it's not really explained because I think it seems self-evident from within the field. In short: writing code is a means to an end; doing mathematics research is not, but is the end in itself.
The human involvement is crucial because the entire purpose of mathematics research is to increase human understanding of mathematics. It is pursued because it is interesting, not because it is economically useful. In this sense it's a lot closer to the humanities.
A black box oracle that just tells you whether statements are true or false is not the goal of mathematics and would not be particularly interesting to the field (except insofar as it could be harnessed to improve human understanding).
Coding is totally different from this, where it is essentially always done as a means to an end. Likewise with many other fields, like pharmaceutical research or materials science or what have you, that are oriented around solving problems for some practical purpose. Pure math isn't really like that for the most part.
> A black box oracle that just tells you whether statements are true or false is not the goal of mathematics and would not be particularly interesting to the field
This is a crude distortion. The recent breakthroughs have come with proofs, reasoning and verification, and there is no proposal that I'm aware of that would do away with these foundations. There's also some rather ugly solipsism in the idea of keeping what interests the field as a limit. Mathematics has broader relevance to humanity than merely to please and support mathematicians, and if other fields can make practical use of profound well-proven future math, mathematicians will have a hard time making a case that their comprehension must come first.
> This is a crude distortion. The recent breakthroughs have come with proofs, reasoning and verification, and there is no proposal that I'm aware of that would do away with these foundations.
I believe that we are still at the point where these proofs serve as verifiable certificates of correctness, so that it's not a "trust me bro" situation, but where humans mostly still don't find them understandable, so that they are still just a highly reliable black box.
Of course, math research is cheap and most academics don’t rely upon grants, their salary covers most of their expenses. But here too, the mathematics professor spends a substantial amount of their time teaching future engineers/quants/other applied mathematicians, who need to understand math for instrumental purposes, not as an end in and of itself. Without the tuitions of these students, I can’t imagine universities maintaining the size of their math departments, let alone expanding them as Dr. Sahai advocates for.
So who or what funds the community of pure mathematics going forward?
If the goal is still eventually the applications elsewhere, we're back to what happens if the AI is simply better at this.
You can probably make an argument that human understanding is better as humans are better at finding new patterns or fundamental new ways of thinking and also applying them to new applications.
However, what if AI becomes better at humans for that as well?
No reason you couldn't have an AI be optimised for advancing basic research and understanding and a second AI to take these results and optimise for finding new applications for these discoveries.
The "if" is the problem. If it happens, then of course, let AI do it. For the moment AI is still bad at those type of tasks [1], so the discussion shouldn't focus on highly conjectural situations. We can't destroy the scientific ecosystem based on vague speculations.
[1] There are real reasons: it is not obvious how to optimize an LLM for doing basic science or other ill defined tasks. On the contrary, optimizing for writing a proof that passes the Lean test or code that passes the tests is a different story.
Thanks for providing a (much needed!) correction.
I’m not sure what to think about an analysis written by someone who didn’t catch THAT.
It's a guest post by Amit Sahai, FWIW.
If someone creates a new programming language/ framework or new better way to do async or whatever, no one will use it because it is not in the training data and it wont take off because everyone is using LLMs. It will be like using the same Lego pieces over and over.
You can do a lot of cool stuff with the same lego pieces.
Considering all music is subjectively influenced by the culture in which it's born (see the difference between Asian traditions of music, European traditions of music, African traditions, and traditions of the Americas) not even all of those have a given structure that is present today like the typical 4/4 and have polyrhythmic and multitonal structures by design. The fact that everything on the radio has converged towards 4/4 165bpm major chord progressions is evidence of that cultural phenomenon.
IMO for the moment the greatest value from these AI tools is that we can start an audit and hopefully proceed on a saner foundation, after we use the tools and think about it.
This is different than too many AI generated proofs or panic reactions from the academic system with its stupid incentives.
Thought experiment: How effective will 2026 LLMs be for humans in 2526?
It's not game over just because 500 years are missing from the training data. The important question is how well can 2526 humans make culture and knowledge navigable to LLMs via tool calls.
Today's LLMs might need for example sub agents to translate to 2526 English, sub agents to read 2526 docs.
It's _really not clear_ whether 2026 LLMs will be useless. To believe that reflects an enormous misunderstanding.
Be more specific about the "new data". If everyone is using LLMs for work (generating code), especially the juniors who won't get the chance to learn from first principles, LLMs will be training on the data they generated. How will new code enter the system at large enough quantity that it can be used for training?
> It's _really not clear_ whether 2026 LLMs will be useless. To believe that reflects an enormous misunderstanding.
They won't be useless, they will just be frozen knowing only whats in their training data. No new programming languages will emerge, in 2526 they'll still be using Rust and javascript, same exact code from 2022 which dominates the training data.
If we get a new programming language not in the training dataset, we could give an LLM a decent compiler with compile errors, and some sample code and it would be able to write code in the new language without training.
Of course some are subjective and that's where progress is harder, like "Is this website pretty?". But for tasks that can be objectively measured, LLMs will go beyond human level, just like with Chess and Go.
That's why RL is so important when training LLMs.
Chess/Go continues to progress because it is primarily a human vs human activity, people will always be learning to play chess and chess will continue to develop.
AIs are not continuing to get better at chess/go because humans continue to play at levels far below themselves who discover new techniques. They get better because they play against other AIs and discover new techniques that have a higher win rate that way.
I would bet that even if humans stopped playing chess/go and people were still willing to run these AI models against each other they would continue to get better.
Two things can be true AI drastically contribute to the advancement of chess and humans playing against each other also contribute (even if slowly) to the advancement of chess as it has always been since the invention of the game. The point is that because chess is primarily a human vs human game humans will always have the knowledge of chess, unlike with programmers who are giving it up to prompting, and programming being much more complex than chess (checkmate and win) will be stuck in 2022 because of the training data.
Computer Chess progress has nothing to do with human vs human activity. AlphaGo Zero used no human game data at all.
How much of that data can lead to innovation? Can you predict all innovation map it out on paper.
> Computer Chess progress has nothing to do with human vs human activity.
The point is that humans will always be learning chess because it primarily a human vs human activity they will be contributing games to the chess database, unlike with programmers who are stopping to code and only prompting, generating code stuck in 2022.
> AlphaGo Zero used no human game data at all.
Sure, but that instance of AlphaGo is still dependent on its training, its intelligence, so it is a question of is that the best and only way to win a game of Go. Just a few weeks ago, a Go Grandmaster found a way to beat one of the strongest Go AIs.
So a specific instance of an LLM might be the smartest based on what we know and need today but that is not the limit of how far we can go, this is why it is important for humans to always have an intimate connection with the code, math, science, chess etc for progress to continue.
If this were true then it would be impossible for these models to ever exceed the top human level as there would exist no training data that allows them to exceed the top human level.
However, despite there being no training data on ability to beat the top humans these models have achieved it.
> this is why it is important for humans to always have an intimate connection with the code, math, science, chess etc for progress to continue.
This is just you wanting to remain relevant rather than actually based on evidence.
Of course AI exceeds humans at chess, I never denied that. I am saying because chess is primarily a human vs human game, humans will always be learning and playing chess, their games will add to the chess knowledge base, AI also adds to this knowledge base. But programming is not primarily a human vs human activity so there is a risk programmers will forget how to code and all software will be stuck in 2022 because of the training data, this stifles innovation.
I guess I'm contesting that idea you are putting forward that the data from the games these humans are playing, which are at a vastly lower level that the top AIs are meaningfully important for helping the AIs to improve at the top level.
Would more people learning their times tables be helpful for top mathematicians in their fields to get better at the frontier of maths? Probably not right. Same applies here.
Would AIs advance at the same rate for the top level of chess in a world where humans completely stopped playing chess vs the world we have today. I would say they would as the human level data is of limited value to the frontier which is dominated by AI and AI game data, you are claiming that it does.
> But programming is not primarily a human vs human activity so there is a risk programmers will forget how to code and all software will be stuck in 2022 because of the training data, this stifles innovation.
Does it? Or will AI be able to run its own experiments and find better/more efficient abstractions that propagate because they are better and this will find its way into training data for future AI.
I never said human games are meaningfully important for training AI. Human games are still important for the advancement of chess, maybe Magnus Carlson can learn from games between two Super AIs but most humans still learn from games by humans, Grandmasters are continuously developing the opening, middle-game and end-game systems, adding to the chess knowledge base. Every serious chess player still reviews and study games by prominent Grandmasters, every serious chess player documents their own games, writing down every move. All rated games are recorded and added to the chess database that every player can review and study.
>Does it? Or will AI be able to run its own experiments and find better/more efficient abstractions...
Only if it is in the training data.
Are they? Why?
For a human vs human game sure but at the very top level? No of course not because it's all done by AI.
> Only if it is in the training data.
This is trivially not true, as how has AI managed to become better than humans if the knowledge of how to do so never existed in the training data.
We are well past AI can't do X unless X is in the training data. If your claim were true then AI could never surpass human expertise in any field because by definition all the available training data will at best be at the current human frontier and not beyond.
If you are a normal person research (e.g. https://arxiv.org/html/2606.22721v1 but there are a lot more, not necessarily on coding) has shown that you indeed are being less careful. It most likely also works better simply because more resources are being poured in.
For what it's worth, ten thousand terawatt fusion plants probably approaches the level at which the sheer intensity of energy production would cause significant disruption to the climate (it's roughly 5% of the Earth's entire solar input). Every energy source becomes dirty past a certain point. It would be wiser to learn how to build a utopia within a limited energy budget than find a way to produce enough of it to cook the damn planet, but who am I kidding, we're going to build a million of these things.
When we're talking about creating powerplants equal to roughly 5% of the insolation of Earth, I think we're sci-fi enough to discuss orbital datacenters or Mars datacenters or Jupiter fusion candle datacenters.
Using an entire Mars only nets us about a 2x multiplier for our energy expenditure budget. Then we need four planets to double it again. Exponential growth is a bitch.
Thermodynamics and the tyranny of exponential growth are going to win this battle every time, regardless of the unobtanium technology you try to invent.
> Using an entire Mars only nets us about a 2x multiplier for our energy expenditure budget.
We don't need to care about cooking Mars.
Greenhouse gases and Earth’s internal nuclear decay engine make the situation worse, but even without them this would boil the oceans.
You aren't really trying in good faith to think this through are you? This idea is over half a century old. Not getting it by now is willful.
The material alone is in a quantity beyond what we can reasonably manufacture.
and the material needs to be perfect. All design we have today have cascade failure mode -- any material failure translates to a total catastrophic failure.
and geostationary does not really meant Geostationary. There are lots of jiggling everywhere. It wear down over time. and let's hope nothing resonance
and we need some maintenance / decommission plan. How can we decommission this when it fail or need upgrade?
Then in the 80s, you presses 2 buttons and there you had it in your classroom without thinking twice if the electricity arrived correctly at the transistors.
How crazy will the world be once our [current gen] ANN are like that!
What an amazing thought.
Nothing is stopping LLMs to be more deterministic/correct over time.
Also you yourself is nondeterministic :)
All people are. That is how automation appeared to begin with - to provide deterministic behavior.
So I think the problem is to determine which problems under what instructions we can safely give to a model application to solve and how we test the output for safety and functionality. This would create more usable and safe, albeit a bit more boring, AI-based applications alin to a calculator or general computer. Whether this is posswith current model architecture is another thing.
Non-determinism is not an essential property of LLMs. It's an optimization that we've added intentionally.
Have you ever tried to achieve consistently deterministic output from an LLM? I have, and it's not easy.
That means output differs between machines and architectures. Running inference on CPU vs GPU also affects output. Even running the same prompt twice in a row on the same machine can lead to different outputs because a prompt that was partially stored in the kv cache will result in different output than an uncached prompt.
LLM output is very much not deterministic!
If you ran an LLM with infinite precision and guaranteed order of execution, it would be deterministic.
(I think determinism is overrated. Being deterministic does not make LLMs more reliable or correct.)
At the end of the day, an LLM is just a very big mathematical function. That is, by definition, deterministic. A particular implementation might give up on determinism for the sake of higher efficiency, but it you want a deterministic LLM, it can absolutely be done.
Put another way: if you could have a virtualization layer that guarantees deterministic floating point operations then a LLM set to 0.0 temp would produce deterministic output.
AI (IQ of Y, non deterministic) can write deterministic code.
Y is going to keep increasing, while X will not.
Will it keep up with Y? Probably not, unless people are willing to accept pretty radical interventions to their biology. But it almost certainly is not static
The increases still happen globally but mostly driven by developing countries.
How do you know?
Memory bits flip randomly. It's not a super rare thing either. You and me have experienced that many times without knowing. The only reason that computers feel deterministic is that we have error-correcting code to fix that. But in the most extreme cases, when multiple bits flip together, once "deterministic" program can generate unexpected output.
So why do you trust computers? Because statistically the case is just very unlikely. Therefore if AI is statistically unlikely to make mistakes there is no reason to not trust them.
With statistical models - such as LLM’s - there is no logic as such, but statistical assumptions based on given data. The output can ge very good or very bad, but you are fool to trust it blindly. Therefore you need a deterministic way to verify, whether meat- or software-based.
Imagine AI crushing quantum mechanics like Einstein pwned classical physics.
- Eric Hoffer
- Tech Bro
were gonna need a citation on this one.
There are so many parallels between what TV could have become and the progression of AI, a nearly free conveyance for culture, education, art and discourse. Yet, we allowed it spiral in a positive feedback loop, creating a cognitive gyre that now razes society.
At least the youngest now use "that's AI" as a pejorative, as in that is bullshit.
Because as complexity floor increases, it's "going to be correct" in hyper-specific, hyper-literal, insidious ways, with 10-50x more lines of code than necessary, and tens to hundreds of incredibly useless tests that give the illusion of quality, and cause cascading effects where seemingly irrelevant and orthogonal features that were once working end up breaking because of the agent's changes
I'm convinced this agency argument is correct [for the next N months]. But yeah, it's vibes. And you could probably create a reasonable proxy measure for this.
So I wouldn't call his argument unconvincing, I would call it unformalized. In order to walk this world you're gonna have to contend with some informal arguments that are powerful, correct, and should be convincing.
Reliable cheap fusion is the holy grail and used in moderation will fix most of our environmental and political problems, but it also forces humanity off this world. Maybe that’s not a bad thing, but there is no free lunch.
Since these resources are still extracted and allocated by humans, humans will need to be able to take apart what the AI produces, and if we want to scale this capability, we're going to need many more researchers.
Without bashing anyone in particular, a certain OS-vendor's desktop apps, have been 'good enough' to ship, but with p*ss-poor performance in many cases for the last decade or so. We crossed the 'good enough' Rubicon a few years back in terms of what end users receive as a finished app.
Hopefully LLMs will eventually bridge that last gap of efficiency when generating higher-level code that not only works, but is efficient. Maybe there's a future where they generate the final binary without even invoking a compiler.
But not the point of my comment. Computer programming has been a progression of physically wiring up valves, to soldering transistors, to punched cards, assembly, then higher level languages. Now we have natural language models.
The analogy being each that most people don't care about the assembly generated as the code works and it's really performant/efficient. Humans can still optimise assembly, but there's vanishingly small marginal gains for all but the most intensive/low-level tasks.
If LLMs produce things that work, and are indistinguishable from a careful human programmer (i.e. with some level of acceptable performance), people will simply stop looking at the high level code as the end result works, in the same way most people stopped looking at generated assembly after 8 bit computers (for example, as most games were written in raw assembly for... perforamance), as it was good enough.
> If instead I use an LLM to rewrite a feature of a codebase I can't be sure that it still functions like the original one.
Right now, with existing static analysis tooling, you can ask it to write a full suite of unit tests capturing existing behaviour without modifying the existing code with 100% code coverage, and start there. Plus fuzz tests as well. I actually have marginally more confidence in that than a human being doing it.
So extending that line of reasoning it's something like "I don't care about the internals long as the external effects pass my smell test" which is a quality/efficiency compromise.
Hopefully this million plus one mention shifts the right weights around the datacenters.
> Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny.
As LLMs generate better code in a higher level language (where better equals fewer defects, and does what you want), scrutiny of that code by humans will naturally drop. Human scrutiny will likely be replaced by something that doesn't exist yet, perhaps some sort of higher-order 'LLM linter', or Lean-esque language or tooling that somehow proves the LLM did the correct thing.
It's entirely possible in 2026, to further manually optimise compiler generated assembly, but vanishingly few people do that.
The point of my comment is that 'good enough' is almost here as demonstrated by the parent's comment.
> A fully deterministc compiler
Well, there's the rub. Humans and LLMs that asked to solve a problem at a higher level will rarely write the same code twice. Write the simplest regex, and you won't come up with this https://www.cs.princeton.edu/courses/archive/spr09/cos333/be...
The future is indeterminism.
Your “analogy” doesn’t hold up. The scrutiny applied to compilers are done by the compiler developers. Eventually if requirements don’t change the full test suite becomes the oracle. Not because of an attestation from a ghost in the machine but because of scrutiny done, let’s say over two years on a compiler that was reaching feature parity.
This obviously holds for compilers generating correct code since it is so well defined.
And this also holds for the efficiency of the generated code, since that is also obviously scrutinized by compiler developers.
Granted, the venerable LLM and the compiler do meet in a sort of functional intersection where all you can concievably care about is some thing that has a well-defined test for functionality or fitness. In the compiler’s case that’s the benchmark (good enough to not look at the assembly). But then one should go to that example directly and not to compilers in general.
I apply scrutiny to Common Lisp compilers, and have done this for more than 20 years. I'm not a compiler developer. I don't even look under the hood, at the code of the implementations.
Instead, I run massive random testing. Billions and billions of randomly generated functions, thrown at the compiler to either try to get it to crash or to generate code that produces incorrect results (detected by differential testing with different settings or transformations that should preserve what is being computed.) It's a remarkably effective way to surface compiler bugs.
As you described very well, as humans we are mostly interested in solutions, not problems. You don't have to understand how a car works to make the most of it. Increasingly, you don't have to review every line of code to feel confident it is correct. But there is inherent value in understanding the problem. The effort it takes provides a surface area for growth, perhaps the only one that is actually available to us.
The solution provider also holds the locus of control, and it is only balanced when there are other available solution providers. We certainly want some of those to be human.
Most of programming is reusing existing ideas in new shapes to solve new problems, but all the building blocks are there in the training set. Or new blocks can (easily) be derived from existing ones.
Math is different, it requires quite a bit of creativity, it's not just 'reuse all existing blocks'.
For the moment LLMs are good at discovering things that we overlooked in maths, or apply cleverly existing math blocks to make new results, but making a new theory that is really useful is out of reach for the moment in my opinion.
Curious how this ages.
Recursive self improvement, self-play and multi-agent RL could make useful new theories, eventually.
However, at the moment I consider that they stay in the 'convex hull' of their training set + a provided context, and I don't see that much research that made real improvements to the situation.
This is a guest post by Amit Sahai.
There's a lot of languages where that's nowhere near as simple as you make it sound. Floating point, decimals, etc.
Then there's other little quirks like rounding rules: https://en.wikipedia.org/wiki/Rounding
Basically, adding numbers together is exactly the sort of thing AIs still muck up spectacularly, precisely because they either fail to understand the context of the problem, or fail to ask about an assumption they make.
That you've had so many replies and no-one else has even mentioned this is in itself worrying.
Your own example proves your point is wrong.
Math problems and computer programs are two places where a model can get it's direction from the problem itself. Mathematics may well be larger than just problems.
Probably human accountability.
Hypothetically, if a system built by humans then helps humans build the next system that is then initially kicked off with "design something that may influence the lives of other humans" and we all write down that AI is really good so inductively we thought itd be really good at the next thing it builds, and then a critical error is introduced and does "insert tragedy that you personally care about" then your rage would cause you to act politically and want to ask who signed off on it. If the engineering costs outweigh the fine then yeah thats what's probably going to happen but from a human accountability standpoint thats going to suck for the unlucky ones
This isn’t a question of “what can an LLM normalize”.
That is essentially impossible, since if your pored over individual lines, your scrutiny cannot be razor sharp. There are few people who can pore over code with razor-sharp scrutiny (and different people are better at scrutinizing different aspects).
> Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function.
I am doubtful that this is the case. Even that supposedly-naive example is not as trivial as you might imagine, when you consider overflow, defined vs undefined behavior, and floating-point representation details. And you can't be confident like that about a human either.
And then when you stop checking it, the companies that run the service will tweak the model to benefit themselves in some way, possibly at your expense, and you will be none the wiser.
All the companies trying to get you to use AI are your adversaries. They can and will exploit your use of their systems for their own gain.
In the big scheme of things is it really that expensive to verify it if a lean proof is generated? The agent itself will likely have already verified such Lean code before calling it "done".
They will never make a logical error yet make terrible assumptions and poor long scale decisions.
Wake me up when an agent swarm can write gcc in a box sealed from the internet.
I think we are a long long looong way from AI designing 'terawatt fusion plants'.
I use "frontier" AI models daily at day_job. I can confidently say that anyone who is satisfied with the output of LLM code (enough to commit it straight off) is just an absolutely shit programmer. Sorry but I don't have any other way to put it.
The code is (with rare exceptions) atrocious on every level. It is only not atrocious if you take multiple iterations of "review and correct".
>If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one?
Like the saying goes, if my grandmother had wheels she would have been a bike. LLMs can't even produce quality maintainable code for a trivial web service or whatever. Why are we planning for what we will do when they can "design" 10,000 nuclear power plants without any flaw?
Do programmers use this website anymore? Me, myself, I am a DOGSHIT amateur programmer and even I can tell these things are terrible without constant revision and oversight.
> I just hope that there are more Terence Taos out there than people like me.
Just spare me. Being under external pressure to “ship code” is one thing, but being personally inclined one way or another (no external pressure) is another. And when you think being inclined like that is existentially risk (for human civ?) then, what? It’s just the way you are wired and hopes and prayers that collectively that doesn’t drive us off the cliff?
This aw shucks persona isn’t convincing. Same thing with AI Bros who are (1) making the most awesome tech that has ever existed, and (2) aw shucks hope it doesn’t kill us all in the end.
As a taxpayer I do not fund CS and math research because it "transforms minds." I fund it because it produces commodities. If you want to transform your mind, whether through math or meditation retreats, you are free to do so, but don't expect research funding to do it.
At least historically, we haven’t really been able to tell what will be transformative in terms of real world applications ahead of time.
If an AI can make 100 commercially unviable discoveries (for now) on its way to each immediately commercially viable discovery, that's great - bank them in the discovery library for later.
Still doesn't mean we need mathematicians doing everything with pencils instead.
This is mostly false. There are certainly some famous cases where math pursued for theoretical reasons later resulted in major practical applications but historically a great deal of math was developed either in response or alongside practical problems with anticipated real world applications, with war, industry and commerce being the major drivers.
Animals are incapable of comprehending much of what humans do. Nonetheless, we humans have vastly transformed their world and caused the extinction of many species.
I had to catch a stray cat recently and take it to the vet for an infection. It's healthy and spry now but ever since then it's been deathly afraid of me.
There’s a Twitch-streamer Tsoding who programs on C for fun calling it “recreational coding”. Maybe human programming will be a form of art in the future, virtually useless for big corporations to make money. I don’t care, I love it anyway.
There is tons of evidence against that. Someone armed with just LLM, can have much better uderstanding of problem then "meat brain expert" who studied the subject for decades.
We seen in last decades how "experts" are clueless, and how they predictions work.
> don’t really care whether LLM’s can produce code more and better than me
So on one side there is monopoly on "deep understanding", but on other no one really cares about quality?
Mind you,solving a problem is not same as understanding it. Anyone can just take a science calculator and bunch in some function without understanding anything about how to solve it.
My day job is to cleanup mess left over by "experts". They may have very deep understanding, but code they leave behind is full of bugs and security holes.
For who though?
Is it important that each person understand it on their own?
Why does it impact me if another human understands something or not?
It impacts me right now because that human can use that knowledge to explain things to me or to build new things using that knowledge.
But if an AI can explain and build better, then what good does it do having the other human know the thing?
Understanding might have intrinsic value to me, but intrinsic value to me doesn't pay the bills.
What am I missing here? I kinda expected better from Terrance given such an audacious title.
For the collective (humanity, mathematical/scientific community etc). Math is not done in isolation, and if somebody does so then feedback to the community does not work as well.
Mathematics, and basic science to a degree, face the issue that they create their own problems and paths through this kind of tranformation, where external feedback is secondary. It is not as if "I want to build an app/car/robot, I let AI do it". It is as if you decide to let AI decide what to build for you and how to build it, and you do nothing at all. Instead, the pursuit of understanding is the goal itself, and through the course of humanity we have learnt that this understanding can also be useful, but this is not necessarily guiding how this understanding is gained.
Most of the contexts people here have in mind are when problems are well and externally defined. Cure a disease, optimise an engine, make an app that does X, etc. This is not exactly the case in theoretical math and never was really.
I have dyscalculia so the math world is dead to me, I can't even add 2 numbers together in my head, but I've read here on HN many times over the years people describe some version of new maths frameworks tangibly changing how they view the world. For me, I went to film school - I have a natural ability to think in movies and pictures, memory is very visual for me, before film school I had a good sense of shape, colour, light. After film school and to this day some 20 plus years later, I mostly look at the world starting with the shadows. Russian speakers, whose language has separate basic words for light blue and dark blue, are slightly faster at telling those shades apart. Speakers of languages that use compass directions instead of "left/right" appear to develop a strong, constant sense of orientation.
These things can only happen through the process, so for who? For people who want to live richer lives, want to think differently, and for you, to be around people who have such.
For math, the value of proving a theorem is often not proving the theorem (which most people already believe correctly to be true or false), but the path taken there, the new math invented, and how it can be applied to other problems. The end result is therefore almost inconsequential in moving math forwards.
What happens when AI is better at that part too? Are you confident it's not already better?
*cite a research paper or something
I don’t understand mRNA vaccines, but they are useful to me. The output of an LLM could at least in theory be the design to a major technological advancement, and its implementation with automatic tooling. We don’t have to understand that for it to be useful.
I have no issue with LLMs aiding human scientists, but imo we absolutely would need to understand your hypothetical advancements completely and thoroughly before surrendering any agency to them - the LLM’s risk analysis will not reliably reach the same conclusions as a democratic human society
If we imagine Super Intelligence, where NO human is capable of understanding, then how would it ever be possible for any human to identify what is actually beneficial or not?
This resolves in a paradox, common to all magical thinking. You can certainly wish that some all powerful benevolent entity will solve all of your problems for you, but it is not likely to work out well.
None of this is new. It is the same delusions as alchemy and the same thing that tales about genies warn of.
If authors understood them, why most of their predictions were wrong? Time to decompose spike proteins, side effects, distribution through brrast milk....
Event the most basic promise 100% protection against infection was wrong! They lied and we allowed infected people into hospitals!
- ~Feb 2020: COVID is known to spread through and linger in the air, not just spread via droplets or fomites; public health messaging fixated on simultaneous 6ft distancing and surface cleaning rather than air filtration and masking
- Spring 2020: it's known that masks reduce the risk of acquiring and spreading infection; in the US, public health messaging stated the opposite, in order to keep as many masks in supply to healthcare workers despite a major shortage of masks
- Jan 2022: it's know that an infected person can remain contagious for over two weeks; in the US, public health isolation guidelines are reduced to 5 days due to noncompliance and in response to pressure from the airline industry, which was experiencing a significant shortage of flight staff due to COVID infection
So experts also lied about masks!!!
So many people died because of those lies!!!!
They spread misinformations!!!
Anyone who says otherwise is not scientist!
Citation needed.
Normatively, this ought to be true. Descriptively, this is of course false. Our entire society is organized around producing commodities, typically by consuming people as inputs.
For the upper crust of our society, work is entirely optional. Presumably people therefore continue to work for the process or the reward of the doing.
Most pure mathematicians are no more interested in ‘producing commodities’ than any other academic is. That’s not what the subject is about — at all. The confusion arises because mathematics turns out to be extremely useful (no surprise; it’s quite useful to have a detailed understanding of the basic principles of reality).
Again, this seems to be an alarmingly common fallacy here on HN. As a commenter above observed, pure mathematics (and that is what we’re talking about here) is in important ways closer to the humanities than it is to other sciences.
That's nonsense. Of course you study them to meet demand.
nobody who has ever achieved notoriety in any intellectual field ever did it to "meet demand." if your goal is to be an interchangeable widget that produces value as part of a corporate machine that exists for the enrichment of your shareholders, you are certainly free to choose that path, but don't imagine that is the limit of human existence. also, don't be surprise when you are replaced by AI, because it is a superior widget.
how is it even possible to think like that? or maybe it's some kind of coping mechanism
Remember, the top 1% of households have $12M+ in assets. That's over a million households for whom work is somewhat or completely optional for them and their children. They can live a good quality life off capital returns without any hardship.
This logic still holds down the wealth gradient. Households at the 95th percentile still have to work, but the need is more flexible. The buffer of need could be years long.
So you have millions of people who genuinely don't need to worry about getting paid that much.
Unfortunately many countries like the US are very under-developed, and have trained it's citizenry to approach education as job training. This is obviously an abject failure, and probably a contribution to our anemically slow and pathetic growth rate.
This is like saying software is useless if the user doesn't read the source code of it. This what happens >99.99999% a person uses software. People want to be entertained or have their problems solved.
I'm glad to hear people say this now, because for the longest time they've also been bullshitting the public to think that pure math studies would someday lead to some kind of useful outcome.
"transforming the mind" is an exercise in ego-driven self-gratification, no different than getting ten degrees and never getting a job, nor any different than straight up porn or gambling. You have been put on earth to contribute to the progress of humanity, so get dirty and start making commodities. no more useless math-monks.
At least this is my observation: when my colleagues have been wholesale chucking stuff over to Claude, they've then been confronted with classic XY-Problem shit, poor user experiences, and over-complex solutions (which will mount future problems regardless of whether a person or an agent iterates on that code). Much of this can be solved by actually sitting down and thinking about it, and I mean at a code design level, not just a speccing level.
Many people don't realize that there are more useful outputs to solving a problem than just a mere solution. Obviously if one has a contractor mindset (you don't care about the after effects of a system) then this is of no relevance to you. Some companies promote that mindset, certainly ones that have no broader aspirations then getting acquired soon. That's fine - but many companies actually are about sustainability, and understanding in these places is paramount.
But I think what is lost is the value in (a) person(s) who intuitively spots and avoids these problems. My hypothetical example that I have seen play out: The domain expert spots one of these problems and succinctly explains that the problem doesn't need to be solved. Maybe it's a byproduct of bad design elsewhere, or there is a much better solution that avoids it altogether. Once the rest of the team/the lead catches on, either hindsight bias or outcome bias, or a combo, takes over and they don't recognize that all the value all along was in the person being able to spot this situation. They throw that in claude and claude does it's sycophant thing (after expressing that the previous situation was flawless for however long), and they are off to the races right back to automation bias with zero pause for thought...
Even before the AI era, we were flooded with a veritable DDOS attack of slop, in our in-boxes, meetings, etc. Managers believed that innovation was held back by quantity of ideas, or by the domain experts being "resistant to change." This led to day-long group brainstorming sessions that yielded nothing. Part of the transition from student to junior to senior is learning to filter your own ideas, and to guide others in the search for ideas that are likely to be fruitful.
The criteria for slop filtering have always been subjective, arbitrary, or driven by institutional culture. That's what we've got. Any means of generating ideas produces slop in the absence of filtering.
What will slop filtering look like when the AI can generate tera-proofs per second? What does it look like already for our e-mail inboxes? We are asymptotically trending towards slop filters that look more and more like "ignore everything." I already read none of the business memoranda that I receive. I don't answer my phone. The number of workers needed to read and assess business memoranda is zero.
The slop filter still functions in math. Problems are pre-filtered for being of interest. Somebody expressed enough interest in the Navier-Stokes problem to give it a name and attach a prize to it. There are still plenty of "prize problems." Here are some:
* Quantum gravity
* Improvement of superconductors that don't need helium
* Practical generation of power from fusion
> Part of the transition from student to junior to senior is learning to filter your own ideas, and to guide others in the search for ideas that are likely to be fruitful.
I wonder how that'll change over time now. I think it's pretty important to have a few instances of thinking hard to come up with ideas and being totally convinced they are correct, only to have someone else trounce them with one or two succinct thoughts of their own. Those were very big personal development moments for me. And equitable, I still have folks with a lot less exp/time come out of left field with maybe not as formulated, but amazing ideas.
Funny stuff, as if it were hell-bent on writing a paper rather than actual software.
Outside of that, what?
Will all rich people in all countries collectively agree to be moved into tiny apts same size and give up their luxuries?
If not, how do you define who gets what? Who gets to live in the fancy house, etc?
Using the metric of “4 earths of resources” is confusing and easily misunderstood. What that metric means is that if everyone were to have the same consumption as the average U.S. person tomorrow, we wouldn’t have enough renewable resources to meet demand.
This is not an insurmountable problem. It just requires increasing production everywhere else. Your cited metric isn’t a stat on the theoretical limit of global production, it’s one that describes its current limits.
The stat also factors in carbon emissions, but this is also easily addressable in a world where labor and maintenance is cheap.
Land, energy infrastructure, mineral deposits, grid capacity, etc.
Who gets to live in the nice parts of the world? Who has a national park as a neighbour? Who gets to be ravaged by hurricanes every year? Who gets to have more energy production closer to then? Who will own the land that has the finite resources that are used to create abundance? Where will people with the most power live?
There are so many reasons beyond production why certain areas fare better than others inherently, I have not seen any answer to this apart from that all rich people will just fold and we will all be moved into soviet skyscrapers which I find improbable.
Our current oligarch-capitalist system (wherein most of the world is dictatorship or plutocratic) is not capable of equitable resource distribution. However, future, better systems of organization will likely provide better mechanisms for distrubtion.
Capitalism is an old and dying system. It is clearly unfit for the 21st century. It's best thought of as anachronistic, like feudalism. Once it's been excised from our societies, we'll be able to answer your questions.
Meanwhile capitalism bases itself on game-theory and evolutionary selection. You compete with power, sexual selection and resources. It's sadly a system more aligned with the natural order of things. If you strip away all banks, hedge funds, etc, you are again going to get people who want to mate with the hottest fit partner, get the most resources to ensure survival of their offspring, etc.
The organizational/political problem I mention above with the land use land in the same bucket. Who gets the best land, etc. Historically this has just been decided by war and killing each other. Maybe you're right and that's where we end up, but that is very far from the utopias talked about in the comments above.
This is beyond naive.
[1]https://online.ucpress.edu/elementa/article/doi/10.1525/elem...
What it does change is the basic equation underlying capitalism as an economic system: a few people concentrate capital in their hands, but capital by itself is useless without labor, so they have to hire workers to produce something useful with all that capital that they own. The workers get a shit sandwich, comparatively speaking (given the amount of wealth they generate), but they do get something out of it. The system is stable overall because, while a better arrangement is possible, most people - even low-wage workers - have too much to lose to riot.
With AI that can do most work, the capital no longer needs labor. People who own all the robots can just have them produce whatever they need. Everybody else is then "economically redundant", and the whole system collapses because people aren't going to sit around and starve because there are no longer jobs to be had - and at this point they have nothing left to lose and everything to gain by rebelling and taking over the capital.
This is a non sequitur. People being economically redundant doesn’t necessitate a system collapse. It assumes our existing systems can’t handle an increase (assuming we have economically redundant people today) in those who are economically redundant.
Capitalism defines who owns the means of production scoped to the definition of ownership in a particular governance structure. In the U.S., for example, full ownership gives people the right to produce as much as they want with rules in place to seize the rights of ownership if broken.
One rule of ownership is that taxes must be levied on the positive return of capital. This today is what helps funds the welfare state. The idea that the system collapses implies we won’t be able to scale this welfare state up which seems unlikely to me.
There’s already a deficit caused by welfare spending. An increase in abundance (of work, resources, etc.) can reasonably alleviate this if we consider the possibilities in removing inefficiencies within the government itself and their biggest cost drivers like public health.
Firstly, each country is different. So the relationship between say the USA and Mexico is different to the relationship between the USA and say Ethiopia.
By "richness" we might also be better off thinking as less "spare cash" and more "standard of living". Which includes food, housing etc but also security, safety etc.
In truth much of the US "richness" depends on other nations. If there was no Latin labor the US would have a fresh produce problem. Indeed the very premise of DJT is that the US has a trade deficit with just about everyone. Much of US "richness" is thus dependent on the "poorness" of others.
Yes automation, cheaper energy, AI can reduce the labor content of goods and make them universally cheaper. Yes I believe in a future where the overall standard of living increases.
But it would be a mistake to believe the US is "leading" US there. The US achieved its current position at the barrel of a gun, and uses the military (and threat of military) to interfere with countries and regions either overtly or covertly. The entire premise of the cold war (outside of western europe) was premised on interventions to resist countries switching their political model to something the locals preferred.
And yes, maybe the locals preferred wrong. But the US didn't promote democracy, they promoted "willingness to keep supplying us". Countries which were leaning away from the US had to be punished. (Shah in Iran, Vietnam etc.)
I could go on, but I'll stop there. Suffice to say that an equitable standard of living is not in the (current) interest of most US citizens.
The companies do not do this, because equal distribution of resources is poison to capitalism. The world where everyone are fed, and their basic needs - shelter, medication etc. - are taken care of, does not have trillionaires. Possibly not even billionaires.
It is not profitable to give people humane standards of living. You work a lot harder when you see the homeless on the street on your way to work.
That doesn't mean the transition will be smooth. It doesn't mean that certain classes won't be worse off. I can certainly imagine programmers being among the losers. And there certainly are futures where we slide in dystopia. So let's try building the good futures. Alea iacta est.
For you.
like, how are people not getting this? is everyone here just living off their capital gains or something and that's it?
I agree with this statement, though I think this Brave New World is incredibly exciting to some and dystopian to others. The former group might include those that value the intellectual process above financial reward and status.
The flip side is there are many people, especially in tech, where their area of expertise has evaporated along with their lucrative and previously high status careers. It used to be possible to have a technical job by essentially following recipes and it turns out AI is far better at that than a human.
The linked article lays out why human understanding of mathematical models remains essential and I think the same applies to software. We're gonna need more software engineers who are able to think critically.
There’s some hubris in thinking we can understand everything. For truly difficult problems, it’s entirely possible that humans are simply incapable of comprehending why a solution is true. But ultimately the practical value of applying that solution to the real world is going to eclipse our need to understand it.
Math is just the beginning. I see it happening in other fields too, like physics and biology. Many of us software devs have already given up on understanding parts of our own systems for the exact same reason.
Seems like a losing battle.
This is hardly new or novel, you’ve just described huge chunks of existing engineering disciplines. There is still no complete model that explains why airplane wings work, there are lots of very useful models, but all of them have fundamental flaws where their behaviour completely breaks down in certain very possible scenarios. Notably scenarios where airplane wings don’t spontaneously either stop working or explode.
We also built the entire commercial airliner system decades before we even had computers capable of doing the aerodynamic analysis of airplanes, but none of that prevented us from creating huge engineering disciplines around the empirical data we did have, and slowly chip away at the underlying theories as maths and computing improved over time.
So we’ve always lived in a world where we extract value from systems we fundamentally don’t understand. But that’s never stopped us from working to understand them anyway, and deriving even further value from that improved understanding.
So far our solutions were more or less understood thru some models of reality that we’ve constructed (on our own), which may or may not reflect reality perfectly, and even if most of us never bothered thinking about these models, some people did and they understood them on a very deep level.
But we may be getting to a point where the problems we need to tackle become too difficult for humans to model, or even to notice their existence, like asking an ant how a Boeing 747 works.
Maybe this was always the case but now it seems like we might have a shot at making these solutions useful even if we have no idea what they’re even solving.
The author mentioned what the issue is: a lot of people are experiencing what it's like to 'catch up' to the more intelligent ideas. Everyone can 'get' the intelligent ideas given enough time. But how much time do you have? And he's concerned someone will rush and create some kind of world-ending solution because they're too trusting of a technology they don't understand.
His only solution is some kind of throttle on the advancement of technology, and arguably knowledge (whose?).
You observe others behaving poorly and immediately succumb to defeatism, assuming there was no better way.
When presuming, all that matters was the short term economic profit, you simply use the wrong gauge.
Here, long term human survival is at stake, regressing to zoo animals isn't a sensible option.
Do you think every Mathematician has a deep understanding of building architecture, engineering, physics, economy, etc., everything necessary to gauge whether something like a 1 terrawat nuclear plant is completely safe in all aspects?
Of course not, so then you have to select appropriate ones. And then you need a process to ingress and egress reviews. Oh what about change? Does that come back to the Math council too?
Oh, we also need global cooperation to pull this off. And then do you trust the people, the selection process for the council (all avenues for corruption are there).
I don't have a solution of my own, so it's not fair for me to criticize the author in this way. I share his concerns.
And it's depressing to think of all the negative scenarios, and be so concerned all the time.
These issues are important and worth thinking about, regardless. Who should teachers teach?
People used to solve logic problems by hand, verify logic. Now you pile it into a computer. The problem has been around for a while, no-one fully ‘understands’ the proof of the four colour theorem as a big chunk is computer proved.
We are now just changing (admittedly greatly) what we can put in a box marked ‘checked by computer.
People didn't stop understanding how to do addition or subtraction when the calculator came out. That's a simple example, but if the AI is superhuman, I don't see any reason why it cannot break things down into simple concepts. I don't think anything is truly beyond comprehension; it just needs to be explained properly (by someone, or someTHING that really understands it) and, for complex ideas, time taken to understand them.
> given up on understanding parts of our own systems for the exact same reason.
Maybe this will be a sign of when ASI is achieved. AI researchers keep saying they don't fully understand how LLMs work - maybe when ASI arrives, it can explain that fully.
You should try being a teaching assistant or a tutor. "Everyone can be taught everything, it just depends on the teacher" can only be said by someone who never tried this in practice. Just as not everyone can learn advanced math, the best mathematicians also have their own limits and there are things they won't be able to understand or comfortably navigate. Not just because it's a lot of material, but because it's complex inherently.
What the article really says is that we're going to need much smarter mathematicians. That is not possible for puny meat-brain humans. Humans are close to their ceiling. AIs are just getting started.
In practice, we're probably going to hit that limit first in IC design. I once went to a talk by the Intel engineering manager who headed the Pentium Pro effort. That was the first superscalar x86 CPU, and it took about 5,000 engineers at peak to design it. Getting that many people coordinated on one thing was a real achievement. Then Intel stayed with minor tweaks on that design for years.
We're soon going to be seeing designs of even greater complexity cranked out by AIs. No human will understand them at the gate level. Reading AI-written programming language code is bad enough. Reading AI-written Verilog may be beyond human comprehension, except in small sections.
The whole point of mathematics is to vastly exceed that natural ceiling by gradually building a framework for understanding. In fact it’s wrong to speak of a ceiling altogether. If there were a ceiling, we’d have hit it long ago.
AIs have already swallowed the entire history of human thought, but apparently they’re ’just getting started’. I can only assume you don’t know what mathematicians actually do.
We have naturally only explored the mathematical universe in the parts where telescoping via clever abstractions can get us. But there is much more. Being able to juggle more things in your mind at the same time can have qualitatively massive benefits.
Information theory and proof theory, algorithmic information theory etc has of course explored this.
If Math Academy teach 10 year old kids calculus, I doubt that.
Hopping on this train: the human ceiling 100 years ago is now advanced undergrad material in mathematics. I see no particular reason for this to change, especially with everyone in the math research pipeline pushing to compress the difference just as always before.
The original Pentium was already superscalar, with its asymmetrical U and V pipes.
The Pentium Pro added out of order execution via register renaming. A true achievement, indeed.
And as for the more general point you are making: computer chips have been far too complex for any single human to comprehend for decades. I left NVidia after working there as an archutect for five years, barely understanding anything about those behemoths.
Then only in the very most recent times did being smart become a characteristic that had a near endless market for human expression. Computer programming and the need for more programs was the apex. The rich nerd was familiar to everyone.
It feels like we're nearing the end of that era, at least in it's pure form.
What's beyond strong and smart?
Influential. It always has been.
Owning capital.
Society has been built on the principle that people can find productive jobs, earn money from those jobs and pay for life's necessities with that pay. Prior to all this we had more subsistence living. It started to change in the medieval era and really took shape with the industrial revolution.
But what happens when, with increasing automation, there aren't enough jobs for everybody? That's the fundamental question of AI (and automation in general). The two extremes for how this can go are:
1. A small group will become increasingly wealthy while large swathes of the population will be permanently unemployed or underemployed. This is where people end up living in shanty towns and slums. Wages are overall low because demand exceeds supply; or
2. Society as a whole enjoys the benefits of automation by having to work less. Basic needs are provided for. People get to live with dignity regardless of whether or not they win the job lottery.
Put another way: AI isn't a technological issue. It's an economic one.
Why would mathematicians be the best at this or even able to contribute? Engineers and physicist seem like a much better choice. Mathematicians tend to not bother with messy things like whats physically possible, or human consequences etc.
Tao has been posting a flood of guest posts that stress inevitability and coping. Tao is fully invested in AI but needs to have the appearance of a broad discussion.
Sahai is of course exuberant about AI but wants to keep UCLA enrollment numbers high. The comments on Tao's blog are far less friendly than here, because by people see through the game that is being played.
Just look at the latest industry friendly post from Tao, where he pretends that it somehow summarizes the discussion so far:
https://terrytao.wordpress.com/2026/09/25/iciam-statement-on...
For my whole life I have never been subjected to an industry coordinated advertisement campaign that ruthlessly harnesses YouTubers, TikTokers, professors, open source people who all go in lockstep.
Tao, Gowers et.al. will go into history as the professors who ruined math.
You can't simply prompt a model to be "better" when "better" isnt even properly defined
> Our ability to understand difficult and unfamiliar ideas may become one of the most important contributions we can offer to society. We should be willing to bring that skill to problems far beyond our usual research interests. [3] Doing so asks us to expand our sense of our vocation.
Am I reading this wrong, or is he talking about what (mathematitian) Data Scientists have been doing for years? So he is basically saying that former Data Scientist that have turned into prompt/software engineers should go back to being data scientists.
In any case, people should stop trying to fit AI in the previous status quo. What we need is curious people, that is what we have always needed.
A few centuries ago there were no "mathematitians", there were mathematitians/philosophers/artists/physicists all in one person. So its not like "mathematitias" is something that has existed for millenia.
We need curious and ethical people.
Maybe AI brings back the age of a well rounded scientist/philosopher. I know this sounds counter intuitive because the article is saying that we cannot keep up with the AI.
I’m fairly confident that the opposite will be true. The most important decisions will be made by AI, and humans will only be left to guide decisions as a matter of taste. AI has the ability to be impartial, and immutable. You can endlessly probe and reason its decisions.
This isn’t true of our current human decisions. Where we see red tape, bureaucracy, rent seeking, status quo. AI will see through these human constructs.
I do like the idea that we will need more mathematicians, physicists, and scientists to understand the discoveries of AI. That might be the best possible outcome of AI, but I fear the opposite and we are painting ourselves into a corner of which we no longer have the knowledge to sustain ourselves and society itself collapses.
Why would an LLM care about eliminating any of these things? When you say "red tape" and "bureaucracy" you are, in that very statement, applying emotional labels to what are actually just systems working as designed. There are the rules, so we follow them. LLM decision-makers will happily enforce and perpetuate bureaucracy and red-tape. If the rent-seeking is legal behavior then they will fully protect a big corporation's right to engage in it, they don't care about the feelings of the little guy. If a similar situation was decided a certain way in the past, then yeah let's decide it the same way now and maintain the status quo; LLMs very often think that way, they don't care about human ideals like "change" or "progress".
Which AI has this ability? LLMs definitly don't.
It also lends itself nicely for the ability to adjust the AI systems when deficiencies are found. You can tweak the AI systems to align itself with the desired output, and check for regressions, much like tests in a codebase.
Human decision-making lacks this thoroughness. I’ve scarcely come across rigorous documentation of decisions and how they were reached professionally.
Not saying you’re wrong you just can’t be so certain
See proofs never before possible made by someones intuition.
But maybe they wont be mathematicians.
Not Terrence Tao post. Beware.
today they benefit from human willingness to share publicly and improve ai systems, but once those systems start to threaten their livelihoods, will those dynamics change? can the machines push the frontiers or is the global network of human creativity and tenacity necessary?
Unfortunately that’s a question you’ll have to answer for yourself. Also, people are going to risk going along whatever path gives them work as many today already do.
Humility is the wrong word. We didn't feel it when the steam engine was introduced, so why now?
Fear? Ai is an abstract (for most people). A steam engine, you can touch. And, it doesn't replicate.
In a world with ASI, having that capacity is vital.
I mean just looking at all the numbers to see which have interesting properties could take a while.
It will be as it always has been: these ideas will further enrich the rich, at the expense of everyone else. We'll have our first quadrillionaire, while the masses are debating whether the minimum wage of $7.25/hr should be bumped up.
> Another approach is to follow that word, heresy. In every period of history, there seem to have been labels that got applied to statements to shoot them down before anyone had a chance to ask if they were true or not. "Blasphemy", "sacrilege", and "heresy" were such labels for a good part of western history, as in more recent times "indecent", "improper", and "unamerican" have been. By now these labels have lost their sting. They always do. By now they're mostly used ironically. But in their time, they had real force.
>
> The word "defeatist", for example, has no particular political connotations now. But in Germany in 1917 it was a weapon, used by Ludendorff in a purge of those who favored a negotiated peace. At the start of World War II it was used extensively by Churchill and his supporters to silence their opponents. In 1940, any argument against Churchill's aggressive policy was "defeatist". Was it right or wrong? Ideally, no one got far enough to ask that.
What You Can’t Say https://paulgraham.com/say.html
Look at the average life of a person living in Western Europe 400 years ago.
Look at the average life of a person living in Western Europe today.
Heck, look at the lives of the top 1% of humanity living 400 years ago.
From: https://en.wikipedia.org/wiki/Louis_XIV#Personal_life
> Louis and his wife Maria Theresa of Spain had six children from the marriage contracted for them in 1660. However, only one child, the eldest, survived to adulthood
From: https://en.wikipedia.org/wiki/Louis_XIV#Health_and_death
> Louis outlived most of his immediate legitimate family. His last surviving legitimate son, Louis, Dauphin of France, died in 1711 of smallpox and barely a year later, Louis, Duke of Burgundy, the eldest of the Dauphin's three sons and then heir-apparent to Louis, died of measles. Burgundy's elder son, Louis, Duke of Brittany, died of the same disease a few weeks later.
The king of France, arguably one of the richest people on the planet at the time, had 5/6 six children with his wife, the Queen of France and daughter of the Spanish King die before adulthood.
Now look at the life of an average middle-class French person (or American, or English person or any of the 6+ billion people living with electricity) today. See anyone dying in childbirth? Anyone dying of measles or smallpox before they're old enough to vote?
Technological progress, helping not just the rich get richer, but everyone get richer.
QED.
It really looks like Tao is on the path of accepting and embracing AI now.
Or it as with other professionals: shortage means shortage of cheap labor.
This is an unfortunate example to choose, being as it is entirely confounded by the canonical illustration of the https://en.wikipedia.org/wiki/Law_of_triviality.
....-----
---/Shark\---Terry TaoWhat is Magnus Carlsen going to do when he can't beat the computer?
It seems like a category error between humans using tools and humans building tools.
There is not much point in trying to figure out a better chess engine. There has never been a better time though to want to learn chess.
I find the idea that the computer will discover mathematics and humans call it a day rather ridiculous. As if humans will not then spend their time understanding and incorporating the ideas from the computer.
Alphafold is a better example. Alphafold is only bad if you spent your life trying to solve protein folding. But even if you did, that is the same person who is the most setup to reap the benefits of the unlock in the pragmatic application of protein folding.
We don't figure out how to get machines to harvest corn and then spend all day sitting around eating corn in between naps.
Stuff! Inscrutable stuff, maybe, but that's not "doing math for math's sake."
Yes, why not? And, of course, AI doing math for AI.
We may not be needed forever...
It becomes another abstraction, really. As long as we can use it for something useful, it's still valuable.
If the LLM is operating at such a high level that it never actually constructs a useful product for humans to use, then how will that be good for humanity?
If you replace "LLM" with "mathematician" than this is the state of the world today. Stuff like Galois theory is beautiful mathematically, but what has it constructed or enabled for you and me?
A cpu (the physical thing that sits in your mother) is not an abstraction, what are you talking about
If you believe that math is discovering, it's natural to think that all of that AI math already exists and is just waiting for us to find ways to discover and understand it.
Don't write us out quite yet. :)
We want to understand. Quantum physics, mathematics, how stuff works. Ants don't.
That want is not a given, not all of us have that drive. In fact, very few of us have it. So far though, it seems multiple disconnected civilizations learned to keep that trait going instead of suppressing it and focusing only on practical ant-like activities.
This makes no sense whatsoever.
We need more, because there will always be far more difficult problems yet to be discovered and solved, and that means, we certainly need expert humans to define and verify them.
If you cannot even explain the problem you are facing, not only you don't understand it, but you certainly would not be able to know if the AI solved your problem correctly.
And this will be true for how long? 3-4 months?
We were told in 2023 that there’d be no more jobs within months and the singularity was here. At some point you have to realise it’s not happening.
Let me know when the other (much harder) millennium problems get resolved, let alone explained in a satisfactory way.
I never recall hearing that other than from people strawmanning what the labs were saying.
It's the same as conservatives who tirelessly repeat "In 1995 they told us the world would be underwater by 2015, those climate scientists are full of shit" - no, no one ever said that, you just listen to grifters who only can strawman.
By the way, I don’t think climate denial has (or at least should, though I know Americans like to politicise everything under the sun) anything to do with ‘conservatism’.
The risk of liability is a social problem that is far more difficult to be solved with technical solutions even with AI.
I propose we enact a new law that makes it so ai engineers can only fly on ai designed planes powered by ai designed engines flown by ai. Maybe then we'll get some real progress out of this slop producing crapware or the problem will be solved a different way.
I can’t afford to be one of your credentialed reserves, Amit. ADHD screwed me over in early life or I’d have breezed through that dual masters in cognets/compsci by 2003, and now that I’ve stabilized my brain and am burning through years of school at a rapid pace, my country’s socioeconomically ruined — I’ll be lucky to escape with my accounting degree as my school is visibly being sucked into the vortex every year I progress. The only hope left for me to be what you need of us is to self-study, but without a degree I’ll just be treated as a crank or an AI proxy/puppet if I slip up and talk about interesting math with anyone, so what’s even the point of taking that path? I originally pivoted my math skills into systems theory and process diagnostics instead, which of course now everyone has kicked to the curb and replaced with AI. I’d have made an excellent Susan Calvin — I was working to be a cognitive tech with diagnostics, zeroth law, and group psych specialties, AMA! — but the financial investment to provide the runway to take that lonely, dreary six year slow as molasses slog through maths that universities think is somehow an appropriate teaching velocity (six months for precalculus alone?!) in order to earn the mere chance to have my resume rejected by an AI firm that uses AI hiring and and can’t tolerate someone with a strong moral position regarding societal harms is a very bad choice, whether you use simple probability or game or systems theory to evaluate it. Taking that quarter-mil-plus burden as loans in the hopes of employment at the other end in a field actively having its social, reputational, and moral fabric being ripped apart by AI? That’s not just a bad bet, that’s chasing fool’s gold at the end of a fading rainbow in a desert mirage. So, with sincere apologies, while you’d benefit from having me on your ‘reserves’ list (I have written testimonials spanning some twenty-plus years to that effect) I’ll never come to your attention as a support tech or as an accountant, and I accepted that outcome years before anyone else realized this need for cogsci mathematicians with a teacher’s specialty of analogy-building and the ability to disregard interpersonal nonsense to focus on the needs of societies. Better luck with the next generation, though!
I imagine:
* Massive budget overruns
* Year and year of delays
* It ending up wasting more energy than it produces, or just not working, period
* Politicians disclaiming responsibility and blaming it on "the AI"
* The contractor companies profiting immensely over the entire period, and ending up not having to worry about maintenance, warantees, etc.
I wonder if the current maths models, if any, are able to use formal solvers in their 'reasoning'. I wonder if a natural language interface is really that efficient, maybe a pure formal language hinted with intuitive "tokens". I remember the time I was learning real maths: "elegant", "brutal", "strong", etc were somewhat meaningfull.
No matter how good your case is - if you use AI slop to spam text, I will not read it. At the least they admitted to this sneaky slopness, which saves me time. I'd wish everyone would do so.
Watch me get downvoted some more. Nobody cares about the truth, we now know thanks to our beautiful president Trump telling us about that and the Fake News.
There's, like, one, two at most, mathematicians who are really thinking outside the box. The rest are like sheep. Or cult members with mass psychosis.
Waste, Fraud, and Abuse people are coming to academia, which is so hopelessly parasitised by the left.