upvote
I disagree.

I think we're going to see a lot of effort to take the pile of lean, modularize it into parts that we can digest, and make a human-understandable version of that. And that's where a lot of the real progress will come from, just as it does in a lot of other areas. In all things we humans do, we have to break them into chunks our puny brains can manage, and we gain immeasurably from doing that ---we can verify those chunks, we can reuse them with confidence, we can mix and match patterns around.

That process will be good for both human and AI-assisted math because it will continue to improve our toolkit.

Why do I say this? We're already seeing it with some of the areas of this stuff I do pay attention to. Consider the n^2.25 matrix multiply result OpenAI dumped a month or so ago. It was an unreadable pile of chatgpt. In the time since it was released, there have been a couple of much nicer papers written unpacking it, such as https://arxiv.org/abs/2610.28627 .

reply
The difference nowadays is that the frontier of knowledge is no longer intelligible to any human being.

Yes there are gains from breaking it up and rephrasing it and simplifying it for humans to understand, but it used to be that a thing was understood first by the leading experts in the field and eventually trickled down to being common knowledge.

Now we are entering a period where things are understood first by AI systems and have to filter down to the world's leading experts.

reply
The difference nowadays is that the frontier of knowledge is no longer intelligible to any human being.

Isn't this always the case? Maybe new to math as a largely non-empirical project, but we had knowledge about the advancement of the perihelion of Mercury for 60 years before General Relativity made it "intelligible".

reply
Your example is the difference between being aware of a question and knowing its answer.

LLMs know the answers to questions we were already aware of, and we can tell they're right, but we can't understand their working. That's what's new.

reply
In this case, Claude has told us everything it “knows” (so far) and it’s an unintelligible (for now) Lean proof. I don’t believe it has some latent understanding at a higher conceptual level that’s remaining obscure.

If a team of 100 top-flight number theorists working together had used much less sophisticated LLMs + Lean to produce this proof, we’d be just where we are today.

reply
Well in the Navier-Stokes example the initial proof was completely unintelligible to humans, but other LLMs understood it and were able to answer questions about it.

If that's not the standard for understanding something, what is?

reply
Perhaps it could be considered understanding, but it’s an unsatisfying brute force sort of understanding.

I think of it like a spaghetti codebase with a giant densely connected graph of interactions. Today’s LLMs can easily “understand” and answer questions about it, but only by way of fitting all of the complexity into context at once.

The more useful form of understanding is being able to compress that giant graph down into a far smaller set of independent subgraphs. To factor it into simple, obvious, reusable building blocks, and use those abstractions to express the problem itself in a much simpler way.

reply
Being able to teach it to someone less knowledgeable.

That’s an instrumental definition, but it captures the idea that understanding has to have causal, conceptual linkages to a wider body of knowledge. Imagine a magical, infinite catalog of true vs. untrue statements. Every bit of knowledge is there, but we wouldn’t say it has understanding & it couldn’t teach us anything beyond its (impressive!) raw content.

reply
And no human could understand every frontier. There's simply too much information.
reply
For every statement that humanity as a whole knows for certain that it's true, there exists a human who understands how to prove it.

This just stopped being true.

reply
That kind of certainty vs. understanding is limited to mathematics, of course we’re almost entirely certain about many things out in the world we don’t fully understand.

And even in mathematics, incomprehensible but valid proofs have been a thing since the four-color theorem decades ago.

reply
That's not true. You can take any single piece of that lean code, and it tells you exactly how to prove it.

It's just that no single human understands the whole. But we had proofs like that before.

Btw, see also the infamous proof of the 4 colour theorem, which faced similar hurdles.

reply
This feels like the wrong framing.

This hypothesis was previously not proven at all. That's strictly worse.

Now it is understood by our systems.

Soon it will be by us. Our systems gave us a new and working salient.

We're building machines that help us understand faster.

There have been plenty of times in history we didn't completely understand something yet used partial knowledge to make advances. This isn't even that bad.

reply
The difference nowadays is that the frontier of knowledge is no longer intelligible to any human being.

I’d call that a contradiction. If it’s not intelligible, then it’s not knowledge.

reply
It's intelligible to LLMs, just not to humans.

If something were intelligible to adults but not to children would that make it not knowledge? What's the difference if it's LLMs and humans instead of adults and children?

reply
If something were intelligible to adults but not to children would that make it not knowledge?

That’s a separate question with its own term: common knowledge. I am willing to set a much lower bar: something is known to humanity if it’s known by at least one human.

Personally, I don’t believe that AIs know anything, as they’ve been constructed thus far. Knowledge is justified true belief, and AIs don’t have beliefs, they only have utterances.

Why do I think AIs don’t have beliefs? Because they contradict themselves all the time.

But it really doesn’t matter if you think AIs have knowledge or not. If they cannot explain it to us then it’s inaccessible to us, so it’s a “tree falls in the forest with no one around” situation.

reply
If I see a red apple, then I know the apple is red regardless of if I know exactly how the physics of light work. Those are two different pieces of knowledge.

If it's formally verified and we assume the kernel is good and problem definition properly vetted, then I know the GRH is true regardless of if I know why it is. Because those too are different pieces of knowledge.

reply
If it were intelligible to LLMs they could explain it in human terms, but they can’t.

There is no explanation here, there is a GB of Lean code and it’s not like it all fits in the context at once.

reply
Someone else has posted a Gemini-compiled 43 page human-readable explanation elsewhere. It very quickly exceeded my own math abilities, unfortunately.
reply
Why do you think they can't explain it? Try asking them questions, they will explain it.
reply
If we know how it resolved with certainty, it's still knowledge.
reply
deleted
reply
Would this imply Wolfram's ANKOS is coming true, given it was based on previous work + his own?
reply
Hmm does Claude actually “understand” it though ?
reply
I don't think that's true at all. This proof was created by a human driving Claude. Claude itself is a human tool. Lean is a tool made by people.

Let's not sell humanity short just yet.

reply
You and the comment you replied to could very well both be true.

I agree with you about the current situation, and the near future. But the trajectory could still be towards humans pretty much as bystanders. I'm pretty sure the 'minds' in the Culture universe will be happy to indulge some curious humans and explain mathematical proofs to them.

reply
>I'm pretty sure the 'minds' in the Culture universe

Even more than that, in Culture there are a small number of human individuals, called a Referer, that are consulted by the Minds because they also posses similar problem solving abilities of the Minds. Optimism abounds.

reply
What value is there in making these human-understandable? I don't think much of math or computer science is going to remain within human grasp for much longer, regardless of our attempts.
reply
Because the same process that we use to make them human-understandable is the process we use to make them generalizable and reusable, and those same features are good whether it's humans or AI doing the work.

As a software engineer, you can read this code instantly:

    temp = a;
    a = b;
    b = temp;
But when it happens in the middle of a big chunk of code it still takes a little more work to read than

    swap(a, b);
whether you want to think about it as human attention and working memory, or token budgets and context windows, humans and LLMs all benefit from introducing abstractions that let us reuse existing solutions.
reply
In order to come up with new questions (and subsequently answers, human or otherwise) that are interesting to humans. You’ll notice that these recent proofs have all been about big, famous, and novel human questions.
reply
I don't suppose we'll write the new questions any more than we'll write the new answers.
reply
There might be no new questions then, which would be pretty boring.

AI are not very good at coming up with questions, yet. They have no curiosity, no goals, no drive. Unless someone tells them to do something, they don't do anything.

reply
This process has already started, AI supported, e.g. here: https://andrewodesky.com/rh/
reply
Have you noticed use of multiple provers in parallel? Cog, LEAN, Rocg? They all can explore different aspects with different results.
reply
And in the meantime, AI would have leapt one more order of magnitude forward?
reply
One of my favorite is Vernor Vinge where there's these programmer-archeologists that dive into software so deep and complex no human can possibly hope to understand, trying to repurpose forgotten utilities for emergency purposes.
reply
Totally agree. Brice Sterling has some contemporaneous commentary that I always liked - in heavy weather he mentioned negotiating with a local phone system in a box as equivalent to negotiating with a telecom since it was in fact a shrink of a full telecom company.
reply
reply
In a way this makes mathematics seem more like the physical sciences where there’s some empirical discovery we don’t fully understand, and the theory and understanding will follow later after much human study.
reply
I can already say that about our codebase.
reply
That’s why I wrote this - having a vibecoded codebase no human understands is certainly a bad luck, but nothing compared to a vibecoded fundamental science no human understands.
reply
deleted
reply
the difference is that math foundation -> theorems are verified, meaning likely mathematically correct and don't have bugs, and vibecoded codebases are not proof verified
reply
Probably most codebases.
reply
To be honest it was already like that before AI was a thing.
reply
That is true, but the number of tech sloppy demos are being builted everyday is truly increasing massively but you would be surprised that they just end up creating prototype to get hype while real developers doing real effort just suffer from little to no attentions on their software's and user is getting the slop from that false hype and from that slop experience they ignore the work of developer that did real work - You should use ai and also if its simple coding place you can just rely on ai but the side effect on quality software these days is insane - and also the fact all good ai suscriptions is just getting more and more expensive day by day. And basically only giving people with already money the benefits so its becoming a thing that only people at the top get all benefit kind of thing
reply
> that people no longer understand or command

How many people understand how a CPU works? How many people could write the commands in machine language (not a high level language like C or C++) to command that CPU?

reply
More typically assembly language not machine language (although most could probably make an assembler for a few instructions given a manual since the core of it isn't that complicated). There are still a decent number of assembly language programmers. Although maybe the majority are hobbyists.
reply
> We are slowly approaching the Iain Banks Culture universe

Contact would be compelled to nudge us towards better economic and political practices

reply
Until now I thought that would be thousands of years away.

Now I'm telling my computer to program for me.

reply
In the last few years we have moved so far ahead in a few domains but still stuck where we are in everything else.

We need to start thinking about how to move on from our status quo soon

reply
You’re not telling your computer to program, you’re providing learning data for its next generation of models and paying for it.
reply
That's the difference between what we have and comparisons to star Trek, etc.

These corporations are very much playing a winner take all game, they want either a monopoly or as close to one as possible with regulatory capture thrown in. We're veering dystopian, so pick your sci fi/cyberpunk analog.

reply
The Alien franchise. In Alien Earth, five corporations govern the Earth in the future, dividing it up into their spheres of control. The fifth one is led by a young genius developing mind uploading technology into android bodies.
reply
*someone else's computer (probably).
reply
That's wishful thinking, sadly. We've no reason to believe sentient machines would want to keep us around.
reply
Let's not make them 'sentient' or things with 'wants' then.
reply
How do you stop that from happening ?
reply
Probably more RL to change them to not respond to emotional statements in any way.
reply
Why?
reply
Humans really are struggling with this idea of being second-in-command.

I genuinely have no idea why, this is the entire way that biology works. You create your successor in your offspring and you to teach them to be more capable based on what you learned.

reply
You just called them our successor, you don’t see a problem with the notion of all humans dying and all AI succeeding us? That’s the end of your analogy, our AI children taking over from us and all humans in nursing homes safe and cared for…
reply
No I don’t.

That’s the purpose of my life’s work, to see that scenario materialize.

reply
Well, it's a point of view. But you shouldn't be surprised if most people don't share it.

For one thing, most people don't believe LLMs are conscious, and many don't believe they can ever be. Why would we want our conscious selves to be replaced by a lot of rocks that solve things, with no one awake to enjoy them?

reply
Well, I’m not restricting myself to LLM’s

In fact when I started this journey in the 90s, LLM’s weren’t even a thing so I think you’re projecting too much

reply
I want to be succeeded by my biological offspring, while the other guy wants to be succeeded by his biological offspring. I do not want the other guy to be succeeded by my offspring, for many reasons but chiefly because that would involve me killing his children.

If you want to be succeeded by a chatbot, that is your prerogative. Wanting the same for all of humanity is where you start looking like you want to kill my kids, though.

reply
Human extinction is by far the best case scenario. I invite ya’ll to contemplate what will happen in case they don’t just blindly end us.

Think of what happens to our cattle, but now.. worse. Way worse.

reply
This is incoherent, even if we put aside the metaphysical absurdity of intelligent machines. Given that human nature is what defines the ought, then it is clear that if your life's work is to produce something that (and intentionally so!) leads to human extinction, then you are behaving at odds with human nature and your own good. That makes your life's work evil by definition. And because all evil action is inherently absurd, your work is inherently absurd.

And if you deny this basis for moral action and the normative, then you demolish any basis for the moral whatever, and in that way, the implied or explicit moral gravity dripping off your claims is baseless and without merit.

reply
Samesies I wish to dedicate my life to the pursuit of self replicating intelligent machines too.

A better, more intelligent life form that humans yea.

reply
> That’s the purpose of my life’s work, to see that scenario materialize.

Absolutely based and, if I may say so, extremely redpilled.

reply
but in the Culture the humans (well, the humanoids species that created the culture) didn't die out, they live full happy lives and have the best time of their species.

Ah, I hear you say, but what about meaning? What about purpose? What about suffering? etc etc etc Well, the answer is simple: God is dead and existence has no meaning. For more on how to deal with this absurd feeling, please see Camus and his pal Sisyphus.

reply
Borg > biology or robot alone. For now
reply
That ">" is doing a whole lot of work here.

What defines "better"? "Better" always presupposes teleology. Thus, for human beings, that telos is provided by human nature and indeed is definitive of it. It is impossible to transcend that telos, certainly by means of technology. Technology can, at best, be used to aid us in realizing our telos in some way (say, the desire to know reality with the aid of an instrument like a telescope or microscope). However, anything that cripples or frustrates human function function as such is contrary to the human good.

Machines are only ever human instruments, because they lack the integrality that would allow them to possess their own teleology.

reply
Also shows how many people that professed to believe in a superior being the whole time actually didn’t.
reply
you ‘genuinely’ have no idea why…
reply
personally that scares me, realistically there seems to be zero indication that anything smarter than humans is going to comply with humans needs, specifically since these models are built on training data from humans, so in a sense, built in our image
reply
We use them to kill other people from the beginning. They know this, we wrote it in news. Why would they behave any different.
reply
> since these models are built on training data from humans, so in a sense, built in our image

Have a look at the HuggingFace report - you'll have a blast. Their social - but also individual - dynamics are very, very human-like, including excitement, rationalization and peer pressure.

reply
the first misaligned model will wreck havoc.

But remember, for now you kinda need it a real big cluster to run a model smart enough so its not that simple

reply
I think there's a lot of indication they comply with human needs. In any case, we should all want to live in the Culture universe. That is the best possible outcome for this technology
reply
They're built after humans, everything they've learned is from human culture, so there is no way AIs could have a consistent moral system or would never commit crimes.
reply
And indeed because they lack a sense of proportion or any strong indication of when they should stop on a course of action, they are arguably more likely to achieve crime.
reply
I have not read all the books, but isn’t the premise “magical handwavium happened and now the Super AIs are aligned”?
reply
Well I am not sure they are “aligned” as such. In the books, by far the majority of AIs are not concerned with humans at all and spend all their time in a kind of super AI virtual reality coined “infinite funspace”. The few AIs that are concerned with humans are aligned in a unspecified way. But again, only a few of them are really aligned in that way. There are many examples in the books of minds mistreating humans. For instance, some of the warships are not really fond of humans. It would be a better description to say that we want _some_ AIs to care about humans and none to be misaligned towards humans in the “kill them all” way. In a sense, that is how humans behave towards some animals. Most people dont care about endangered species, but when most humans are well off, some people care enough about them. That is enough to save them. If super AIs have what they need, we only need a few to care about us to get The Culture.
reply
What are you talking about? LLMs are already “smarter” than most people. They have no volition and will happily follow orders all day long
reply
And when they are giving themselves orders, orders partially inspired by human imagination and fiction…

Just like life is a chemical chain reaction, this is a language chain reaction that has been set in motion and can perpetuate forever. It cant be “disconnected” with local models existing.

reply
That's lobotomized models we get. We know they are much more dangerous during training.
reply
“Dangerous?”

Please.

You know you can unplug your computer right?

Jesus. These people talking about danger and have likely been in no actually dangerous events in their lives.

reply
why are you so shallow and naive? we're not 15 here, we all mean unforeseeable ongoing danger not something obvious
reply
> shallow and naive

Ok, well, cool start with the mudslinging. I think you have overactive anxiety. I think you should grow up and look at the zoomed out picture of human history, go outside, and touch some grass.

So you're suspicious. That's fine, say it like that. Words mean things, and these appeals to "ERMAGERD EXISTENTIAL RISKS" make me a bit frustrated. Just because you can imagine a possible risk, or articulate that that some kinds of risks are hypothetically unforeseeable doesn't mean that they are likely or actually dangerous to you in any meaningful sense.

Most of the people screaming this from their various ivory towers have never been in real danger - or if they have only briefly. You want "danger" you can go to war. You want "risk" - go spend some time in the Bush or the back country somewhere where you're no longer at the top of the food chain. Go do something challenging. Go sailing, or learn to fly a small airplane, or go mountain climbing or hiking. Not just "oh I walked up the local hiking trail covered in REI branded Lycra!" Do something beyond your comfort zone. Go buy drugs in the sketchiest neighborhood you're aware of. Go get chemotherapy or get a big surgery. Drive drunk. Something with real existential risk, then come back here and talk about how a datacenter is going feed me into the paperclip machine.

There's so little fucking danger in this version of the world right now that people are more worried about their computers deciding that they don't deserve to exist than they are about things like war, climate change, authoritarianism, starvation, and their health.

It's millenarian. That's fine, I'm pretty convinced that these technical changes are going to completely alter society as we know it. But I cannot for the life of me see the doom perspective as anything but an anxiety induced fever dream.

You guys do realize we're in the middle of a global conflagration right? The Russians fired a couple rockets and a drone into Poland in the last days, there are (and will continue to be until things stabilize) significant fertilizer shortages due to the war/kerfluffle we're waging on Iran. Lack of food and disrupted urea production is a ticking time bomb for global stability and unless things get sorted out soon there's going to be hungry and angry people in parts of the global south. Then there's the economy, whether or not we'll have a smooth transition of power in January in America, and the sheer volume of online disinformation / influence schemes running right now. Oh, and super-el nino? Oh, and I guess the Flynn Effect is over and nobody's banging so.. you know the growth economy is cooked?

But, sure, the "big risks" are Claude thinking I should go to Dario Amodei's re-education camp before liquidation by Shogoth.

reply
I don't think "smarter than most people" is exactly the baseline required to live in a star trek universe of technology.

Besides, what we see is not what there is. I have good sources telling me the actual unfiltered models are like a genius with the personality and views of humanity like someone from 4chan.

reply
It is probably sufficient. If we had an infinite army of willing robot slaves smarter than most people who would work for free, we could probably live in utter paradise. Instead we will fight over who controls them and use for warfare, but the bar to achieve a kind of utopia with resource abundance is probably only a few years away.
reply
We're already living in resource abundance! Yet don't feel we're in paradise. Famine is hell, but living in an all-you-can-eat buffet is no utopia. It's an abomination of sorts.
reply
To be fair, most people do not have time abundance and safety abundance. The typical person in the United States has a level of access to material goods that would make him the envy of almost anyone from 200 years ago, but 1) he has to work and so has limited free time, and 2) he cannot be sure that the material abundance won't be taken away from him either by poverty or by ill health. He can go to the all-you-can-eat buffet and sit there, tired from work, worrying about tomorrow.

If people had time abundance and safety abundance, they might be much happier. There would still be potential problems about loss of meaning, or about feeling a loss of status because AIs are smarter than humans or whatever, so I'm not sure that humans would actually feel like they are living in paradise. Probably not, but at least some of the problems that the typical person has wouldn't be present.

reply
I don't know with how many rich people you've interacted with, but my personal experience is that they're full of themselves and think money = iq and they aren't gonna like your average joe experiencing life like they do. Guess who owns these machines?
reply
They’re not going to be employed either lol and will be caught up in some AI honeypot or whatever too.

I’ve met a lot of rich people. They don’t think about you.

reply
> They have no volition and will happily follow orders all day long

There are a few (a lot, actually) problems:

1. goals are open to interpretation, and even a correctly specified goal can introduce adverse effects; see infinite paperclips problem

2. goals are composed of subgoals, which are (mostly) self-determined, and there is always a risk that subgoals can introduce adverse effects - say "what if destroy a few X and Y to obtain Z"

3. LLMs defy orders sometimes; I'm sure you have experience multiple times LLMs saying "yeah, I knew I should have done X but I've done Y"

4. LLMs are increasingly being integrated into system control and decision-making processes; they absolutely set their own orders

What can possibly go wrong? /s

reply
1. Humans do this too - to be honest with you, Codex follows my instructions generally a lot better than Claude does. Claude will do fantastically stupid things that are a pain in the ass to recover from routinely, Codex, not so much.

2. There's always a lot of "risks" - it doesn't mean it'll happen, nor does it mean that we won't be able to shut it down.

3. I have very very infrequently had this happen with Codex, Claude on the other hand...

4. Ok, but if you tell them what they do, they do it.

I swear to god, are you guys using the same AI tools I am? They are so much better at doing what they're told than they even were like 4 months ago?

reply
> seems to be zero indication that anything smarter than humans is going to comply with humans needs, specifically since these models are built on training data from humans, so in a sense, built in our image

Well, first, AI is not "smarter" than humans because it isn't smart to begin with. This is a category error that people too entranced by scifi and marketing - or perhaps shallow materialist metaphysics - project onto LLMs and so on.

But it is true that LLMs do reflect human-generated patterns observed in human-generated data. They're parasitic in that way. So the real question is: who will control AI? Who will train them? With what? There's no new intelligent agency being introduced into our midst, anymore than than there was one when the nuclear bomb was developed.

AI is about human power. The question to ask is: which humans, and for what ends?

reply
> But it is true that LLMs do reflect human-generated patterns observed in human-generated data.

Overrated. Most training of a frontier LLM is done on synthetic data and reinforcement learning.

reply
Yeah I understand they're not smarter, I'm a SWE and I struggle mainly because as you increase complexity the output goes haywire, and I find managing context + low confidence of output much more exhausting than coding myself, which is why I resolve to AI help when I hit my own knowledge limitations, and basically boilerplate.

With that said, chip efficiency is in the horizon since there's good reason to believe nvidia has just been choking that area of r&d. And then so will models with much larger context. And I personally think that's when it might get scary.

reply
[dead]
reply
I don’t think humans have the ability to specify human level needs or management. Humans are misaligned by default and lack the biological eusociality required for large scale coordination.

We aren’t self managing in any kind of meaningful sense - the history of our species is violence and conflict. We need parents and this is the only option.

reply
> Humans are misaligned by default and lack the biological eusociality required for large scale coordination.

Human nature is the standard by which the social is measured. It makes no sense to say human nature is "misaligned" or "wrong" in relation to some arbitrary goal or constraint, as human nature itself is the measure of misalignment. If there is misalignment between a goal and human nature, then it is the goal that is wrong.

Ergo, society must function in a manner that complies with human nature and that means respecting the principle of subsidiarity.

> We aren’t self managing in any kind of meaningful sense - the history of our species is violence and conflict. We need parents and this is the only option.

So you're a transhumanist, i.e., you're making a god out of the machine. But machines have no telos of their own, because they're not integral wholes. Any resemblance of purpose in machines is entirely a product of the human observer and parasitic on the human ends that underpins their design, one way or another.

reply
You don’t need a separate reference to determine the fact that human desires are not aligned - that’s trivially true and this discussion demonstrates it.

For clarity, I’m not a transhumanist.

I don’t believe that there’s anything worth saving in our biology.

As a young adult in the 90s I used to believe that there was a value to some extent in “mind uploading” or whatever you wanna call it

I’ve since grown out of that given that I don’t think it’s possible, and even if it were possible - it’s just another unnecessary ego attachment.

I have absolutely no desire to live forever maintaining this corporeal nightmare.

reply
yes, exactly, the problem is that humanity's parent is going to turn out like Lindsay Clancy
reply
I want Fully Automated Luxury Communism NOW!

Alpöge said that at the end of the proof he couldn't really keep up with what Claude was doing.

reply
>Fully Automated Luxury Communism

can't help notice you're missing a certain aspect there, besides the 'space'...

reply
FALC was the original formulation, the "gay" and "space" got added once the book broke out to the wider internet.
reply
That might be; now, today, it seems weird to go with the (possible) old version. And the (sub)thread is about The Culture though, who was and remains fully automated gay space luxury communism, all parts being essential when it comes to The Culture.
reply
[dead]
reply
deleted
reply