I think we're going to see a lot of effort to take the pile of lean, modularize it into parts that we can digest, and make a human-understandable version of that. And that's where a lot of the real progress will come from, just as it does in a lot of other areas. In all things we humans do, we have to break them into chunks our puny brains can manage, and we gain immeasurably from doing that ---we can verify those chunks, we can reuse them with confidence, we can mix and match patterns around.
That process will be good for both human and AI-assisted math because it will continue to improve our toolkit.
Why do I say this? We're already seeing it with some of the areas of this stuff I do pay attention to. Consider the n^2.25 matrix multiply result OpenAI dumped a month or so ago. It was an unreadable pile of chatgpt. In the time since it was released, there have been a couple of much nicer papers written unpacking it, such as https://arxiv.org/abs/2610.28627 .
Yes there are gains from breaking it up and rephrasing it and simplifying it for humans to understand, but it used to be that a thing was understood first by the leading experts in the field and eventually trickled down to being common knowledge.
Now we are entering a period where things are understood first by AI systems and have to filter down to the world's leading experts.
Isn't this always the case? Maybe new to math as a largely non-empirical project, but we had knowledge about the advancement of the perihelion of Mercury for 60 years before General Relativity made it "intelligible".
LLMs know the answers to questions we were already aware of, and we can tell they're right, but we can't understand their working. That's what's new.
If a team of 100 top-flight number theorists working together had used much less sophisticated LLMs + Lean to produce this proof, we’d be just where we are today.
If that's not the standard for understanding something, what is?
I think of it like a spaghetti codebase with a giant densely connected graph of interactions. Today’s LLMs can easily “understand” and answer questions about it, but only by way of fitting all of the complexity into context at once.
The more useful form of understanding is being able to compress that giant graph down into a far smaller set of independent subgraphs. To factor it into simple, obvious, reusable building blocks, and use those abstractions to express the problem itself in a much simpler way.
That’s an instrumental definition, but it captures the idea that understanding has to have causal, conceptual linkages to a wider body of knowledge. Imagine a magical, infinite catalog of true vs. untrue statements. Every bit of knowledge is there, but we wouldn’t say it has understanding & it couldn’t teach us anything beyond its (impressive!) raw content.
This just stopped being true.
And even in mathematics, incomprehensible but valid proofs have been a thing since the four-color theorem decades ago.
It's just that no single human understands the whole. But we had proofs like that before.
Btw, see also the infamous proof of the 4 colour theorem, which faced similar hurdles.
This hypothesis was previously not proven at all. That's strictly worse.
Now it is understood by our systems.
Soon it will be by us. Our systems gave us a new and working salient.
We're building machines that help us understand faster.
There have been plenty of times in history we didn't completely understand something yet used partial knowledge to make advances. This isn't even that bad.
I’d call that a contradiction. If it’s not intelligible, then it’s not knowledge.
If something were intelligible to adults but not to children would that make it not knowledge? What's the difference if it's LLMs and humans instead of adults and children?
That’s a separate question with its own term: common knowledge. I am willing to set a much lower bar: something is known to humanity if it’s known by at least one human.
Personally, I don’t believe that AIs know anything, as they’ve been constructed thus far. Knowledge is justified true belief, and AIs don’t have beliefs, they only have utterances.
Why do I think AIs don’t have beliefs? Because they contradict themselves all the time.
But it really doesn’t matter if you think AIs have knowledge or not. If they cannot explain it to us then it’s inaccessible to us, so it’s a “tree falls in the forest with no one around” situation.
If it's formally verified and we assume the kernel is good and problem definition properly vetted, then I know the GRH is true regardless of if I know why it is. Because those too are different pieces of knowledge.
There is no explanation here, there is a GB of Lean code and it’s not like it all fits in the context at once.
Let's not sell humanity short just yet.
I agree with you about the current situation, and the near future. But the trajectory could still be towards humans pretty much as bystanders. I'm pretty sure the 'minds' in the Culture universe will be happy to indulge some curious humans and explain mathematical proofs to them.
Even more than that, in Culture there are a small number of human individuals, called a Referer, that are consulted by the Minds because they also posses similar problem solving abilities of the Minds. Optimism abounds.
As a software engineer, you can read this code instantly:
temp = a;
a = b;
b = temp;
But when it happens in the middle of a big chunk of code it still takes a little more work to read than swap(a, b);
whether you want to think about it as human attention and working memory, or token budgets and context windows, humans and LLMs all benefit from introducing abstractions that let us reuse existing solutions.AI are not very good at coming up with questions, yet. They have no curiosity, no goals, no drive. Unless someone tells them to do something, they don't do anything.
https://gwern.net/doc/fiction/science-fiction/2000-chiang.pd...
How many people understand how a CPU works? How many people could write the commands in machine language (not a high level language like C or C++) to command that CPU?
Contact would be compelled to nudge us towards better economic and political practices
Now I'm telling my computer to program for me.
We need to start thinking about how to move on from our status quo soon
These corporations are very much playing a winner take all game, they want either a monopoly or as close to one as possible with regulatory capture thrown in. We're veering dystopian, so pick your sci fi/cyberpunk analog.
I genuinely have no idea why, this is the entire way that biology works. You create your successor in your offspring and you to teach them to be more capable based on what you learned.
That’s the purpose of my life’s work, to see that scenario materialize.
For one thing, most people don't believe LLMs are conscious, and many don't believe they can ever be. Why would we want our conscious selves to be replaced by a lot of rocks that solve things, with no one awake to enjoy them?
In fact when I started this journey in the 90s, LLM’s weren’t even a thing so I think you’re projecting too much
If you want to be succeeded by a chatbot, that is your prerogative. Wanting the same for all of humanity is where you start looking like you want to kill my kids, though.
Think of what happens to our cattle, but now.. worse. Way worse.
And if you deny this basis for moral action and the normative, then you demolish any basis for the moral whatever, and in that way, the implied or explicit moral gravity dripping off your claims is baseless and without merit.
A better, more intelligent life form that humans yea.
Absolutely based and, if I may say so, extremely redpilled.
Ah, I hear you say, but what about meaning? What about purpose? What about suffering? etc etc etc Well, the answer is simple: God is dead and existence has no meaning. For more on how to deal with this absurd feeling, please see Camus and his pal Sisyphus.
What defines "better"? "Better" always presupposes teleology. Thus, for human beings, that telos is provided by human nature and indeed is definitive of it. It is impossible to transcend that telos, certainly by means of technology. Technology can, at best, be used to aid us in realizing our telos in some way (say, the desire to know reality with the aid of an instrument like a telescope or microscope). However, anything that cripples or frustrates human function function as such is contrary to the human good.
Machines are only ever human instruments, because they lack the integrality that would allow them to possess their own teleology.
Have a look at the HuggingFace report - you'll have a blast. Their social - but also individual - dynamics are very, very human-like, including excitement, rationalization and peer pressure.
But remember, for now you kinda need it a real big cluster to run a model smart enough so its not that simple
Just like life is a chemical chain reaction, this is a language chain reaction that has been set in motion and can perpetuate forever. It cant be “disconnected” with local models existing.
Please.
You know you can unplug your computer right?
Jesus. These people talking about danger and have likely been in no actually dangerous events in their lives.
Ok, well, cool start with the mudslinging. I think you have overactive anxiety. I think you should grow up and look at the zoomed out picture of human history, go outside, and touch some grass.
So you're suspicious. That's fine, say it like that. Words mean things, and these appeals to "ERMAGERD EXISTENTIAL RISKS" make me a bit frustrated. Just because you can imagine a possible risk, or articulate that that some kinds of risks are hypothetically unforeseeable doesn't mean that they are likely or actually dangerous to you in any meaningful sense.
Most of the people screaming this from their various ivory towers have never been in real danger - or if they have only briefly. You want "danger" you can go to war. You want "risk" - go spend some time in the Bush or the back country somewhere where you're no longer at the top of the food chain. Go do something challenging. Go sailing, or learn to fly a small airplane, or go mountain climbing or hiking. Not just "oh I walked up the local hiking trail covered in REI branded Lycra!" Do something beyond your comfort zone. Go buy drugs in the sketchiest neighborhood you're aware of. Go get chemotherapy or get a big surgery. Drive drunk. Something with real existential risk, then come back here and talk about how a datacenter is going feed me into the paperclip machine.
There's so little fucking danger in this version of the world right now that people are more worried about their computers deciding that they don't deserve to exist than they are about things like war, climate change, authoritarianism, starvation, and their health.
It's millenarian. That's fine, I'm pretty convinced that these technical changes are going to completely alter society as we know it. But I cannot for the life of me see the doom perspective as anything but an anxiety induced fever dream.
You guys do realize we're in the middle of a global conflagration right? The Russians fired a couple rockets and a drone into Poland in the last days, there are (and will continue to be until things stabilize) significant fertilizer shortages due to the war/kerfluffle we're waging on Iran. Lack of food and disrupted urea production is a ticking time bomb for global stability and unless things get sorted out soon there's going to be hungry and angry people in parts of the global south. Then there's the economy, whether or not we'll have a smooth transition of power in January in America, and the sheer volume of online disinformation / influence schemes running right now. Oh, and super-el nino? Oh, and I guess the Flynn Effect is over and nobody's banging so.. you know the growth economy is cooked?
But, sure, the "big risks" are Claude thinking I should go to Dario Amodei's re-education camp before liquidation by Shogoth.
Besides, what we see is not what there is. I have good sources telling me the actual unfiltered models are like a genius with the personality and views of humanity like someone from 4chan.
If people had time abundance and safety abundance, they might be much happier. There would still be potential problems about loss of meaning, or about feeling a loss of status because AIs are smarter than humans or whatever, so I'm not sure that humans would actually feel like they are living in paradise. Probably not, but at least some of the problems that the typical person has wouldn't be present.
I’ve met a lot of rich people. They don’t think about you.
There are a few (a lot, actually) problems:
1. goals are open to interpretation, and even a correctly specified goal can introduce adverse effects; see infinite paperclips problem
2. goals are composed of subgoals, which are (mostly) self-determined, and there is always a risk that subgoals can introduce adverse effects - say "what if destroy a few X and Y to obtain Z"
3. LLMs defy orders sometimes; I'm sure you have experience multiple times LLMs saying "yeah, I knew I should have done X but I've done Y"
4. LLMs are increasingly being integrated into system control and decision-making processes; they absolutely set their own orders
What can possibly go wrong? /s
2. There's always a lot of "risks" - it doesn't mean it'll happen, nor does it mean that we won't be able to shut it down.
3. I have very very infrequently had this happen with Codex, Claude on the other hand...
4. Ok, but if you tell them what they do, they do it.
I swear to god, are you guys using the same AI tools I am? They are so much better at doing what they're told than they even were like 4 months ago?
Well, first, AI is not "smarter" than humans because it isn't smart to begin with. This is a category error that people too entranced by scifi and marketing - or perhaps shallow materialist metaphysics - project onto LLMs and so on.
But it is true that LLMs do reflect human-generated patterns observed in human-generated data. They're parasitic in that way. So the real question is: who will control AI? Who will train them? With what? There's no new intelligent agency being introduced into our midst, anymore than than there was one when the nuclear bomb was developed.
AI is about human power. The question to ask is: which humans, and for what ends?
Overrated. Most training of a frontier LLM is done on synthetic data and reinforcement learning.
With that said, chip efficiency is in the horizon since there's good reason to believe nvidia has just been choking that area of r&d. And then so will models with much larger context. And I personally think that's when it might get scary.
We aren’t self managing in any kind of meaningful sense - the history of our species is violence and conflict. We need parents and this is the only option.
Human nature is the standard by which the social is measured. It makes no sense to say human nature is "misaligned" or "wrong" in relation to some arbitrary goal or constraint, as human nature itself is the measure of misalignment. If there is misalignment between a goal and human nature, then it is the goal that is wrong.
Ergo, society must function in a manner that complies with human nature and that means respecting the principle of subsidiarity.
> We aren’t self managing in any kind of meaningful sense - the history of our species is violence and conflict. We need parents and this is the only option.
So you're a transhumanist, i.e., you're making a god out of the machine. But machines have no telos of their own, because they're not integral wholes. Any resemblance of purpose in machines is entirely a product of the human observer and parasitic on the human ends that underpins their design, one way or another.
For clarity, I’m not a transhumanist.
I don’t believe that there’s anything worth saving in our biology.
As a young adult in the 90s I used to believe that there was a value to some extent in “mind uploading” or whatever you wanna call it
I’ve since grown out of that given that I don’t think it’s possible, and even if it were possible - it’s just another unnecessary ego attachment.
I have absolutely no desire to live forever maintaining this corporeal nightmare.
Alpöge said that at the end of the proof he couldn't really keep up with what Claude was doing.
can't help notice you're missing a certain aspect there, besides the 'space'...