While they’ll never have the same subjective experience as humans, what stops an LLM from applying similar lines of thought* in a manner that results in a novel conjecture?
They are prediction machines, and so are we in a way. We can give them nearly limitless resources to scale their predictive capabilities. We have billions of years of training baked in. They distill directly from our knowledge and can walk down paths that no human has before.
It’s silly to say they’ll never do anything novel.
At their current capabilities, it sounds like they are already capable of being a specific type is research assistant. What will that look like in 10-20 years?
AI can be totally biased...
The fact that it can spout bullshit all day long to a human who can be tired and would actually act on the said bullshit, is not very comforting...
For example, an LLM could confidently declare something a tired human would take as a fact, but would backfire in a real world.
One thing is that an LLM can never assume, or find out, an inconsistency in its training data. Novel ideas often require correction of existing assumptions. As far as I understand, it is impossible, by design, for LLMs to contradict what is in its training data.
For example, an LLM trained on the data from an internet comprised of people who believe in the earth centric hypothesis can never say "Hey, that cannot be correct", or come up with the heliocentric alternative
But maybe it is not applicable to pure Math...
You state this as a fact - are you aware the question is unresolved?
EDIT: I'd love to know why you're downvoting me for stating a known fact.
I can confidently state that GPT-5.6 Sol is not experiencing the same reality as me. They _might_ be "experiencing" and I personally think they are, but their reality and experience is not the same as ours.
Maths was already infinite, it's still infinite, but who wants to spend all their lives changing rooms inside Hilbert's Hotel?
Most of us aren't Terence Tao
How's It Hanging, Brother?
Sure, it's not a breakthrough that opens new roads in mathematics- is this where the goalpost has moved now?