They also have ability to go deep and wide in a way that humans just can't. We have limits, get tired, distracted and biased where AI does not. I think there a lot of problem where all the information needed to solve them is there, but we just can't put the pieces together. Like no matter how many people you throw at some problems, you hit human limits and more people won't help, but AI will because it is just relentless.

>what stops an LLM from applying similar lines of thought* in a manner that results in a novel conjecture?

One thing is that an LLM can never assume, or find out, an inconsistency in its training data. Novel ideas often require correction of existing assumptions. As far as I understand, it is impossible, by design, for LLMs to contradict what is in its training data.

For example, an LLM trained on the data from an internet comprised of people who believe in the earth centric hypothesis can never say "Hey, that cannot be correct", or come up with the heliocentric alternative

But maybe it is not applicable to pure Math...

> While they’ll never have the same subjective experience as humans

You state this as a fact - are you aware the question is unresolved?

EDIT: I'd love to know why you're downvoting me for stating a known fact.