Even if LLMs or some more advanced mechanical processes were able to generate novel ideas that are "good", people won't recognize those ideas for what they are.
You actually need a chain of progressively more "average" minds to popularize good ideas to the mainstream psyche, i.e. prototypically, the mad scientist comes up with this crazy idea, the well-respected thought leader who recognizes the potential and popularizes it to people within the niche field, the practitioners who apply and refine the idea, and lastly the popular-science efforts let the general public understand a simplified version of what it's all about.
Usually it takes decades.
You're not going to appreciate it if your LLM starts spewing mathematics not seen before on Earth. You'd think it's a glitch. The LLM is not trained to give responses that humans don't like. It's all by design.
When you folks say AI can't bring new ideas, you're right in practice, but you actually don't know what you're asking for. Not even entities with True Intelligence can give you what you think you want.
Other types of problems require measurement in the real world in order to solve them. Better telescopes, better microscopes, more accurate sensing mechanisms to gather more precise data. No AI can accomplish this. An AI can help you to design better measurement techniques, but actually taking the measurements will require real time in the real world. And some of these measurement instruments have enormous construction costs, for example CERN or LIGO.
All of this is to say that there will color point at our current resolution of information that no more intelligence can actually be extracted. We’ve already turned through the entire Internet. Maybe there are other data sets we can use, but everything will have diminishing returns.
So when people talk about trillion dollar superclusters, that only makes sense in a world where compute is the bottleneck and not better quality information. Much better to spend a few billion dollars gathering higher quality data.