Or maybe it can; the early image generators famously did output horrific images sometimes because of their limitations, but these outputs qualify nonetheless creations.
Now the interesting thing is that it is humans who evaluate them as horrific, worthless or interesting.
Certainly AIs could pretend they have that sort of opinions and judgement, but hopefully humans will remain more interested in the opinions of other humans.
Online, how would they even know? Offline -- either we give up on the internet, or the offline discussion will be heavily influenced by what AI said online.
Humans have cognitive limits. Humans hold a specific slice of all knowledge at a time and necessarily limits what correlations can be leveraged. This is similar to "context windows" in modern AI. Human minds are wildly variable in efficiency. AI is has constant inefficiency, largely controlled by the speed of light, physical RAM, and processing power, with superior (no cost) "thought" context switching. The AI also has superior accuracy without the need for "concentration" because the AI is processing on a Turing machine with the limited operations that it's superior at in terms of time and energy - eg find an element in a list of 1 million elements.
Neurons and synapses change, especially synapses (the connections between neurons). Some neurons have a few connections to other neurons. Others have hundreds of thousands of connections to other neurons. These grow and change. "Estimates are between 100 trillion and 1,000 trillion neural connections" Is it finite? Depends on your definition of finite. If anything countable is finite, then it is finite but the number may be practically uncountable. Is it fixed? No.
- random,
- from the environment,
- from other AIs.
That increases the space considerably.
In particular, note that your argument also applies to humans: A human, being finite, can only produce finitely many things. But you add other humans, the environment, and pure chance to the mix, and it'll be pretty infinite for all practical purposes.
Maybe that is the limits of my imagination, but it seems unwise to take humans completely out of the loop. Having mathematicians around that can understand what it is doing so that we can utilize that knowledge in a way that we can have some confidence in seems important.
It sounds increasingly dumb to bring this up given how advanced models have become, but it remains true - they are still fundamentally token prediction algorithms. And that imposes hard constraints on their potential. These constraints are relatively academic in our current world because there's just so absurdly much information and no human can hold more than a negligible amount in his mind at once. So they'll certainly be able to connect many 'obvious' dots that we don't see simply for a lack of carrying capacity, but they're not going to discover concepts that do not clearly follow from the comprehensive knowledge of an era.
1. LLMs aren't run deterministically - they use a bit of noise to improve the output. So even with the exact same input you're going to get a whole distribution of outputs.
2. Feedback from the environment. LLMs (and humans!) work best when they can test ideas. That's why coding agents work so well. If you ask an agent 1000 times to just think of a compression algorithm, it'll probably spit out very samey answers just like a human would. If you give it a test corpus and let it try out different things it will absolutely be able to come up with novel ideas.