François Chollet made an interesting geometric analogy on Twitter. We tend to view Intelligence as something unbounded, like the height of a pole. But it could be something closer to the roundness of a polished sphere. Once close to the limit, incremental improvements are less and less significant.
replyThe intuition we have about the nature of cognition and its whole shape and limits is almost entirely built by looking at ourselves and our peers, comparing that to an ant. I feel this is largely naive and misguided.
I've been interpreting AI task solving capabilities as a search problem: being smarter lets us prune more of the tree and try out different things more efficiently. I think this fits into the sphere analogy quite well: at the limit, we need to experiment, use the experiments to create new abstractions and build on them repeatedly. This doesn't necessarily require more raw intelligence, but the time or resources to find how the pieces fit together.
replyThe cap will be reached when LLMs run out of original material to train on and the large majority of their training material is AI generated content.
reply