Neural networks are function approximators. They're great because they can be trained to approximate almost any function.
What if, somewhere out there in platonic space, there's a complicated function that represents what a philosopher might call true intelligence. Perhaps it's structured totally differently than a neural network and shares no common architecture with a transformer.
Transformer-based LLMs are likely completely different in their fundamental structure. But perhaps the neural networks we have today are succeeding in intellectual tasks because they are beginning to approximate it. Like a bitmap approximates a vector image, it's not the same; the neural network is merely made of a bunch of repetitive simple elements, tuned to imitate the real thing. A higher resolution bitmap image will be a better approximation but still never become a vector image no matter the pixel count.
But at some point, the neural network might be good enough at intellectual tasks that when tasked with searching function space for an algorithm that does the job its neural network does but better, it finds the real thing it's been approximating all along.
First: modern LLM architectures are quite far from the GPT-2 era (delta/linear attention, GQA, sparse attention methods, looped transformers, mixture of experts, engrams, etc).
Second: RL environments are largely autonomously generated at this point, as models get smarter they can generate even more complex environments to train in. While this does go into the "data" category, it's not human-generated or even gathered data. This gets particularly spooky when you think about things we cannot yet simulate, that models can develop mathematical methods & simulators for. For example: better/faster physics simulations for robotics RL, better biological simulation environments etc.