I think he argues more that LLMs are a form of supervision because you need a whole batch of text about specifically what you want the LLM to learn for it to be useful at that one topic. That they predict on human supplied data instead of learning purely from existing in the world and from observations.

Either way, there's something fundamentally inefficient about the whole business of training LLMs.

The human brain manages to self-organize with only a fraction of the information that LLMs get trained on. To train an LLM you need a lot of high-quality data.

There are two threads in history: firstly the compute thread leading to GPUs and AlexNet in 2012. Secondly the model architecture thread that started long before we had the compute and lead to transformers in 2017.

If the compute thread had been 30 years behind then we might be spending this century coming up with better architectures to make do with the more limited compute. However since the compute came first, we settled on the first thing that worked (transformers) and all effort went into polishing that.

There's something wrong with transformers though. No matter how many trillions of dollars get thrown at the problem, they still don't learn like humans do.