This is a big leap. So far LLMs are really good at turning training data into accurate results. The more data, the better. LLMs are very, very bad at making intuitive leaps based on the 'shape' of sparse data- a technique that is, to be fair, pretty rare in humans as well, but essential to progress. Maybe they can make up for it with brute force and the precision and breadth of knowledge that only an LLM can have, we'll see.
replyI don't think we've nailed the architecture that will allow things like generalized self directed training, yet, which is what would be needed for something like 'make fusion better'
Are you sure about that?
replyThey seem to be able to make intuitive leaps pretty well. They need to make the same leaps over and over, though, because they lack online learning, so the discoveries only persist after the next training cycle. Context only goes so far.
We're pouring billions into solving that, though, so I would be surprised if we don't get there soon.