The Ouro looping results are interesting [1] and they are focused more on the improved reasoning from looping middle layers rather than the parameter efficiency aspect. They train 1.4 and 2.6B parameter models with 7T tokens. The training includes learning how many times to loop on any given token (there’s an early exit module). My guess as to why (as far as we know) looping is not in frontier models yet is that, at frontier training run scale, it’s probably going to require a lot of trial and error and at-scale research. While currently they already probably have a list of dozens or hundreds of of promising ideas that don’t complicate things as much. In the other hand, Ouro’s looping technique shows ability to compete well with models with 3x parameters which seems attention-getting to me. If there’s another 3x to be had down that path. It’s order of magnitude opportunity. Btw there is a great related work section in the paper.
reply