That assumes scaling laws still hold up. A bigger model might end up only incrementally more intelligent.
replyThey do. Mythos kicked ass while it lasted. And what we know of the scaling law curves promises us even more gains in the future.
reply"The future" being "whenever training and inference at increased scale becomes economical". Which is probably bounded by new generations of hardware, but might also be pushed forward by algorithmic advances.
I think they're out of training data though...
replySynthetics are often used for "data amplification" nowadays. Extra compute covers a multitude of sins.
replyNot only you could: you would also want to.
replyThe likes of Mythos show that the scaling laws are real, and you can x5/x2 the total/active params and get meaningful gains. If "inference per param" gets cheaper? Up the params and get more intelligence for the same price.
Quite true
reply