MoE and such are basically performance enhancements, they don't make the model smarter.
replyseparately trained experts can surpass performance in their activated regime and DOES result in a smarter model, the Claude system cards talk about this and eg there is https://openreview.net/forum?id=iydmH9boLb to read...
replyPerformance enhancements are huge though.
replyIf you can make the existing model faster, you can then save your inference budget to then make your model bigger, which then makes it smarter.
A lot of how smart the models can be comes down to budget. If you can make your existing thing cheaper, you can instead make it bigger for the same price.
> to then make your model bigger, which then makes it smarter
replyThere's diminishing returns and at some point making a model bigger makes it dumber.
Not really “smarter” though? It’s just a big probability engine.
reply(Not trying to flame bait or anything. I just wouldn’t call LLM as exhibiting intelligence. It is great at making connections based on probability but doesn’t have a semantic understanding of what it is doing)