I feel like this is just the marketing conflation of AI=LLM, versus regular old ML? We're never going to need to deploy a full reasoning model on a low-power device just to do some fancy image recognition in the field. Specialised ML models are just intrinsically able to be a lot more efficient than their generalist equivalents
What do you mean with more robust?
The usual argument against is that if you have "a number of specialized models" that perform well in ensemble, you can take that ensemble, and distill it into a single larger model (dense or integrated sparse, like MoE), and get the same improvement in performance with an efficiency win.
This works because having those "specialized models" duplicates a lot of the highly conserved "low level" wiring that's required for a model to function at all. As such, you end up running a small scale version of the same "backbone" computational processes many times. "Merging" those models into a larger, denser model allows for a singular strong "backbone" to be used for everything.
It seems like for LLMs, "general intelligence" is expensive, but "one more domain" is fairly cheap.
How are small isolated language models more similar to that than MoE in LLMs?
The original Mixtral paper [0] (in the "Routing analysis" section) found:
"surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic"
A quick skim of more recent analysis on MoE shows that this hasn't changed. MoE models do appear to work, but don't appear to do what the name implies, if anything they're routing based on the structure of the text and not the semantic content (and we're still not entirely sure what they're doing).
I have a long-ass post about how this could be implemented. https://old.reddit.com/r/VisargaPersonal/comments/1um9uyv/st...
Why didn’t OpenAI release a math specific model? Why not a literature specific one? Why do they instead have generic models of different sizes? And how did all labs converge on this?
Why does Fable just not train on non cybersec and non biology data but instead have clearly costly and annoying classifiers?
"This will never work" is a pretty confident assertion for a field that's so young and rapidly evolving.
This is what the parent said. AGI won't rise out of AlphaZero and AlphaFold in the same way AGI won't rise out of Houdini chess engine. This is the industry consensus.
That's a straw man. Nobody thinks AGI will rise out of domain-specific systems. The question is whether domain-specific systems are necessary for AGI.
Of course, the problem is that AGI isn't a well-defined concept. But if we define it as achieving superhuman performance across several hundred domains where there are objective measures of success, it doesn't seem far-fetched to predict that it will involve some general reasoning system paired with a bunch of specialized modules.
> I don't foresee AGI arising out training bigger LLMs
I agree that AGI will involve tool usage but not only involving domain specific AI models.
But lets try to find the discriminating point in the discussion - do you believe AGI will necessarily involve training bigger LLM's or not?
I believe they are necessary. WBU?
No, I don't think LLMs are necessary for AGI at all. I think there are multiple paths to AGI, some of which involve LLMs and some which don't.
This bet is too early.
> Why didn’t OpenAI release a math specific model? Why not a literature specific one? Why do they instead have generic models of different sizes? And how did all labs converge on this?
Because they have a very early product and they could train it, brute force, with access to an extraordinarily large pool of money. So did all the other labs. Because it was thus easier to scrape everything rather than spend enormous effort (with tools that did not really exist) to partition the training set. Any number of other "because"s.
It's just what they are doing now and it showed the earliest results.
LLMs are still less intelligent than rats, which have tiny brains.
The point I am getting to elliptically is that larger models aren't necessarily the solution. They are one solution pathway.
It is fully possible (I think actually likely) that the ultimately successful path for LLMs will emerge from the pressure of keeping them small, not making them large. Very small, domain specific models could well outperform large models in their domains, and they might even show that domain specialisation is not necessarily much of a limitation, just a useful impetus to stay small. Like how rats can drive those little cars.
(I think the frontier models are potentially already too big. Can't prove it or even close to it, but it feels like this is going to be a story.)
The answer to your question is “because the market isn’t big enough”, not because it doesn’t work. Why would knowing about 2019 internet memes help you in any way at coding?
99.99% of the knowledge an LLM has is useless for a given scenario, the hard part is knowing what the .01% that’s needed is. Knowing as much as it can means the model can handle edge cases, turns of phrase, etc.
Put another way, it avoids overfitting. That’s basically the insight that’s given way to the current AI boom.
https://github.com/Brainrotlang/brainrot
"Brainrot is a meme-inspired programming language that translates common programming keywords into internet slang and meme references."
They did and retracted it because they found that GPT 5.5 beat codex pareto optimally. This keeps happening.
> because the market isn’t big enough
Huuh? market isn't big enough for AGI? The parent suggested that AGI would emerge from this process.