Unless the domain-specific training has significant negative effects on performance in other domains, I'm not sure there's a strong reason not to just train one model with all the capabilities. Maybe there's a monetization play here: release a generally capable model as open weights alongside specialized models that you only provide API access for.

Imo, we are going to see a lot more specialised models in the near future as pushing the frontier gets harder. Qwen 3.8 27B for example is a pretty rubbish chat and general world knowledge model, but is stellar at coding (for it's size). Its small enough anyone with a modest gaming PC (16GB vram and 32GB of DDR5) can run it at speeds good enough for overnight coding sessions with an PR to review at the end of it.

Eventually, I hope we'll see many more specialised small models that run nicely on consumer and prosumer hardware. Like LLMs specific to income tax law for UK residents, or LLMs specific to building power delivery circuits for PD USB C devices. There probably is a point where the niche-ness does become to niche to justify the post training, but again, small models means much less training required.

Kind of begs the question if this even needs new/better data? Could you take the foundation model that's trained on everything, and then run a final fine tuning training run on just the set of foundation training data that you want the model to specialise in? Eg, extract just the training data that is about UK income tax law and then run a few epochos on just that? Any ML people know if this is a thing or if not, why it doesnt work?