There are "experts" which do divide parts of the model that are found to activate together for specific tasks, so they can be processed in parallel to join the result at the end, but it's nowhere near the granularity of a SwiftUI expert and a python expert. The difference in those things is so trivial from an abstract point of view that it would make no sense. They would be 99% the same.
Distillations also come into this but I'm highly skeptical you could make one guaranteed to only know programming and only in one programming language (especially with as small a sample set as SwiftUI relative to something like C) without its efficacy being hobbled by tunnel vision. Reminiscent of the SpongeBob episode where he empties his mind of everything except fine dining and breathing, then can't remember his name and goes insane. Beyond the basic concepts of general coding and the trivia of syntax, getting anything done requires a large intersection of disparate world knowledge and the ability to apply it to new situations.
Yes, all three together would be even better. But it wouldn’t be if you had 100x more fanfiction, mostly synthetic, generated during RL to teach a model to be better at writing fan fiction. There are real limits to the amount of knowledge you can cram into fixed-size (downstream of hardware availability) weights. For a period scaling with data was basically “free” because we had the Internet and all the books/media that humans had already created; the data was accessible and limited (at least, the parts we think models should know about) enough and top-hardware big enough that we could basically compress the whole thing.
Post-training/RL are making this obsolete because they’re more about skill/capability acquisition rather than knowledge. They can generate much more data (most of it quotidian/useless, ie an agent made a typo in batch 382829) and clearly seem to cause a kind of mode collapse even in the most advanced frontier models.
We don’t need to make LLMs forget about SpongeBob SquarePants so they learn more about bash. But if I have a question about SpongeBob SquarePants, I don’t need to hear about load bearing seams prefaced with honest caveats after a model writes 400 lines of bash to look up SpongeBob’s family.
And there is probably a lot more SpongeBob knowledge we could put into models if we wanted to: interviews with the creative staff, a SpongeEnv/SpongeHarness modeling how the art/story team work together to create entertaining kids tv, a SpongeBench measuring entertainment value, etc. If a SpongeAgent spends 2000 years in Agent University learning how to Spongemaxx we probably don’t need or want to have it spend another 2000 years writing smoke tests
You just defined a liberal arts education.
- Understanding of protocols like HTTP.
- HTML, JS, CSS, SVG, and everything "web".
- Understanding of databases, SQL, etc.
- Abstract code architecture patterns.
- Understanding the users' requests in English.
- Responding in English.
- Command line tool usage (agents/harnesses)
- Industry-specific knowledge that can be applied.
- Frameworks, SDKs, applicable libraries.
- Relevant legal requirements.
- Etc...
It would be trivial to have a pre-flight convo with an llm to guide the user thru module choices. "Build a site" -> "ok, describe the purpose" -> "local council in XYZ location" -> "that implies you won't need localization since XYZ has a monolingual government" -> "english and catalan localization please".
Right now, you prompt and it builds using assumptions, and we prompt to adjust. I think it would be great to be able to pre-load a set of assumptions.
Everyone assumes that carefully crafting a specific AI architecture with bits and pieces bolted together based on their human intuition is necessarily superior to simply using a bigger monolithic AI model. It turns out that the opposite is true, and has been demonstrated over and over again.
The bitter lesson is this: You can simply ask a frontier model to do the things you suggested, in a few terse lines of English. Dump a few lines in AGENTS.md and you are good to go.
Your approach is to "fiddle with inadequate tools" for weeks or months until you can finally attain a pale imitation of what the frontier models can do effortlessly.
It's the classic "But I can customise EMACS endlessly, why would I use an actual IDE?" argument all over.
I get it. You don't feel ownership over someone else's AI. You don't feel involved, you don't feel like you have agency.
It's like LEGO or IKEA furniture: study after study has shown that people enjoy things more if they "put it together themselves", even if fundamentally the thing is worse and/or still essentially nothing more than plastic made in a factory.
There’s no need to bring religion into this.
You don't _have_ ownership of someone else's ai, and that comes with real risks.
Security risks, privacy risks, business risk.
They might rug pull you, they might charge you more, or like atrophic, silently corrupt the answers, or code...
The labs are happy to jump on any emergent capability the scaling and training impart: generate prose, teach you things, cyber security, design, code, etc.
Do you really think that the frontier labs won't turn a popular capability, or trend they notice, into a first party tool if the ROI seems there? If it's your own private ai in your datacenter, you can keep it all secret, and not lose your business.
On the bitter lesson you're right of course:), wish I had a super computer to just scale that instead.
A) You can always self-host something like Kimi, DeepSeek, or GLM.
B) Just because you use a specific proprietary AI for programming doesn't actually bind you to that provider in any meaningful way. The authored code remains even if you stop paying them!
Of course, if you use AI as an active component in some sort of service, then the EULA, rug-pulls, etc... suddenly start to matter. That's a different story.
That paragraph sets me off. I’ll take Vim and Emacs over VSCode and Eclipse any day.
And of course the neural network series.
Instead we ended up with no finetuning. We give audio snippet to 2 AsR models, take 3 best transcriptions and ask the LLm to pick the best based on the context. That produced significantly higher accuracy in how an agent understands the users.
The problem is that “finetuning” was a 2023 AI FOTM associated with products/demos that were almost exclusively using it for LLM character role-play/output style purposes (ie not in actual systems where they served a more functional role).
This made people think you could train models without replay/real evals by yoloing it with SFT (this is partially an artifact of that era being much heavier on autoregressive training and not so much evals). You really can finetune and get results but you have to treat it like a small ML training run, with real evals, and more intentionality than just “more examples”.
You can find pretrained and -instruct models on huggingface that clearly demonstrate what specialization/staged training runs do.
I’d be very wary of conflating finetuning with specialization/extending a model’s capabilities in general.
Turns out the world is made of simple, specialist processes, not generalists trying to achieve them. Adaptability may be of great benefit in evolutionary terms or for a walking anthropoid, but the majority of biology, chemistry, and mathematics rely upon specialist process for good reason. See also the old trope about robotics: that's what you call it before it works, otherwise it'd be a dishwasher.
The upshot is: use a generalist to create a simple solution once, and scale that. Don't deploy the generalist at scale, that's a waste of resources and an inefficient solution.
If I had to guess, the weights necessary to encode "how to program" are much larger than the final step of "output python."
ie what everyone asking for this fails to immediately realize.
This is roughly what multi-agent systems are built for.
This is possible with models too, but "making one on the fly" is much easier with agent coordination rather than model weights, since they all speak the same language.
There is an IBM Mainframe vs Google Distributed system division here. Like Seymour Cray said - two oxen or 1024 chickens.
Chickens are harder to harness, so a lot of my work is in sled-dog territory for agent harnesses & command structures.
I think I disagree. For some things, maybe that works - but think of a multi-agent system where one agent understands the code, and passes it off to the reasoning agent to figure out what the bug is. This system is going to suck. Because encoding enough info to figure out what the bug is would just be dumping every single line of the code.
So say agent 1 (reasoning) asks agent 2 (swift) to explain what is happening in File.swift. Anything agent 2 passes to agent 1 short of the entire code is a lossy transfer - and then the bug gets missed.
https://linux.die.net/man/1/ls
I'd always thought we'd eventually hotload loras or MoE experts.
It would certainly be useful on the robotics/VLA side of things as well; more limited mobile hardware, download and load/unload new skills as needed.
Tbf I also don't really care what facts my models have baked in (for llms at least). I care most that the model understands general logic and then general knowledge of some level is secondary. Reason being is that everything is RAG'd in anyway.
Models spitting out well established facts is cute but I don't really ever want to rely on say "electronics knowledge" that exists in a tenuous and vague form in the model weights.
Humans write books (and datasheets) for a reason. Books are RAG.
I think the vast majority of people do want general purpose models. They want to be able to ask it any question, or ask it to perform any task, and for it to do a decent job at it.
I agree that it's really hard (maybe even impossible) to build something that's everything for everyone. But your average (or even above-average) LLM user doesn't want to choose from a catalog to stitch together a model that does just what they need.
I do think for certain domains this is useful and will make sense: the model backing a coding harness doesn't need to know about the politics of 400BCE Rome. But I'm skeptical that many software developers will want to do what you propose, picking knowledge bases that are tailored to their current task or project. And at any rate, for web-based chat interfaces, most users just want to type a query and get an answer.
The vast majority of people listen to the music of Ed Sheeran and think that it is good.
Hearing has volume, direction, pitch, it's spacial processing etc
I can change which one I hear on a whim, so much that I can even get Brain Needle and Green Storm out of this video.
I think you're right that current architectures don't compose like that - but I feel like that's a result of the focus on MOAR DATA, and a "race for AGI" - if we set those ideas aside, a more composable architecture seems very possible.
Part of the problem of this is likely that the deep meanings of words you might use in chat to describe a business problem or task that you wish to see implemented are essentially inseparable from scenarios in which they are used.
Putting aside the bouba/kiki effect and anything like it, complex words only have meanings from usage. That usage is built on grammatical structures that also emerged only from usage.
(This is something I was taught as a sort of fact but I gather it was basically abbreviated Wittgenstein? … who I cannot claim to have studied)
So what you're looking for is a language model where fundamental word meanings are encoded without the weight of knowledge of where they come from. This is plainly difficult, because complex words are used by extension and analogy, and these days, many are neologisms or portmanteaus, even ephemerally — developed and discarded within a single context.
Reasoning about language itself to its full meaning is quite hard.
Like my favourite word of the moment: "obscurantist". You see that and you have a glimmer of what it might convey. But why do you? How much of that comes from explicit grammatical knowledge of suffixes, and how much from simple experience of using words like obscured, informant, attendant, dentist, artist?
So a language model might be able to deduce what "obscurantist" logically means when applied to a tract or to a person. But without lots of parameters covering its use, could it properly grasp that in some circles it would be pejorative to the point of being deeply offensive?
I think the best hope for your pluggable knowledge base idea is model delegation: strong reasoning models that know how to dictate to smaller specialist models and draw conclusions from their responses. I find myself wondering if there's any way that can be done the same way that, say, Gemma 4 12B's integrated vision encoder works — within shared weights, somehow, without them to speak in some intermediate language, like a partitioned brain. But I find it difficult to believe that is pluggable at all.
Another approach would be to have basic coding and reasoning model and then load specification for language and libraries into context, it could work for self-hosted models, but I don't want whole specification of the language to be send to API and waste tokens on that.
THe original MoE paper from Noam Shazeer et al. is worth a read on this bit, though the paper is admittedly pretty dense. But TL;DR is that each expert layer is learning highly abstract, localized structural and syntactic patterns in the data to minimize the loss function, and its doing this token-by-token (which in some cases may have some domain clustering, but that's just incidental).
When you start batching your queries, even if they all seem like theyre in a single domain, if you visualized the activations you'd notice that most if not all of the network is lighting up on the batched forward pass.
And so far even the biggest model doesn't seem to have a working Make No Mistakes module, so maybe that's not needed
Sure, 3.8 maybe it's better now, but an accurate comparison would be with a new Gemma4-31B iteration (that doesn't exist).
Assuming you can tweak the training data, regenerate qwen3.6, and get a better coder, then presumably you could have variants - e.g. qwen3.6-swift-27b and qwen3.6-python-27b. Or maybe all coding is too intertwined and you can only get splits like qwen3.6-research-27b and qwen3.6-coding-27b. Which isn't quite my pluggable-models dream, but it's a step closer.
But maybe the difference isn't the training data, it's the architecture, in which case pluggable models is probably not possible.
Tell me you don’t know how llm work without telling me you don’t know how llm work. That’s not how they work!
And then ideally, make it pluggable so I can pick what I want from off the shelf components, but if that's not possible - then just train up as many variants as you can so we can all pick the best variant for our current need.
Aren't you describing RAG or even MCP servers? Heck, nowadays you get that also with agent skills and specialized tool calling.
Definitely not MCP, as that pulls info into the context. Unless contexts become REALLY big so that I can add 10B in swift knowledge, that's not gonna help me.
Possible RAG? I don't know enough about how that works, but I think that's not quite it either. I don't want to import facts like "the swift standard library contains a reverse array function", i more want to import knowledge - e.g. the parameters used to generate the text to reverse an array in swift.
Tool calling wouldn't do it either. You'd have to encode every single possible bit of useful info into the tool call, and the tool response would have to encode every piece as well (variable names, function scopes, types defined in other files, etc). E.g. how does it find a bug, if you have to pass understanding back and forth between the brain that understands debugging and the brain that understands THIS code?
I think having unused or rarely used weights doesn’t influence the results as poorly as RAG injecting irrelevant facts.
It sounds to me like some sort of “dynamic MoE” where you can add/create or remove experts on the fly.
I think what you’re describing is the closest approximation we reasonably have right now though.
There is nothing optimal about needing a few billion more parameters to be able to piece together probable answers that can be asserted by querying an oracle.
> I think having unused or rarely used weights doesn’t influence the results as poorly as RAG injecting irrelevant facts.
Those aren't free. The more parameters you add, the higher the computational cost required to train and prompt a mode.
And all for what? To piece together info that you can just query from a data source?
> so LLM performance in swift benefits from pythonic patterns
What I hear you saying is that the best way to make a swift-trained-only LLM smarter is to train it on some python too. And then with an infinite parameter budget, every other programming language or really any other data you train it on makes the model smarter - I accept that premise.
But in a fixed parameter budget, what is better? training on 50% Swift + 50% Python, or 50% Swift + 50% Rust. Because if I am doing Swift programming, I want whichever of the two is better for Swift. If I am doing Rust programming, maybe I want the model trained on 50% Python + 50% Rust. Sure, it would be smarter if you tossed in the swift code too - but we have a budget to stick to.
Now is it possible to make those pluggable? i.e. can you take a model trained on 50% python, and layer on 50% rust OR swift depending on what language you're using? Probably not right now, but maybe one day?