That depends on the question. If the question is "How do I utilize 10,000 software engineers?" then yes, a monolith is not the answer. However, this isn't a problem most companies face. Starting out with microservices is rarely a good idea either as you'll have a hard time getting the boundaries right.
So if that’s a part of the future, what’s the purpose of spending billions of dollars on large AI models? Is this just a continuation of the story that there is no moat? For the big model makers?
Why do you need a master of everything? when you can just use a much smaller specialist AI model.
If you want a model that can do X but don't care about Y And Z, intuition says you only have to train the model on data relating to X. However, early LLM research showed that doing this results in a far less X-capable model than one trained on data for Y and Z as well.
I would assume a smaller model trained specifically to produce code based on natural language input would perform at least just as well, which would already be better given the smaller size / probably lower training resource requirements. This is of course, as already said, an assumption. But generalizing typically comes at a cost, there is no reason why this wouldn't apply to LLMs, it even applies to humans.
Coding an exercise app will benefit from a model that can code and understands what exercise is. A car showroom website will benefit from a model that understands what relevant filters would be.
Nah, let’s stand firm here and not give in so easily. Parent suggests a coding model needs exercise info in case you want to code an exercise model? Absurd, that leads to the know everything monolith to do anything.
Language, reasoning, tool use, and domain knowledge.. in this case the domain is coding. Someone wants an exercise app? Great, so we look that shit up, we don’t bake it in. We don’t want to bake anything we don’t need in.
We don’t even need to explain this completely reasonable and totally normal engineering preference that small is beautiful. it’s the other side that needs to justify wtf exercise has to do with coding or why everything from twitter posts to a working knowledge of 15th century French poetry just might be helpful for disconnected domains, like say serious work in contemporary astrophysics. Back to basics, No one asked for that, No one should accept as given that this is the way anything needs to work
No, the point is having broader knowledge than just "can code" is important for building software for humans to use.
> We don’t even need to explain this completely reasonable and totally normal engineering preference that small is beautiful.
It is, of course it is, as long as that small thing is useful. My argument is not "let's put literally everything in this thing just in case" it's "broader world knowledge is important for building many things".
Also, we don't look at CVs or histories of people and say "hmm, this person could be good at coding for us but I really wish they didn't have such a broad and expansive history".
> Great, so we look that shit up
But what do you have it look up? What is exercise? What is a muscle and what is a human? How is it going to understand those references? Does it have the broad enough understanding to see where the gaps are?
At some level unless all you want is "loop over the array and sum the results into a variable called sales_total" converted into code, you have to have some broader knowledge about things.
And once you start considering that maybe it needs more than simply code, we're not disagreeing we're just discussing where the line should be. Which I've also not placed, because there's no simple statement to make on that.
Damn right we don't say that. And I don't look at a carpenter and wonder if his head is hard enough to drive nails like a hammer either. People aren't tools, and tools aren't people. You've conflated them because they are both capable of doing work. And that's exactly the type of confusion and category error that the perspective I'm arguing against will lead to.
Wanting a multipurpose tool or system is occasionally reasonable. Much more often it is very, very to close misunderstanding the nature of a problem, what a tool is, or what a system should be. The particulars of purpose and mechanism of action matter, which is why a blowtorch is sometimes a wrench but only if you're kind of desperate. It is always a very scandalous thing to speak ill of someone's best new electric banana-phone / dishwasher-piano that they want to sell you. But that just sounds messy to me, I'm not sure it will improve the sound quality or taste of anything involved.
That might mean that spending all that money on that giant master of none model might lose out to the much smaller specialist model, uh-oh…
The Bitter Lesson is that human-written rule models will always lose out to trained models in the end.
Does it? All of the latest frontier models that have made technical details publicly available are using MoE; smaller specialized units that work together.
LLMs are the powerful general-purpose CPU of the ML world. Using an LLM as a classifier is woefully inefficient compared to using a dedicated classifier. On the other hand, that LLM exists right now and you can hook it up in ten seconds. You could ask: why use an LLM when a dedicated classifier would be so much more efficient? You could also ask: why build a dedicated classifier when a sufficiently capable LLM is available off the shelf? Neither one is the right answer every time.
"AGI"