undefined

points

[-]

> General purpose models are always more robust and generally better than smaller narrower models

I feel like this is just the marketing conflation of AI=LLM, versus regular old ML? We're never going to need to deploy a full reasoning model on a low-power device just to do some fancy image recognition in the field. Specialised ML models are just intrinsically able to be a lot more efficient than their generalist equivalents

by plastic-enjoyer1 days ago|

prev|

[-]

> General purpose models are always more robust and generally better than smaller narrower models.

What do you mean with more robust?

by ACCount371 days ago|

parent|

[-]

Less weird unexpected failures, more innate ability to handle edge cases gracefully. Quite important when you're running high on automation and low on oversight.

by plastic-enjoyer12 hours ago|

parent|

[-]

This may be speculative, but couldn't robustness emerge by having a number of specialized models, that are interconnected and feed into each other? Are there any arguments from ML that would speak against this?

by ACCount3711 hours ago|

parent|

[-]

That does work. Even if you drop the "specialized" part. Ensembles of the same architecture at the same scale trained on the same data do outperform a singular model of the same line - especially on corner cases. Successes of an ensemble correlate stronger than failures do.

The usual argument against is that if you have "a number of specialized models" that perform well in ensemble, you can take that ensemble, and distill it into a single larger model (dense or integrated sparse, like MoE), and get the same improvement in performance with an efficiency win.

This works because having those "specialized models" duplicates a lot of the highly conserved "low level" wiring that's required for a model to function at all. As such, you end up running a small scale version of the same "backbone" computational processes many times. "Merging" those models into a larger, denser model allows for a singular strong "backbone" to be used for everything.

by 1 days ago|

parent|

prev|

[-]

deleted

by ACCount371 days ago|

prev|

[-]

You're getting downvoted, but you're completely right. There are very few cases in which narrowing a model down is buying you anything worthwhile.

It seems like for LLMs, "general intelligence" is expensive, but "one more domain" is fairly cheap.

by Danox1 days ago|

parent|

[-]

The demand for smaller models and single purpose AI agents will only get bigger in the future. It’s no different than going for mainframe computers back in the day to personal computers. It’s inevitable…..

by andy991 days ago|

parent|

[-]

Has nothing to do with demand, it’s a question of the most efficient way to train a good model. Narrower models are in general worse than general purpose ones, so you’re better using the largest general purpose model that fits in your compute budget vs trying to somehow remove capability or knowledge and assume the model stays as capable for the task you want.