undefined

points

by alightsoul7 hours ago |

comments

by kennywinker6 hours ago|

[-]

Only if you fundamentally misunderstand what I am describing

by alightsoul6 hours ago|

parent|

[-]

You cannot have a single purpose LLM. Every single topic contributes to the performance of an LLM on any given area. You cannot have what you want. LLMs do not work like professions, college degrees or people. Using your examples, it is damaging to just know a single programming language, because there are patterns that are more common in say python than in swift, even though the only thing you want is swift, so LLM performance in swift benefits from pythonic patterns. Programming logic is the same in all languages so by having only swift and no other languages, you remove the number of examples the LLM is trained on which degrades performance in Swift which is the only thing you want

by kennywinker6 hours ago|

parent|

[-]

> You cannot have a single purpose LLM

> so LLM performance in swift benefits from pythonic patterns

What I hear you saying is that the best way to make a swift-trained-only LLM smarter is to train it on some python too. And then with an infinite parameter budget, every other programming language or really any other data you train it on makes the model smarter - I accept that premise.

But in a fixed parameter budget, what is better? training on 50% Swift + 50% Python, or 50% Swift + 50% Rust. Because if I am doing Swift programming, I want whichever of the two is better for Swift. If I am doing Rust programming, maybe I want the model trained on 50% Python + 50% Rust. Sure, it would be smarter if you tossed in the swift code too - but we have a budget to stick to.

Now is it possible to make those pluggable? i.e. can you take a model trained on 50% python, and layer on 50% rust OR swift depending on what language you're using? Probably not right now, but maybe one day?