These models rely on knowledge that are embedded in their weights, if a new library is released, a new linux version comes out, some new protocol succeeds the previous one, you want your llm to know about it. Sure you can just add that into the context window, but that has its own problems.
replyUnless new research, there are a few which look promising, gives a new method, training is going to be a constant cost sink.
On top of this, if you stop training, it is 6 months until someone releases an open weights model and now you are competing to give the lowest price for the same product.
Also we can't forget that this is a business that *has to* be in the global labor industry, not just a tech tool, they have to have much better models to justify the trillion dollar evaluation
it does not require training a model from scratch for it to be updated. the entire LLM training process is iterative. essentially each step (there are hundreds/thousands for a training run) yields a complete, usable model. tokens with updated data can be added on top essentially at any time in the future.
replyThat's good to know, but I think maybe the bulk of the original point is still concentrated in the "we always need new models to stay relevant" piece.
replyAnthropic has essentially trained one big model for more than one year.
replyFrom Opus 4 to 4.8 all improvements were in RL and post training. Expensive, but not as intensive.
Is Fable/Mythos a new generation then?
reply