World models will definitely incorporate LLMs or at least text decoders in their final architecture, LLMs are here to stay.
replyMaybe this is a dumb analogy but is that kind of like saying “well EVs will still use lubricant! Cars that use oil are here to stay!”?
replyThe point being that the way we heavily rely on LLMs specifically right now is likely to cease to be the case, and that in the future they’ll be almost like an irrelevant component.
There's far more shared concepts between them than you think.
replyYou should check the code of both world models (JEPA class for example) and compare to GPT. Many of the tricks stay the same, representation is still embeddings, there is a loss function, etc.
The exact architecture will change, but unless there's a new discovery in that area, we've cracked the text component already. We're hitting the limits of LLMs because of the intrinsic limits of text as a medium. But the way we work with text is pretty much settled, fundamentally.