JEPA can be applied to text as well in the sense you can use it to build text embedding models, freeze your JEPA text embedder, then train a decoder over the top of that.

OF course your decoder model could end up doing a lot of the heavy lifting if your decoder loss was based on text reconstruction methods (which almost all LLMs use for eval purposes)

For certain tasks, you do need pixel/token level reconstruction fidelity to do a lot of things. Like image generation or text generation.

For other tasks, having a coherent representation of the world that can be computed quickly and intuitively is more salient. I suspect the human mind works more like JEPA than like an autoregressive LLM. Human children train on far less data to achieve extreme fidelity on a lot of tasks generative reconstruction loss based AI/ML techniques struggle with.