What are the useful applications of Jev so far? Not to sound dismissive, I just haven’t seen what people are using it for yet.
replyHere's a third-party (not Jev) showcase of things people built, which helped me kind of get the appeal. https://bentossell.com/jev/ (not mine).
replyLots of use cases!
I've personally used it for the following:
reply1. Evals (once you have your rubric defined and tuned using a reasoning model, jev can be great for running periodic evals especially those that run daily.
2. e-commerce catalog classification 3. quick search using anything as context and query mapping to a pre-defined set.
At least for 1, evils, you’d want to use a good old reasoning model to get the best eval results.
replyWhy not using a cheap LLM with thinking completely disabled ? I don't think it will be much more expensive than jev.
replyI’ve tested this with some local LLMs and their accuracy is in general better than Jev/Laya, but they are super slow in comparison as well
replyFor example, a typical/stock LLM can’t really play Doom in real time, but a Jev-like model can. Just because of latency
Of course, if you want the best Doom player, there are way better and faster adhoc models
LLM inference has two very different regimes of work: prefill & decode. You can think of the former roughly as processing a pre-specified prompt, and the latter as sequential processing (auto-regressive token generation) eg. "chain of thought". The latter is very important for LLMs and cannot be ignored; it deeply influences infra design, even necessitates copious amounts of high-bandwidth memory. Jev-like models can ignore the latter and therefore optimize much better for the former, consequently operating at both better cost and latency.
reply