Anything that required knowledge / familiarity mmBERT and ModernBERT post-trains performed much worse.
So it seems like they did some kind of useful expansive pre-training.
Things that were Qwen or Gemma Diffusion did better at those kinds of tasks but were generally pretty inconsistent in terms of whether they could succeed repeatedly (and be stable + reliable) on the many types of tasks that are in the cookbook part of the Jev docs.
If you ask Jev similar input + questions, it's pretty stable. And does a reasonable job on a lot of questions.
This one public benchmark (the only I've seen) seems to give the open versions way too much credit. It wasn't my experience at all.
It gave my a false wrong sense of what might be required to get it working for something at work to avoid needing a new subprocessor as - at least on Cloudflare / OpenRouter Jev is third-party not hosted.
We are working on running our own benchmark of Jev and some of the other models. Our use case is classification that runs in a UI. Currently LLMs have good accuracy, but are too slow (and expensive).
Jev not being available through a cloud provider (Bedrock or similar) makes it more challenging for us to start testing and rolling it out.
So it's a new subprocessor. Which can often be painful to onboard, especially if not compliant according to your needs.