Agreed. I tested Jev on OpenRouter this past weekend and it’s “okay” but a specific classifier is significantly better. It used to require skill to import sklearn (ok, not really), but now it’s literally one prompt and upload your Excel file or whatever and you can get your classifier out. It’ll run free, instant, more accurate.
replyThis is predicated on you having training data already. I approach Jev more like Langchain -- you can prototype something new extremely fast and cheap, and if the use case works well enough, rip it out and build something bespoke. If it doesn't, you didn't spend a bunch of time curating a training dataset anyway.
replyYeah I think that's right. It's actually nice to have a better-than-nothing placeholder that can be replaced if it becomes valuable to do so.
replyI guess I'm circling toward this view. The question is, are there things that are 1. worth doing, 2. for which jev (or jev-like systems) works well, and 3. are not worth the effort to train a custom classifier. Probably yes, but it seems like it might be a pretty narrow path. But a lot depends on #2. The trade-off between #1 and #3 is less stark the more successful one shot models are at handling use cases successfully.
replyI think the main argument would just be that because the model is general, you don't need to retrain it from scratch for a new problem - just tweak the input prompt. For a typical classifier there's a lot more hassle - collecting the data, training it yourself, retraining under distribution shift... In that sense Jev seems great for prototyping or small-scale use cases.
replyCounterargument: this works for quick prototyping, but for any serious business, you will eventually develop a benchmark/eval to track how well the general model is working, and once you have that dataset, you might as well train a specific model
replyJev's bet is that if it works well enough for random use cases that nobody complains, then management won't feel a need to develop a benchmark/eval, and they won't need to employ all those data science guys.
replyI'd also add that they're hoping Jevon's Paradox also leads to a whole new segment of users who would have never reached for a classifier in the first place, given the barrier to entry.
replyYes this is what I'm interested in. I think they might be right. I'm already finding myself thinking "well maybe a classifier would be useful here now that it's so easy to do...".
replyThis probably just means that I could have been reaching for that tool more often already. But in practice I wasn't, and this has opened my eyes to the potential opportunities there.
And if you do get complaints or feedback on the classification, have a dev log into the user's account, tweak the Jev prompt a little until the issue goes away, and push it to production
replyOr not. And replace the generalist with the next generalist that gets you +15% on that benchmark for the same price, or gives you the same benchmark performance for half the price.
replyOne advantage of using generalist models is that the generalists are improving - regardless of whether you're doing anything about it.
Training a classification model is trivial these days, even for a number far bigger than what Jev can do.
replyUsing Jev as a plain classifier is the least interesting case. See robotic control, navigation, computer use examples, none of it possible with a classifier.
replyThat's the point, they're classification in disguise. Agentic game engines/mods started doing this long ago due to the latency requirements (although they're typically using small BERT-like models that need to be finetuned, or low TTFT generative models and structured outputs). New or newly discovered use cases are great, sure.
replyPrompt ingestion is going to be the biggest differentiator.
replyBeing able to route prompt to features that then route to special models would be a really solid implementation.
It starts to break down once you go over 20 classifications. Which is very basic routing that can easily be done with typical ML models for cheaper and faster.
replyThanks for the breadcrumb!
reply