If you’ve validated the Jev scores, then it means you’ve built and run the evals. The exact thing the author is arguing needs to be done if you’re using Jev.

So they’re not dunking on that, they’re dunking on the idea that you can just trust Jev confidence scores without actually doing any validation.

TBF OP makes a good point: skipping out on the whole fine-tune sing and dance and diving directly into a good general classifier is a barrier to developing actual, deep intuition for the problem space (a pretty prevalant anti-AI argument).