100% agree, have found through tests a big discrepancy between models optimised for SWE benchmarks and usage in prod - there is a somewhat subjective element to it as well in terms of output that can't be quantified effectively (less so with decision models naturally)
replyOr, you know, it's an easy excuse because they're already behind on many of them.
replyThey are? In all my evals at work, Jev seems to win, despite me wanting the open weight versions to.
replyI imagine if I fine tuned one of the open weight models for my use case it would be possible to beat it, though, but as a general drop in decision model, I’ve not really seen much that beats it especially for the cost
Same here, and I have a project 100% hosted in Cloudflare ecosystem (AI Gateway, Workers, Queues, R2, D1, DO, etc) I was so happy when I saw clef. Tested, and it's bad. Slower and wronger than Jev.
replyStill waiting on Decisions API using Luna, as this what I was using before.
Run your own evals, folks.
we've been public about this before the category was even created
replyOr they use it as marketing wording to get liked by devs.., which is most likely given we know that’s typesafes usp
reply[dead]
reply