The model wasn’t horrible at coding, it just made a ton of initial mistakes—but then would catch most of them later on. It can get work done if coaxed, but its peers are just cheaper and/or easier to work with.
I've used local models extensively (mostly qwen 3.8 27b) and now Claude sonnet/opus 5.5 since launch.
Sometimes I really wonder if not being able to see what's going on in thinking is a large contributor to thinking the models are stronger.
That assumes you bother skimming through thinking traces and the output in-between.
In my experience both qwen and claude fumbles and misunderstand, but will in a lot of cases self correct.
Qwen models can feel very indecisive but somehow makes it in the end. If the harness was able to hide that and only highlight the end results, I'd probably think qwen models are smarter.
Reading the tool calls Claude does while thinking shows a lot of trial and error about even simple things like editing code and getting the whitespace wrong. But it happens so fast compared to qwen that it's sort of acceptable. With qwen, it can feel frustrating because it spends so much time on it.
This is all assuming that's what you meant by mistakes, but maybe you meant something different.
I am not sure. The people who actually care enough to look at benchmarks will run evals, and find out quickly. If you post silly benchmark scores too often, you will simply be ignored. You are paying with your reputation.
I can run Qwen Flash Next on hopes, dreams, DDR4, and ancient data center GPUs at 200 t/s :)