undefined

points

by johndough14 hours ago |

comments

by InvertedRhodium13 hours ago|

[-]

I tested Qwen 3.8 on the Blade CTF last night, it took 3 hours but got the correct answer.

I know that didn’t answer your question but I was looking for a test suite and couldn’t find anything.

After reading the logs, there is far less doom looping than with 3.6, but whether that’s a one off or not is up for debate.

Q4_K_P

by Balinares13 hours ago|

parent|

[-]

I anecdotally observed the same.

Interestingly, it also seems to tend toward self-correcting, which makes lower quantizations borderline usable. There'll be more faffing around, but still converging toward a solution. I wonder if that's a deliberate product of its RL.

by danielhanchen12 hours ago|

prev|

[-]

We made something called Divergence-300 @32 (and later @512) which tests actual inference across 32 tokens on a held out test (Terminal Bench, DeepSWE, Math etc)

We do plan to do larger benchmark suites though!

by johndough4 hours ago|

parent|

[-]

Great to hear that you are planning larger benchmarks! I am particularly interested in longer-running tasks with many steps and self-correction. Divergence is fine as long as the model can still solve the task, which Divergence-300 @32 does not measure.

The current benchmark suites that frontier AI labs use are probably a good fit, e.g.

https://z.ai/blog/glm-5.3#:~:text=Performance%20across%20com...

https://www.kimi.ai/ai-models/kimi-k3#:~:text=Performance%20...

https://www.anthropic.com/news/claude-opus-5

https://openai.com/index/gpt-5-6/

But guessing from your current benchmarks, I assume that you are severely compute-constrained. What is your time budget?

by zdragnar11 hours ago|

prev|

[-]

Purely an anecdote, but I've found Qwen3.8-27b doesn't doom loop like previous Qwen models would. With that said, it absolutely thinks in circles- it'll prepare to do something, say it is now ready to do it, then follow that with three paragraphs that all start with Acutally... Oh wait, I should check first... Hmm, hmm... I should stop guessing and just do it. Okay, I'm ready to do the thing now... Actually, wait...

It takes forever, but it does actually get around to making things work, and it is more thorough and produces better code than previous qwen models. You just need to let it run quite awhile.

by zenoprax9 hours ago|

parent|

[-]

I've seen the same thing. I tried the "superpowers" meta-harness and gave it a simple web app task and it spent 4 hours to make a basic timer app. I might try restricting the amount of thinking it is allowed to do to 500-1000 tokens.

by stkdump8 hours ago|

parent|

[-]

There is a native reasoning effort setting. It defaults to xhigh, I guess to get the best benchmark results, but you can just run it on medium or low instead, or for simple things even disable thinking outright.

by lifepillar5 hours ago|

parent|

[-]

According to this guy [0], medium is the level that tends to produce way less tokens in agentic workflows ("low" may output less per response, but then the model makes more mistakes, so it needs to iterate more).

[0] https://m.youtube.com/watch?v=z64J6bC16iQ