The qwen models really seem to have this as a failure mode, its so annoying having a proper trace ending up in !!!!!! Garbage.
replyWait in a regular sentence, what is the probability of "!!!" being followed by "!"?
replySounds like the model is not following a proper probabilistic choice here, so maybe more a programming error than a model training error.
That's what happens when you quant too hard. I'm working on quant strats and evals for the same underlying qwen 27b models.
replyWhen I saw 27b on a phone, I thought not fitting, big phone, or aggressive quant. NVFP4 still takes 27G before KV cache.