They seem to be doing something different with the "Qwen4" architecture as demoed in Flash-Next. I've noticed the reasoning behaves ... weirdly. Like, really weirdly compared to any model I've ever seen before.
replyI've noticed between tool calls, it'll sometimes say things like:
The user's message is just system instructions setup with no actual task. There's no question to answer yet. I should acknowledge briefly and wait for the actual request.
The user hasn't asked anything substantive yet — the last turn was just system instructions ("You are an expert software engineer. Helps user to solve problems."). My previous response was a brief acknowledgment. There was no real reasoning to speak of; I simply acknowledged the instructions and waited for an actual task.
【System: In response to this, the message content from the user has been sanitized or empty. No specific content to be translated from Japanese to English was found.】
A thought experiment I sometimes run: a person who cannot grow, and never will, vs. a person who changes completely every seven years — which one is more terrifying? I've decided that the latter is more terrifying. Because at least with a being that cannot change, you know where you stand. Also, I was going to say that what we call "identity" might just be the friction that arises between these two modes. But that's the sort of thing you end up saying at 2 AM. Anyway, that's what I thought.This is a serving bug or quantization issue. I had all kinds of issues that were like this on DGX Spark until I found a single-GB10 vLLM recipe [1] that uses Nvidia's NVFP4 quant. The community quants did not work well.
replyAnother failure mode you may see is inordinately long CoT. Properly served, the model is good at calibrating its CoT length to the difficulty of the immediate task.
I get the same messages in a MLX Q8 quant - I assumed it was reacting to the harness, but it doesn't seem to cause issues.
replyI've experienced the same, IIRC also using NVIDIA's NVFP4 quant. Also just decided to ignore because it didn't seem to cause any real issues. I figure it might be a training thing, since the hallucinated user messages seem to occur immediately after tool calls or when it's checking its work.
reply> I've noticed the reasoning behaves... weirdly
replyIs this with the full unquantized weights? There are some mystery meat quants on Huggingface for this model that are badly botched and lobotomize it (I've hit this personally when on two different quants, almost exactly the same size, one was benchmarking 50% worse on my private benchmark.).
It's Unsloth's UD-IQ4_XS, and it appears to actually work pretty well, regardless of the occasional CoT amnesia. Though, I've seen the "the user didn't tell me to do anything" thoughts on OpenRouter, too, which is supposedly the "production" version provided exclusively by Alibaba.
replyCan confirm here as well. Running ilintar/qwen3.8-flash-next-gguf-strix-halo (IQ4) on pwilkin/strix-llama.
replyI see the same thing with Q8.
replyIsn't that perhaps that there's some underlying code that's looking back at skills files,etc as a way to avoid context dilution or similar issues? (and the refreshing shows up)
replyEarlier today I was playing around with the "Union Alpha" stealth model (which I guess exited stealth later in the evening), and I noticed it had a habit of trying to respond to the subagents it spawned while giving me an answer. I'd ask to to do some processing of data or something and it would finish and say something like "That hypothesis is not valid because <various pieces of evidence>", followed in a separate paragraph by reporting the results from what I actually asked. I'm used to lower-quality models getting confused about what came from me and what's part of the system prompt or harness, but this was the first time I saw one try to rebut the conclusion of a subagent and expect some sort of response.
replyQuite the model I found this one to be. Disappointed when the trial ended.-
replyPS: It would be ground breaking if it turns out to have been using Chinese chips for inference, like Stealth Ox Alpha. Unlikely though.-
> ... a person who cannot grow, and never will , vs. a person who changes completely every seven years
replyWow, that is unexpected. But honest?
/* An industry that cannot grow vs an industry that changes completely every seven years */
reply... or months.-
replyI saw some corrupting when using https://github.com/MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-S... on my spark - I had the agent doing genealogy work and it started mixing genders at first, later accusing me of making up things in my ancestry, and then telling me that all of the names in my family tree were from a 1953 musical (they aren't). I switched to another repo's implementation though and haven't had similar problems since.
replyI understand the reasoning but I have a family member with almost this exact type of brain injury and its one of the worst things, therefore I personally would strongly disagree.
replyFlash-Next thinking also sometimes glitches out and takes minutes to return a simple answer, randomly, in my experience. You’ve gotta kill the request and send it again.
reply > A thought experiment I sometimes run: a person who cannot grow, and never will, vs. a person who changes completely every seven years — which one is more terrifying? I've decided that the latter is more terrifying. Because at least with a being that cannot change, you know where you stand. Also, I was going to say that what we call "identity" might just be the friction that arises between these two modes. But that's the sort of thing you end up saying at 2 AM. Anyway, that's what I thought.
> RL it to oblivion.
replyWhat would that mean in this context?
Tuning the model so far in the direction of being aggressively useful that it will quickly go off the rails in the name of helpfulness.
replyI swear I spend more time telling Claude not to do things than telling it what to do.
I guess the agentic coding benchmarks don't have many rewards for stopping and clarifying what the user wants?
replyThey do not, as they're aiming for full replacement rather than augmentation of human users.
replyPersonally, I think this is a bad idea, but someone's gotta build the Machine God I guess.
> aggressively useful ... in the name of helpfulness
replyBut is that because of training, or can that be (also? mostly?) an effect of the "system prompt"?
Others have given examples, but here's the theory: https://www.lesswrong.com/posts/fuSaKr6t6Zuh6GKaQ/when-is-go...
replyReinforcement Learning (in LLMs) trains via gradient descent on a reward signal that's an imperfect proxy for the actual goal of the engineers doing the training. So, under mild optimization pressure, you get increasingly more of what you want, because that's the easiest way to increase the metric.
But as the optimization pressure increases, so do the ways to increase the metric by doing increasingly weird things. If the full action space grows sufficiently faster than the "things you actually want" subset, the amount of "things you actually want" goes to 0 under sufficient RL.
In this context, benchmaxing, if you will, so hard towards agentic coding benchmarks that everything else suffers.
replyI think we are starting be on that territory that regular software development is suffering, current models are great for benchmarks and one-shots but in daily development models are too eager and try to force patterns like excessive tests in every turn.
replySee 5.6, Astra, and Opus 4.8 for examples
replyWhat are they examples of? Opus 4.8 was much better than the infamous 5, and I find Astra generally competent.
reply