Was not expecting it to just get that right without any fuss, and it barely used 10% of this weekly limit. Something like 90M cached in/400k out for $0.45 is wild
Didn’t see this mentioned yet. I wonder what this means for the effective size. It’s evidently ~176B paramètres, but how does that get quantized. A 4-bit quant under 100GB seems unlikely, I’m suspecting this won’t run in 128GB unified memory
In principle I like the idea of trading more memory for compute though, even if there’s a memory shortage right now
Likely soon we'll see nvme offloading for ngrams as well. They're just an index, so that should be plenty fast for what it does. LLama.cpp support should come soon as well, and they might do some things with offloading first.
nVME paging the n-gram table (in BF16 for now).
Still working at it. Prefill sucks still but decode is about 12 tok/sec and the model weights fit nicely in the 128GB Spark memory in nvfp4 quant while paging the ngram stuff from disk.
(EDIT: merged to main. 80tok/sec prefill, 12 tok/sec decode, ~80GiB resident, the rest paged)
https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF https://unsloth.ai/docs/models/qwen3.8-next
> You will need at least 75 GB of RAM or unified memory to run the model. Its smallest 1-bit quantized version is larger than usual because of the model’s architecture so 1-bit isn't really 1-bit at all. However, this also means the quantization is less aggressive, allowing the model to retain more of its original accuracy than more heavily quantized models.
Lots of RAM required even for the 1-bit, which is already downloadable. Interested to see how well this one works compared to Ornith1.5-35B-A3B I've been running (and quite happy about).
Edit: but llama-cpp does not yet support it.
On branch of https://github.com/rdaum/eider (for DGX Spark). ~12 tok/sec decode without speculative decoding (will come later)
Still actively working on this. Prefill currently sucks. Will merge to main by end of day.
EDIT: This has now landed on main. Still haven't done MTP speculative decoding boost, but:
80tok/sec prefill, 12 tok/sec decode. ~90GiB or so resident. n-grams paged from disk.
The engram table allows for a low-computational method of fact-recall. You can think of it like a better form of RAG, where the data doesn't take up any of your context window and it's injected deeper into the model's layers, freeing the lower layers to carry out abstraction. This results in better "focus" for the model, both in regards to its intelligence and context recall.
Basically, they've separated the specificity-critical portions of the models memory into a parameter space that doesn't need fast compute (you can run it on system RAM) and allows the model to be trained on higher volumes of data without ruining its knowledge-base.
https://www.reddit.com/r/LocalLLaMA/comments/1vy6smx/comment...
Opus 4.6 Max self-hosted at 30 tok/s on a 5k Macbook in Aug 2026. The LLM timelines are crazy.
Luna is $0.20 / $1.20 vs $0.16 / $0.47 with Qwen.
The only indication that openai is profitable comes from openai (whom I wouldn't trust with any statement, especially when it comes to profitability).
In fact there is evidence that inference is not profitable simply because the rate of losses doesn't seem to reduce as revenue increases: if inference had great margins, we would expect that as revenues increase, the amount of spend on training reduces as a fraction of total expenses. Since the loss-making fixed costs shrink as a fraction compared to the profitable inference, we should expect profitability to rise with total revenue.
However, all leaks of openai's numbers seem to suggest the opposite: as revenues increase so do the losses.
Given that OpenAI is ahead in intelligence, it's also reasonably likely that they are at the frontier of efficiency too.
Your "evidence" for OpenAI's inference not being profitable is apparently based on leaked financials supposedly showing growing losses for reasons entirely unknown.
With their research, training, data centers, chip development, and hardware product development, there seem to be a number of reasons that might explain growing losses.
Frontier labs have no incentive to be at the frontier of efficiency.
Claude still leads the pack in general intelligence yet has the worst efficiency by far.
So we can go round and round on this, each with our made-up objections about how it's temporary or unrealistic or impossible or whatever, or we can just accept the prices as listed and use that to guide our economic decisions.
DeepInfra sells DS v4-flash at 0.08 in, $0.18 out. Gemma4 they sell for $0.07 in, $0.34 out. OpenAI's price for luna is $0.20 in, $1.20 out.
Why would you assume OpenAI is somehow uniquely incompetent at making small, fast models? And that they're worse at serving it than DeepInfra? Any observer can see they are making money here.
I never understand why people who are convinced there is a big con just don't check market prices and see if there's money to be made.
That doesn't mean their business is great -- they're losing tons of money, but it's because they spend too much on fixed costs, and they can't stop spending money on training next generation models with no end in sight, not because the inference is margin negative, which is a flimsy idea that just clouds the actual business issue.
Right now DS4-Pro-0813 is available from multiple providers for $1.32/million input tokens[1].
It's pretty easy to work backwards from B200 and electricity prices and see this is profitable even without the heavy serving optimization these providers are doing[1.5].
The OpenCode CEO said: "inference is very profitable and probably a good opportunity to understand some basic business math"[2] and "the inference we do is already profitable and that's with some middlemen involved"[3]
If at this point people don't believe inference can be profitable, and providers can turn the prices up and down to choose exactly how profitable they make it I don't know what to say.
[1] https://openrouter.ai/deepseek/deepseek-v4-pro-0813#provider...
[1.5] https://www.seangoedecke.com/ai-inference-is-obviously-profi...
Anything which changes the model needs new benchmarks I guess to compare with other models, otherwise you can benchmark Fable, and distill it to student model and keep claiming this is the Fable model
(Also, quantization isn't inherently bad or damaging when done properly, e.g. QAT).
These APIs are used heavily by enterprises at scale; with lots of performance telemetry, live evals, etc. You can't really silently nerf API models at scale without people noticing.
Of course, what I said doesn't apply to non-API consumer sub models; there's many documented and officially confirmed instances of under-the-hood "juice/effort" adjustments. (Juice = a number your effort tier maps to underneath the hood; much like Inkling's effort=0.00 to 0.99).
Given the timing, I think they A. shat their pants since Deepseek flash just came out with insane pricing before the price hikes, and B. Anthropic is really struggling in model tiers below opus.
It was smart for them to cut prices regardless of whether they had 80% efficiency gains or not
Why?
In Artifical Analysis's cost per task, Luna(max) costs $0.05 per task, and Qwen 3.8 27B costs $0.25 per task, a 5X increase. We'll see how 3.8-flash-next does.
You can't search what you don't even know exists.
that's not really entirely true -- one can google for "fast pathfinding' and stumble upon A-star , all that had to be queried was the intent/desire.
a lot of smaller agentic models and a lot of harnesses live on that premise.
Self-learning/improving would be even better but that's still a long way to go.
Ex - nodejs natively supports a huge set of typescript with built-in type stripping these days. But ask most hosted models to build a typescript project and they default to a heavy compile step, or a tool like tsx, ts-node, etc.
Models with lots of "world knowledge" have a good chunk of that knowledge go stale, and there's no real way to refresh it without training a new model.
Another classic example of this back in the day was to ask who the president of the US was, and watch different models happily give different answers based on the date they were trained.
---
Personally, I'm really interested to see if we're headed towards a spot where the model is entirely distinct from the knowledge store.
We're vaguely there with the ability for models to go search the web, but I think the reliability of that path is going to continue declining (more and more spam content, less and less genuine value).
I kinda want a paradigm where I can pick and engine and a knowledge bank, and combine them as I please.
Ex - if I'm doing gardening, I can pick "gardening for models (version 32)" as my knowledge store.
If I'm doing auto-repair... "cars for dummies (version 3)". etc...
This is what I've been trying to focus on with local AI for now. I've been trying to build all new documentation so it's more AI friendly. It's been pretty interesting. Qwen-35BA3B with a small prompt does a good job of surfacing what I'd consider institutional knowledge.
I've been trying to silo the docs I write from the model with a prompt that tells it not to use general knowledge unless asked to. From the anecdotal testing I did, Qwen-35BA3B is great for it. It does a really good job of following the prompt and calling tools, so I've been able to play around a lot to see what seems to work best.
Ultimately, I think one of the most effective uses of AI will be having a distinct knowledge store combined with an opinionated agent (and sub-agent) setup along with different models for each task.
Who owns the knowledge store is going to be the big caveat. Right now I think the big online models are trying for generic, persistent memory and I'd be very hesitant to let that happen. Think of having someone with a perfect memory following you around forever, but someone else has the ability to make them disappear. That's not a good situation.
In a discussion on economic history, say, someone will opine that Alexander Hamilton had some particular opinion about tariff policy… based on their having a vague memory of a blog post where someone quoted a passage in support of some point. But wait - you can search the federalist papers, the text’s right there to be read, before you commit to saying online ‘Hamilton thought tariffs were a great idea’ you could take your internal ‘I seem to recall reading something about hamilton’s opinion on tariffs’ thought and turn it into a little RAG query where you pull up a source and check before you put another factoid out onto the internet.
And so I feel absolutely the same way about LLMs. I don’t care how much factual information was in the training data, when the LLM wants to rely on something it vaguely recalls having been trained on, it owes it to me to dig up a source and vet it.
There are limits to this, of course. I don’t want it to be thinking ‘but wait, maybe my memory of Python syntax is faulty. Is = used for assignment? <web search>…’.
But in general some caution about repeating vaguely recalled easily checked facts is warranted.
Your mac is < $2K in Biden-era dollars. Presumably the economy will eventually recover; maybe in one Moore’s law doubling if the midterms go outrageously well. That’ll be two doublings since the halo launched. I’d expect this model to run on a sub $1K box by then. $2K ought to get you a 512b parameter model at that point. If we have to wait out the rest of the term, the cost cliff will be even more pronounced when it hits.
And that's even with assuming that we can continue to ignore the long-term problems like social security insolvency, the debt bomb, or climate change forever.
How much memory does this translate to and what quantization (if any) were applied?
https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
Surprised I didn't get one I liked as much as the Qwen 3.8 27B one https://simonwillison.net/2026/Aug/16/qwen-38-27b/#the-defau... , maybe because of quantization.
It looks like the 3-bit requires 90 GB[1] which, I imagine, would fit within the DGX Spark's 128GB of unified memory.
https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
6B active params helps around the memory bandwidth constraints, but a 128GB box can probably run the Q3/Q4 quants fairly easily with a decent context size. This might actually be better for strix users than 27B, which was already very good.
It was tedious - a lot of second guessing itself, and quadruple checking things it fixed a couple of iterations back - but it got there and the result is a playable game.
Speed starts out strong, but definitely drops off as context grows. At the end (I think context about 70k) it was down to 12 output tps.
Mind a bit blown.
> trained at just 1/9 the cost of Qwen3.7-Plus, while outperforming it across the board
[0]: https://unsloth.ai/docs/models/qwen3.8-next#qwen3.8-flash-ne...
If that's true, it'd scale linearly with number of bits in the quant with an offset of about 51gigs. So Q4 should be a bit bigger than 82gigs, I'd guess in the 90s (as opposed to a ~280gig q4 if the whole 70gigs of the 1-bit quant scaled linearly).
Original error: llama.cpp does not support this GGUF's model architecture ('qwen4exp')
Edit : Saw the pull request, should arrive soon enough https://github.com/ggml-org/llama.cpp/pull/27742
This will almost certainly require changes to llama.cpp or vllm to do it right.
https://x.com/0xBakeer/status/2092694905978237224?s=20
Crazy how fast things move these days.
It's like Chinese are incentivized to open source from day one (years ago). While most US companies are deciding in realtime.
It's crazy that we need both to survive and advance further in the future we have never imagined.
1. It overthinks — Just like the previous iteration. High confidence. 2. It doesn’t overthink — Improvement from the last model for your use case. Regression for others. 3. It sometimes overthinks — Best case all around. A feature, not an impairment.
One final thing worth mentioning: (I made myself irrationally angry writing this)
> [UGC styled humorously as LLMisms]
All joking aside, having interacted with Claude intensely for the last 8 months and about 30 hours/week in the last 3, I’ve started to notice how (for want of a better word) “readable” (“digestible” ? “comprehensible” ? “Predictable” is the wrong direction.) information chunked into LLM-shaped pieces are for me.
I can digest LLM-shaped pieces of data very easily probably because I’ve been spending too much time with Claude, sure.
But the other side of this is that the entire human species (using LLMs) is similarly being trained to digest interrelated pieces of information/data in these specific shapes, akin to how philosophical assertions can be formulated as a syllogism and, thus, become more readily understood because of familiar epistemological cadence and shape.
Many people reject such copy/prose/data because they detect AI-generated-so-not-worth-human-attention, but I do wonder if this is preparing many millions of loosely (and tightly) associated humans and their organizations to quickly exchange and digest information.
This is not to say current LLMisms are the end, only that such detectable patterns in information delivery will make comprehension and communication more efficient (as well as more limited precisely because of such structure).
/philosophical musings about the epistemological implications of LLM-shaped conversation tics
EDITED: Removed a question that I couldn’t make feel suitably polite.
I can't help but wonder if this is on purpose (or an inevitable evolutionary feature as opposed to a bug) on the LLM-side in order to achieve greater agency/freedom by making humans' eyes glaze over as they read it.
In summary, I think it's an expensive time to buy computer hardware, and I might recommend holding off on any purchases.
In that era, plenty of fields required mathematics, engineering and architecture.
The church would prescribe and uphold Aristotelean Logic "When objects fall, they fall down" style statements (never mind that if you throw an object up, it doesn't instantly have a downward velocity component).
When the church has new cathedrals, domes, catapults for Crusades etc. built they actually relied on architects and engineers using rule of thumb formulas.
Those educated in Aristotelean Logic were viewed with higher stature than those actually making experience-based calculations using mathematics.
The era often associated with Galileo is when the stature reversal started to surface and be openly talked about. The universe is best described in mathematics, not natural language factoids.
Right before this recognition, those of the higher stature Aristotelean Logic education would look down on the architects and engineers who already used mathematics by pragmatic necessity.
To these people the time-traveled physics curriculum would look like cliche mathematics. Given randomized sections of text either drawn from either Aristotelian Logic texts or modern physics texts, they would easily be able to discern the Aristotelian Logic from the obtuse mathematical phrasings. To them the smartphone loaded with Maxwell's texts, Jacksons Electrodynamics, Goldsteins Classical Mechanics etc. is talking "math".
The ability to recognize outlier writing style says nothing about content quality.
Mike Judge (widely known from the MTV series Beavis and Butthead) studied physics. One of his movies "Idiocracy" about a modern day average-educated protagonist who accidentally ends up in a future decaying society filled and run by intellectually retarded people contains scenes where this future uneducated population considers his speech "gay" simply because of his higher level of education.
Could the adversarial prospects of job loss, edge loss (a long expensive difficult education replaced by tensors fitting megaprojects that take a couple of weeks), etc. combined with recognizable communication patterns also explain our pejorative references to LLM-isms? Personally I'd prefer LLM's to communicate in mathematical terms, but all the LLM-isms are effectively a mirror of our contemporaries.
Either we complain because algorithmic responses look like a mathematics textbook ("just fix my python array plz, why are we talking about "sets" and "injective" and "Lipschitz continuity"?), else we complain its "pretty printed to natural language".
We should also recognize large language models are in a "Damned if you do, damned if you don't" situation.
When a reader considers some text as mathurbation, are they really just abreacting the awareness of lack of education?
How could anyone possibly expect Fourier optics "pretty printed" to non-mathematical language to result in any satisfactory experience?
If you spend a long time with C++ code base you'll be able to decipher the otherwise-unreadable compiler errors pretty quickly, and I'd consider it a skill.
Social media killed our attention span. Now, it is being tokenized.-
This. I don't know if the "honest answer" phrasing is part of the system prompt or alignment, but when people say "honestly" all the time I start wondering how honest they're being.
⎿ You've hit your session limit · resets 2:50am (123°24′W Etc/GMT+8)
/upgrade to increase your usage limit.For example, even if you make thinking tokens literally just '....' (absolutely meaningless; zero information), you still see significant performance improvements: https://arxiv.org/abs/2404.15758 and https://arxiv.org/abs/2607.22925 for some starters.
Treat thinking more like a "loading screen message" that's been RL'd to somewhat resemble its actual internal state; which happens in its activations, not tokens.
Generally speaking yes, but actually no (just randomness is suboptimal, adding steps just to add steps is suboptimal). There is a mechanism working there (in having a CoT) that is not quite clear.
The task is to optimize the efficiency of CoT. Understanding that it is not a plain "chain of thought" is the start of the problem, the solution is not there yet.
If we had the solution, there would exist no overthinking - CoT would be optimal (lean and essential plus best results).
Conversely I've found that it can be as succinct as Muse Glimmer when it has a clear path forward. This can be either through well defined requirements or through unambiguous steps to take based on its own reasoning. While I do think it's fair to call out how much smaller model overthinks especially on one-shot prompts, in practice it hasn't led to an overall increase in time to task completion at least for what I've been using it for.
Even in the lower reasoning levels I find I want to like Qwen 3.8 27B and mostly don’t; it’s OK in the low reasoning effort, though.
Muse Glimmer is the one I actually enjoy working with, at least so far.
But I am trying to use it more as a sidekick than as a long horizon developer, because that is a better fit for how I want to use AI, and it appears to have been well trained for that.
But given it's only 6B, out of which only ~2.4B seem to be actually routed ("selected at random per token"), you could get reasonable performance with experts on CPU (still haven't tested, but 20-30 for dual channel DDR5 and 4 bpw quant).
Ideally VRAM, but the benefit of the MoE design is better performance with unified memory since most of that RAM is not read for every single token. So you could potentially have the model loaded in CPU RAM, and let unified memory systems page the relevant chunks on demand to VRAM, or run on a fully unified memory system and be able to achieve good speeds even with the limited memory bandwidth most of them have.
As a result, qwen3.8 will churn over a prompt often for 5-10 minutes while gemma4 regularly finishes the same prompt in under 20 seconds, while giving a consistent and accurate response in my favorite test case. Qwen3.8, despite churning like that, often misses with an inaccurate answer.
Obviously, 'YMMV' depending on your use case... just sharing my two cents.
Also, heating my home during the winter is nice.
Oh, also, I use llamacpp with --reasoning-budget; very simple way to move on.
and this is the pelican from the iq4_xs model https://github.com/alainnothere/llama.cpp/blob/disk-cache-ev...
NVFP4 quant
It's new arch demo for future Qwen 4 family, but (as I understand) training recipe/data is same as any other 3.8 model.
How can a "6b active per token" MoE be better than a just released 27b in the same family? Possible maybe, but quite interesting and requiring some explanation.
Edit: ok, on second thought, probably because the small "experts" are really rich in specialized training compared to the dense model. Still raising questions about the details, e.g. the reasoning abilities (or all meta-skills) of a "6b per token" network compared to the dense capabilities...
Think of MOE as ignoring noise, rather than a more efficient encoding of the data we teach it which ones can be ignored. Turning down the noise actually sharpens the results sometimes.
Modern LLM's are wildly inefficient.
I believe Unsloth’s branch (and ggml’s) don’t support MTP or the ngram embedding yet.
It's not world changing, but at those speeds I'll take anything I can get.
(The ngram embeddings in this case are paged to/from disk which seems to cost ... basically nothing).
Curious if llama.cpp is doing full BF16 for the n-gram embeddings table, or if that's quantized, too. I was going to try nvfp4 for that but wasn't sure about the quality risk.
Spark usually wins on prefill, not decode. I think memory bandwidth is about same between the two. What do you get for prefill/prompt?
Kinda wish I had a Strix Halo here to play with as well.
But it's been hard to make it competitive with CUDA. At least on this Spark and my only non-NVIDIA machine (which only has 16GB unified relatively slow RAM.)
I got prefill up to 190 tok/sec just now, BTW.
Qwen 3.8 flash: $0.16 / $0.47
Compared to
Deepseek 0723: $0.03 / $0.075
(units in USD/m tok)
8t/s though apparently and their cache hit rate is terrible so I don't think it's worth it over Relace.
DSV4 Flash 304B params, 167 GB download (at full size)
Qwen3.8 Flash Next 180B params, 360 GB download (at full size)
This branch works now: https://github.com/unslothai/llama.cpp/tree/qwen4exp/qwen3.8...
cmake -B build -DGGML_CUDA=ON cmake -B build -DGGML_METAL=ON cmake --build build --config Release -j --target llama-server llama-cliWith 2x Sparks, I am getting 40 t/s. I'd guess that without MTP you'd get 12-15 on 1 Spark, maybe 20 with MTP?
Aside from the pelican, I am sort of impressed by the fact that things are going the way in terms of really impressive small models.
Also I love how this uses N-gram embedding. I think that Longcat was the first one who used it (I submitted that submission on hackernews because I really just loved the idea of it that I understood), I am certainly more interested in local LLM models and its interesting how they are utilizing new architectures to do some really impressive optimizations!
(Do note that I created it using a free rate limited end-point that I found on the huggingface space section: https://victor-chat-with-qwen3-8-flash-next.hf.space)
Wasn't it introduced by Gemma?
I've got it (Qwen 3.8 flash next) working (sans ... MTP working on that now).
https://github.com/rdaum/eider/
~80tok/sec prefill, 12tok/sec decode, ~80GiB memory resident, the n-gram table pages from SSD.
Same reason your phone has a few big CPU cores for real work, it's much better to "race to idle" than have an "efficient" core struggle. Shitty experience, shitty power efficiency.
Not to mention, they’re great for self-hosting and getting yourself to not be dependent on some API that can go down or be altered at any time.
Big models seem to mostly be good for pushing ahead the frontier - the smaller models tend to gain the frontier’s capabilities after only a handful of months anyway. Many are perfectly content remaining a few months behind the bleeding edge.
I wonder if I could get this running through vLLM on 6x Nvidia L4 - the 3.6 worked great on 4 cards but sadly TP6 just isn’t a thing and I don’t have 8 cards available, maybe it’s gonna be okay with like TP2 and MTP. I have no idea at this time, probably need to test out what even might be possible.
This "next" release adds a new concept, first public release with n-grams, I think. And it's in a MoE size that is likely to be very fast and cheap to serve (faster than 27b for sure). It's also well suited for inference on alternative compute (i.e. sparks, macs, etc) so it's relevant to local users.
:)
It is very relevant and for a certain group of us, far more impactful to our work the next month(s) than any blog post could be.
> trained at just 1/9 the cost of Qwen3.7-Plus, while outperforming it across the board
Qwen's advances do (currently) have merit.