upvote

points

by hei-lima9 hours ago |
comments
upvoteby SwellJoe8 hours ago|
next
[-]
We're having DeepSeek moments every couple of weeks.

Qwen 3.6 hit hard in the self-hosting space. It's incredibly capable for its size, really shaking up what's possible in 64GB or even 32GB of VRAM.

The Prism Bonsai ternary model crams a tremendous amount of capability into 1.75GB.

And, DeepSeek V4 is crazy good for the price. They're charging flash model prices for their top-tier Pro model, which is competitive with the frontier of a few months ago.

The winners in the AI war will be the companies that figure out how to run them efficiently, not the ones that eke out a couple percent better performance on a benchmark while spending ten times as much on inference (though the capability has to be there, I think we're seeing that capability alone isn't a strong moat...there's enough competent competition to insure there's always at least a few options even at the very frontier of capability).

reply
upvoteby Zambyte7 hours ago|
parent|
next
[-]
> It's incredibly capable for its size, really shaking up what's possible in 64GB or even 32GB of VRAM.

You can lower that to at least 24GB. I've been running Qwen 3.5 and 3.6 with codex on a 7900 XTX and the long horizon tasks it can handle successfully has been blowing my mind. I would seriously choose running my current local setup over (the SOTA models + ecosystem) of a year ago just based on how productive I can be.

reply
upvoteby hei-lima4 hours ago|
parent|
[-]
Gonna try it.
reply
upvoteby trollbridge7 hours ago|
parent|
prev|
[-]
We have Qwen 3.6-35b (6) on a 5090 (32GB) and it's blowing me away. Works fine for most (not all) code generation tasks. One developer here has been extremely stubborn about adopting AI; he's finally adopted it, albeit only when it's coming from a local model like this.

DeepSeek V4 Pro likewise is insanely good for the price. I simply point it at large codebases, go get a cup of coffee or browse Hacker News, and then it's done useful work. This was simply not possible with other models without hitting budget problems.

reply
upvoteby akulbe6 hours ago|
parent|
[-]
Any chance you'd be willing to talk further about your setup? I have 2 x 3090s in a local machine, and I'm still left with questions about how best to use stuff locally.
reply
upvoteby sheeshkebab4 hours ago|
parent|
[-]
You can only run heavily quantized models on all 3/4/5 rtx gpus (with 32gb or less vram) - and you probably want moe versions like Qwen 35b for this to run at speed somewhat comparable to Claude. It’s still not there to be honest but getting there. Personally I mess around with llama.cpp on m5 max with 128gb - it’s a decent setup to try various medium sized things, and runs llms surprisingly well without quantization, at least the moe models.
reply
upvoteby SwellJoe4 hours ago|
parent|
[-]
Two 3090s is 48GB, so it's possible to run the 6-bit quantization comfortably, which is fine. It doesn't start to get notably dumber until lower than that. It won't be as fast as a hosted model, but dual 3090s will be comfortably fast for interactive use with the MoE version and not terrible to use with the dense model. I run the dense model at 8 bits on my dual Radeon V620 desktop machine, which I think would be slower than two 3090s, or at least not notably faster.
reply
upvoteby hedgehog3 hours ago|
parent|
[-]
Have you done comparisons with 4 bit and seen a noticeable difference for coding tasks?
reply
upvoteby SwellJoe1 hours ago|
parent|
[-]
No, I've just seen benchmarks showing most models start degrading around 4-5 bits. That's not to say they become useless, just that down to about 6-bits (with careful hybrid quantizations like unsloth where some of the layers aren't quantized or are quantized at higher bit depths) the quality isn't measurably degraded, but below that there are measurable differences in performance.

People report good results from DeepSeek V4 Flash at 2 bits (the DwarfStar 4 folks are doing it, and I've tried it on my Strix Halo, but it's too slow to be usable, so I haven't bothered to figure out if it's actually smart enough to use for anything).

Anyway, it's obvious models have to degrade in terms of knowledge, at any quantization, even though it may not show up clearly on benchmarks until lower. If you halve the size of the data available, it necessarily loses information about the world.

reply
upvoteby squidbeak9 hours ago|
prev|
next
[-]
Deepseek had another moment a few weeks ago. V4 isn't far behind the US frontier, and so far its flash variant seems a very reliable coder and costs a pittance.
reply
upvoteby ai_fry_ur_brain8 hours ago|
parent|
[-]
Deepseek V4 (not flash) trippled in price too by the way (from Deepseek). Get used to this pattern.

This is what you get for relying on the generosity of billionaires. Keep offshoring your thinking ability to a machine and let me know how competitive you. Hint, you wont be. There's nothing special about being able to use an LLM.

reply
upvoteby barrell4 minutes ago|
parent|
next
[-]
Actually, deepseek v4 was 1/3 promotional price for the first month or so. This was pretty clearly communicated. The promotions window just ended is all.
reply
upvoteby npn8 hours ago|
parent|
prev|
next
[-]
Unlike other providers, Deepseek does promise that they will lower the price when their Huawei cards arrive in a few more months.
reply
upvoteby flakiness7 hours ago|
parent|
[-]
Give me a link. Cannot wait. One PSA is that they have 75% discount right now so it is already cheaper than the full price.
reply
upvoteby npn6 hours ago|
parent|
[-]
Weird, last time I checked it was right on the pricing page.

But even when it happens I doubt it would be as cheap as it is right now. Enjoy it while it lasts!

reply
upvoteby ls6128 hours ago|
parent|
prev|
next
[-]
Anyone can host Deepseek V4 on rented GPUs and sell inference on it. Price will very quickly converge to the marginal cost of inference. This is as close to a pure commodity as it gets in the AI space so competitive market economics will put in work. Same is true for any open-weights model.
reply
upvoteby ai_fry_ur_brain8 hours ago|
parent|
[-]
You dont understand the costs involved to run inference at scale

Please go run some numbers.The hardware needed to Run Deepseek v4 flash at 20 tps for a single session is nowhere close to what is required to run it at 50tps for 5,000 concurrent sessions.

Imagine what it takes to be profitible when running at 150 tps for 30cents per 1mm. You make less than 1k per month and the hardware required to run that cost 10k a month to rent with hardly any concurrent session capability.

reply
upvoteby gpugreg6 hours ago|
parent|
next
[-]
> Please go run some numbers.

- DeepSeek serves DeepSeek V4 Pro at 27 tps: https://openrouter.ai/deepseek/deepseek-v4-pro

- At 27 tps per user, a B300 GPUS will give you around 800 tokens per second (serving 30 users): https://developer-blogs.nvidia.com/wp-content/uploads/2026/0...

- That's 800 * 60 * 60 generated tokens per hour, at a cost of $0.87 per 1M tokens, or $2.50 per hour.

- For input and output tokens, the math is a bit more complicated because we have to make assumptions about their ratio. Using the published values from OpenCode, we get another $2.50 for cached tokens (which are almost free for DeepSeek) and another $3.40 for input tokens (which are a lot cheaper to compute than output tokens), which gives us a total of $8.50 per hour per B300 GPU.

- B300 GPUs can be rented for as low as $3.40 per hour, which is less than $8.50, so hosting DeepSeek V4 Pro is profitable.

You could also host it at fewer tps per user to raise the efficiency and therefore the profit even higher.

reply
upvoteby ls6126 hours ago|
parent|
[-]
Even not assuming Blackwell inference the $3.50/hr price is likely close to the marginal cost. The Deepseek R0 model is a little more than a third of the size of V4 and cost around $1/Mtok to serve at scale based on deepseek's blogs last year and Hopper rental prices.
reply
upvoteby ls6128 hours ago|
parent|
prev|
[-]
Yes it is more efficient in $/tok to run at scale than to run just for yourself. Everyone selling Deepseek V4 inference is selling an undifferentiated good. They have run the numbers on how much it costs and are competing against a dozen other outfits also selling undifferentiated open weights tokens. Whatever the dollar cost they face to rent those GPUs will be what they are able to charge in the competitive market. That is great for you and me because we can buy tokens at pretty much exactly what it costs to produce them.
reply
upvoteby dpoloncsak8 hours ago|
parent|
prev|
next
[-]
Mate why are you so mad at people upset the price trippeled? It's a fair complaint that people built services using the cheaper ones with the expectation future models would be similarly priced. You can avoid 'offloading thinking' while still building ontop of these models
reply
upvoteby zaptrem7 hours ago|
parent|
prev|
next
[-]
V4-Pro is about 2.4× total params and 1.3× active params of V3.2.
reply
upvoteby creationcomplex6 hours ago|
parent|
prev|
next
[-]
You're typing as your handwriting and letter sending abilities deteriorate to dust. Writing down information as your memory capacity decays. Remembering instead of living at the pure leading edge of perception dulling your reactions.

Smh, it's all downhill from the first unadulterated neuron.

reply
upvoteby aurareturn8 hours ago|
parent|
prev|
[-]
I think demand is too great and compute is not enough. Nothing to do with billionaires colluding to increase prices by 3x.
reply
upvoteby boutell6 hours ago|
parent|
[-]
Actually, why should Google collude on pricing? They have deep pockets and could starve out the competition while keeping prices low, if they really wanted.

I think it is priced high because it's basically their smartest model as well as their fastest, so why shouldn't they?

You can still use earlier generations of Flash at a lower cost if you want "fast and cheap and just OK," which often makes sense. (Just checked)

I would predict they will lower this price when 3.5 High appears, but perhaps not all the way.

reply
upvoteby xbmcuser8 hours ago|
prev|
next
[-]
What we need is a deepseek moment in hardware ie China reaching parity on node size that is the only way latest computers let alone latest ai will be available to us in the future otherwise the profit margins will push most production to AI.
reply
upvoteby throwa3562628 hours ago|
parent|
next
[-]
To be honest, China not having access to the latest hardware is exactly what has driven LLM technology forward the last 2 years.
reply
upvoteby humanfromearth97 hours ago|
parent|
[-]
Why?
reply
upvoteby Weryj7 hours ago|
parent|
next
[-]
Because it forced them to focus on efficiency, instead of throwing more compute at the problem.

Just like in software, some of the most beautiful solutions come from constraints. Think, the optimisations that game developers implemented because of the frame budget.

reply
upvoteby Viacol3 hours ago|
parent|
prev|
[-]
On top of that, China is also facing hardware constraints, which is pushing companies to develop better domestic chips for AI training. It'll be interesting to see how things perform once Huawei's newer hardware is fully deployed at DeepSeek.
reply
upvoteby blackoil1 hours ago|
parent|
prev|
[-]
Open Source ASML EUV. But will wipe off trillions from US stocks so 401k may not like that.
reply
upvoteby stared5 hours ago|
prev|
next
[-]
We have a "DeepSeek moment", https://github.com/antirez/ds4 (see https://news.ycombinator.com/item?id=48142108).

Or if you prefer smaller ones, Qwen3.6-35B-A3B, https://huggingface.co/bartowski/Qwen_Qwen3.6-35B-A3B-GGUF

reply
upvoteby segmondy9 hours ago|
prev|
next
[-]
You can use lots of open weight models today.
reply
upvoteby hei-lima8 hours ago|
parent|
next
[-]
That's one solution to the problem. But it still needs some good computational capabilities. Either we optimize the hell out of those models, or we wait for the hardware to become good enough for them.
reply
upvoteby Gigachad6 hours ago|
parent|
prev|
[-]
The real problem is the hardware to run them is still very expensive.
reply
upvoteby pianopatrick7 hours ago|
prev|
next
[-]
Maybe we can figure out better ways to use the models that can run on cheap hardware.
reply
upvoteby GeorgeOldfield8 hours ago|
prev|
[-]
gemini isn't even that good. just tested 3.5 on usual complex prompts to opus/chat 5.5. meh
reply
upvoteby k8sToGo8 hours ago|
parent|
next
[-]
Are you really comparing flash to opus? Shouldn't you be comparing pro?
reply
upvoteby CognitiveLens8 hours ago|
parent|
[-]
The benchmark tables in the Google announcement include Opus 4.7, and the numbers are very impressive. Caveat emptor, but it's not unreasonable to compare a new Flash to a current-gen Opus, even if some of the results confirm expectations
reply
upvoteby bachmeier7 hours ago|
parent|
prev|
next
[-]
Who would have guessed that something costing roughly a third as much wouldn't do as well at certain tasks.
reply
upvoteby kmac_8 hours ago|
parent|
prev|
[-]
Well, the first impression is that Gemini still goes off the instruction rails easier than other models, but I noticed that it tends to go back to the initial goal without holding a hand, which is a real improvement. It's really interesting that these models behave so differently.
reply