What quantization level is that? Because official endpoints are slow.
replyIt doesn't need extra quantization. The official weights are natively mixed precision FP4/FP8, so it fits in ~160GB. The API slowness is probably from being batched with other concurrent user requests. The provider's aggregate throughput gets higher but per-stream speed slows down.
replyV4 Flash fits entirely in two RTX Pro 6000s without any quantization at all.
replyFrom opencode go $10/mo plan I get between 60 t/s and 100 token/s even with large contexts of 150k+ tokens.
replyI wouldn't call 80 t/s slow.
You are right, relatively to other llm providers this is not slow. But if you think what is possible when you have 1000t/s a sec you might find it slow.
replyWhat runtime are you using with the 2x RTX Pro 6000 Blackwell machine? I have the same setup and tried DSv4 Flash on vLLM and ran into a ton of kernel bugs that don't seem to have been fixed yet.
replyThis one works great. https://github.com/ormandj/sglang-deepseek-v4-flash-sm120
replyso..you guys have $20k+ rigs?
replydeleted
replyI'm getting like 25 tok/s on 2x RTX Pro 6000. This is with llama.cpp, but I had GPT tune it for me. I was under the impression vLLM was at most ~2x faster, and usually for highly parallel loads. Any tips on where I should look first for an obvious blunder?
replyI'm guessing tensor parallelism or similar?
You're gonna want a custom vLLM build.
replyHere's a runbook: https://github.com/local-inference-lab/rtx6kpro/blob/master/...
If the newer builds aren't working, you might try running the old v6 build (based on the eldritch-enlightenment image). gilded-gnosis gave me some problems that I haven't bothered to track down, the old builds are still gonna blow away llama-server performance. And that's before you get hooked on vLLM's PagedAttention and can run multiple sequences without a ton of extra overhead.