I've found that once you factor in multiple GPUs things can get complex quite quickly because the default packing routine in the LLM runners tends to be very coarse resulting in substantial amounts of VRAM wasted. More so if you start running drafters and multiple models at the same time.

Let me return the favor:

Deepseek V4 Flash 0731 Q8

This is on a 4x3090 box with 256G RAM and a 5955WX TR; the CUDA drivers are patched to allow the 3090's to talk to each other. MB is a WRX80E-SAGE (which has 7 slots at x16), BAR > 4G on and PCIe BAR resize set to AUTO in the bios. OS is Ubuntu 25.04.

GGML_CUDA_P2P=1 build/bin/llama-server -m ../DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf --alias DeepSeek-V4-Flash --threads 32 --threads-batch 64 --cache-type-k f16 --cache-type-v f16 --metrics --cache-ram 0 --load-mode none --fit on --fit-ctx 1000000 --host 0.0.0.0 -fa on --top-p 0.95 --temp 0.7 --repeat-penalty 1.2 --jinja --tensor-split 1,.8,.1,.15 --cont-batching -b 4096 -ub 4096 -ncmoe 36 --parallel 1 -ot 'blk\.3[67]\.ffn_(gate|up|down)_exps=CPU'

I'm still working on an optimal packing for the draft variation on this theme but that gives all kinds of issues. The 1,000,000 context is very nice to have.