Like I said above, you should benchmark quantized versions. With quantization, the same models can be ran at much cheaper hardware, but quality loss is real and this kind of benchmark is an ideal place to put a finger on it.
replyKimi k3 is natively mxfp4 w/ mxfp8 activations. Since they talk about glm 5.2 running on a 8x b200s, that's likely fp8 since it wouldn't fit at bf16 (1.5TB weights alone)
replySolid point. We debated multiple optimization methods and decided to do a more vanilla run first for the setups tested. We’re moving on to some more typical optimization methods for our next article… As soon as our teams comes out a well deserved summer break. Thx for the input.
replyI second this. Anyone considering running their own hardware would also consider quantization since they would have full control over what quant to run.
replyThe value proposition changes a lot if you can get 90% of the quality for 50% the price with a quant due to halving your hardware requirement.
Thank you for the article, though! Very informative.
> Aggregate token throughput is about 30% lower (122 vs 170 tok/s at 16 users)
replyIs that actually the aggregate throughput? 8xB300 (with 4TB/s/GPU bandwidth) is only pushing 8 tg/s/session? That seems… incredibly low, even for an A100B model.
Is it actually 122 tg/s per session? (1952 tg/s aggregate throughout)?