Hipfire works on my 7900xtx best of all. Biggest surprise - Qwen3.6-27B dense does not grind to a halt with context depths all the way up to 250K! Measured at 1K, 8K, 32K, 64K, 128K, 192K, 250K - Hipfire speed holds close to 40 tok/s. Finished 5 days run of gpu 100% inferencing, Hipfire did not crash even once afaics. I assumed speed dropping like a stone on Qwen models was a feature/bug of the model, and "nothing can be done about it". Hipfire showed me wrong - pleasantly surprised there.
replyThanks, sounds promising. Written in Rust, no PyTorch etc, sounds like my kinda tool. Surprised I haven't seen more folks mention it here.
replyAlso if I may ask, what does the rest of your stack look like (agent, harness etc)?
I just switched to Vulkan, and be done with it. :)
replyAs much as I can tell, the ROCm version of llama.cpp would be a bit faster on prompt processing, but about the same on the token generation as Vulkan. Real life benchmarks don't seem to give any "ROCm or nothing" sort of vibes. And the difference between the performance of different models are way bigger than the difference between the llama.cpp versions (and versus different runtimes like the llama.cpp/GGUF and the MLX runtimes on Mac for the same models)...
I've tinkered enough with the serving, that I'd rather do something with them with, say 10% slower speed, than spending hours on seting things up again... YMMV