9070XT operator here: I'm using llama.cpp with the same model and quant and I'm getting 87,000 for my context limit. I tried the Unsloth models but they lowered it to around 30-40K so I went back to upstream.
replyI'm on Linux and using some sort of unholy mess of ROCM libraries that I don't understand.
Could you comment more on how you set this up? I have a mostly idle 9070XT I use for gaming, and I was considering using it with the newer local open models. Many thanks.
replyI got Qwen3.8-27B running on my RX 9070 XT by setting up AMD’s ROCm 7.14/TheRock runtime for gfx1201 and adding the required HIP/LLVM paths to Windows. The official llama.cpp ROCm build detected the card correctly, but kept crashing with 0xC0000005 while loading the model, so I switched to the Windows Vulkan build of llama.cpp instead, which worked reliably.
replyI’m running Qwen3.8-27B-Unleashed UD-Q3_K_XL, which is a ~12.3 GiB Q3 quant, fully offloaded to the 16 GB 9070 XT. I disabled the vision projector to save VRAM and use one inference slot, Flash Attention, Q4 KV cache, --fit off, and --ctx-checkpoints 0. I’m running it with a 64K context window. The AMD driver also needs to be recent enough for ROCm 7.14; I targeted Adrenalin 26.6.4 or newer.
What are you offloading to ram (or even CPU)? I’m using a 9080 (not XT) and having trouble with context/token rates
replyI’m running Qwen3.8-27B-Unleashed UD-Q3_K_XL, which is a ~12.3 GiB Q3 quant, fully offloaded to the 16 GB 9070 XT. I disabled the vision projector to save VRAM and use one inference slot, Flash Attention, Q4 KV cache, --fit off, and --ctx-checkpoints 0. I’m running it with a 64K context window. The AMD driver also needs to be recent enough for ROCm 7.14; I targeted Adrenalin 26.6.4 or newer.
replyYou can offload the vision projector to CPU/sysRAM
replyAwesome. How would you setup that ?
reply --no-mmproj-offload
Running Q3 on 5060ti with 64k context. It runs great
reply