The typical bottleneck to wider batching on consumer hardware is memory capacity for the KV-cache, not compute (even unified memory/iGPU-based platforms have enough compute to allow for some batching, and SSD offloading changes the scenario entirely). Qwen models tend to have bulky KV-caches for any given token count. But agentic swarms might end up sharing a large cache prefix, so there's scope for potential gains there.

I have no problem running two or three sequences of qwen 27B with a 3090. It's basically the recommended way, LLM inference without batching is super inefficient.

Yes, but I haven’t seen it in Unsloth/llama.cpp. I see it in Sglang and vllm. Unsloth should default to sglang imo