undefined

points

[-]

> You'd have to test batching of multiple thinking streams in order to try and raise overall tok/s via layer-wise reuse of the streamed weights

isn’t this very straightforward to do..? I thought batching for Qwen models is already proven out.

> but this would decrease single-session performance even further

Well let’s take Qwen 3.8 27B. Throughput for M3 at 8 agents is 4x compared to single agent. [1]

It’s really not clear to me that 8 concurrent agents at half speed will be worse task completion latency than 1 agent.

And that’s M3 studio benchmarks, not even M5 ultra, and without the many software improvements we will see

If you haven’t tried Qwen 3.8 27B xhigh on a task you might not get the hype. Idk.

If you’ve tried doing this and don’t like it sure, and be specific about what isn’t effective, but let’s not speculate.

[1]: https://omlx.ai/benchmarks/performance/69kzkrv8?utm_source=c...

by zozbot2341 days ago|

parent|

[-]

That's all well and good but Qwen 27B is a small, dense model; that's favorable to both batching and MTP. Batching of large, sparse/MoE models like Kimi K3 (requiring slow SSD streaming even on a single maxed out Mac Studio) on local hardware is an entirely different game that's mostly theoretical so far: many people would even call it outright pointless. (MTP clearly fares even worse, though - unlike batching, it ends up wasting scarce weights-fetching throughput on wrongly predicted tokens.)