deleted
replyIs it that silly? You could run multiple 27B models in parallel.
replyYou actually don't need more RAM to batch multiple inference tasks of the same model.
reply(Each task needs its own context, but the (e.g.) 27B of constant parameters isn't duplicated).
You definitely need more RAM if you are not satisfied with small context windows, especially if the weights take a large % of the total memory to boot.
replycan run multiple subagents of Qwen 27B though, right? Unless I am fundamentally misunderstanding how VRAM constraints work
replyYou might be. Running another agent doesn't load a set of new weights. It creates a new KV cache for the agent and adds the prompts to the queue. Its just another inference turn.
replythanks, I naively assumed when, for example, Claude Code starts subagents it loads a new instance with empty context
replydeleted
reply