vLLM handles GPU scheduling, not sllm. The model weights stay resident in VRAM permanently so there's no loading/unloading per request. vLLM uses continuous batching, so incoming requests are dynamically added to the running batch every decode step and the GPU is always working on multiple requests simultaneously. There is no "load to VRAM and run" per request; it's more like joining an already-running batch.
replyTTFT is under 2 seconds average. Worst case is 10-30s.
> The model weights stay resident in VRAM permanently so there's no loading/unloading per request.
replyYes, I was thinking about context buffers, which I assume are not small in large models. That has to be loaded into VRAM, right?
If I keep sending large context buffers, will that hog the batches?
deleted
replyNot if you are the only one. We have rate limits to prevent this in case, idk, you share your key with 1000 people lol.
reply> how many work units can run in parallel
replynot original author but batching is one very important trick to make inference efficient, you can reasonably do tens to low hundreds in parallel (depending on model size and gpu size) with very little performance overhead