dang only for certain nvidia GPUs, had my hopes up
replyOk, I need to try that. I'm getting 45tok/s with vLLM on my 6000. >600tok/s concurrent, but 45tok/s single request.
replyEven without ninfer I would get over 80 on LM studio with default settings, so it should be noticeably more on 6000. You might want to try different a different inference engine or settings.
replycan't second ninfer enough. amazing tech
reply