What would tokens/sec performance look like for a reasoning model? An order of magnitude slower?
replyReasoning models are the same speed. They’re just post trained with RL to do CoT inside tags like <thinking></thinking> before a tag like <response></response>
replyThere’s no difference in the inference implementation, parameter count, or speed.
There's a difference in the latency distribution between when you submit a query and you see the response, which is what the comment is (clumsily) asking about.
replyBut yeah, there are a lot of factors, so it's hard to answer, and tokens/s isn't the right question.
Which model? Or how many active parameters?
replyLlama 3.1 8B model
replySo this demo is around 90 times faster than typical speeds for the same model at openrouter, and around 30 times faster than the absolute fastest option available (Groq).
reply
im assuming energy expenditure is substantially lower as well
replydeleted
reply