are you evaluating using OpenRouter.ai or other routers/gateways?
replythey have never been able to keep up with Groq/Cerebras-level inference speeds for some reason.
What's your use case?
replyI bet it is large-scale data crunching for labeling or categorization.
replyProbably not because this is usually not a time sensitive task.
replyAnd the fact that you will never be benefited from the batching they’ve explained either, you’ll be paying for every token sequential or parallel unlike nvidia or amd gpus
reply