They cant push it too low. The license agreement it is released under wont allow it.
reply> If the Licensee or any of its affiliates operates a Model as a Service business, and the aggregate revenue of the Licensee and its affiliates exceeds 20 million US dollars (or the equivalent in other currencies) in total over any consecutive 12 months, the Licensee must enter into a separate agreement with Moonshot AI before using the Software or its derivative works for any commercial purpose.
I think the results might be underwhelming - AI providers need to turn a profit and can't subsidize, and they're working off of the commodity hardware everyone does.
replyI wouldn't be surprised if they started offering potentiall bad quantizations with much reduced capability at lower prices (without telling the users, of course)
I would be surprised, considering that OpenRouter requires disclosing the quantization and shows automatic benchmarks to compare between providers for the same model.
replyThe latter is a joke
replyI saw that it runs GPQA Diamond and TAU-Bench Airline and shows the results over a 32 day rolling average.
replyOther than that they track Tool call error rate and Structured output error rate.
I only discovered this today, and it seems like a good idea. What are the problems in practice?
As long as they are transparent about what quant they serve the model and any other optimization they do that also affects performance of inferred tokens.
reply