I'm curious if you could comment on why your DeepSWE performance is below Haiku. This is not a dig - getting 46% vs. Haiku's 52% is still amazing. I'm just wondering if perhaps your post-training was less aggressively RL'ed on coding tasks than perhaps Haiku was. Having a cheap 1,000 tok/s option for coding work is a dream because - if it's reliable enough - you could blast through work such as high-level security reviews or finicky and repetitive refactorings that don't necessarily require SOTA performance.
replyWe might not be able to precisely explain the 6% difference on that specific benchmark, but overall we have found the model to be Haiku- or Luna-level in our own daily usage. That means that it won't yet replace Opus or Fable as the primary driver of a harness, but we have used it successfully for subagents, including implementing kernel optimization ideas produced by a larger model (and those improvements have gone into production -- a form of RSI).
replyIn general, the model continues to improve when given additional high quality data like a normal LLM, which is what makes us very bullish on the scaling of dLLMs.
Google's research showed some of the speed benefits tended to drop off at larger batch sizes on an H100. Are there published batch-size vs tokens/s figures for Mercury 3?
replyWe've been happy with how performance has scaled with batch size. The model runs in our API at reasonable costs, which is also what allows us to serve the model with competitive prices ($0.10 per 1M input and $0.35 per 1M output at launch).
replyPricing can always be set at a loss, however, particularly for a private company. Are there solid published numbers for a specific hardware spec?
replyWhats the model total param size and will you release the weights?
What does harness integration look like? Do you have an harness? Is it good at coding? What about tool calls? Do we have to wait a fixed amount of time for every step it takes?
replyWe're not releasing the weights, but the model is available using a standard API, and it integrates with your favorite harnesses.
replyHere is a page that compiles the list of integrations: https://docs.inceptionlabs.ai/integrations/overview
For example, it's just a few small steps to run the model in Cursor: https://docs.inceptionlabs.ai/integrations/cursor
You shouldn't expect Opus-level performance yet, but early testing has found the model to be great for coding subagents.
Will you make the weights available?
replyNot at the moment, but we are looking to share more information about the model in the future via technical blog posts and technical reports.
replyOk. Good luck then.
replyIncredible. Any luck in having a model that's multimodal in the future?
replyThis is an area that we're really excited about. We have internal prototypes, and if you're interested in trying it out as an early tester, please reach out to hello@inceptionlabs.ai
replyWhy medium in benchmarks? I've not seen that many people use other than xhigh/max on the cheap models. No low->max performance curve either. Time to task completion would also be nice!
replyWe are going to be sharing additional information like task completion times in the near future as the model undergoes additional benchmarking by third parties. We also plan to share data on specific use cases where the model traces new points of the Pareto frontier and and speed cost.
replyFor a preview of the latter, check out our analysis of the Mercury Voice model: https://www.inceptionlabs.ai/blog/introducing-mercury-voice
If I may, are there plans for visual input support?
replyWe have internal prototypes, so please reach out to hello@inceptionlabs.ai if you're interested in early access
replyIs the model size secret?
replyWe're not publicly announcing the model size right now, but we can compare against other models with a similar intelligence -- what we see is a big boost in speed for the same level of capabilities.
replyI don't see it on Baseten. Is it special in some way?
replyreply
Not listed among https://app.baseten.co/model-apis
replyDoes it mean dedicated inference only?
You have to reach out to Baseten directly for access. You can also access through our API (https://platform.inceptionlabs.ai/) or OpenRouter.
reply