Videos will be next.
I like that they're being up front about this. Benchmarks are a metric that can be gamed. They give you an approximate estimate of where things land, but you really need to test for your use case.
As it's minor change did not expect much, but having a higher context helps for sure. I will own the errors for passing bigger context.
Very refreshing.
> For the builders AND dreamers
> Enter the danger zone at your own risk.
> Size does matter, apparently
> We’re trying something new out here: we’re going to treat you guys like adults
> Lots of people love to ask what kind of sci-fi we’re releasing next
That said, their docs are great and don't seem to include all the Reddit cringe.
I've yet to see any company match the copy writing skills of Valve though.
My team hasn't finished writing their blog posts yet, but the customer demos [2] went out the moment TypeSafe took our call and sorted all the paperwork out.
There's a tiny section at the bottom of the demo which explains how it actually works (or the github repo is the code).
We have a single decision API[1] over all similar models, so we run evals across all the "System One" api products & definitely don't trust any benchmark that you haven't run. All standardized benchmarks have been benchmaxxed already.
PS: just in case it wasn't clear, the demo is a game where we play to calibrate the human vs machine decision making (or just do it in auto-pilot mode).
If I remember well, in the background, it does the multi-choice question: "What is the component requested ? --> Calendar, Slider, ..."
i used luna 5.6 as the ranker/filter to remove useless items from upstream rag(mixing embedding, regex etc) and it did a good job but took 5-10 seconds each time. now less than 1 second and no observable precision loss. hope it helps
I guess their cheeky "Warning: AI Benchmarks, don't trust them" is their way out of that one.