There's no benchmarking because it's not very intelligent at all. Right now everybody's being hype-shotted into believing you can use it for intelligent decisions.
replyI'm working on something from a crappy laptop, those numbers from jev can totally be matched:
reply Local Latency: 0.1813 secondsI can also run a 0.6b model on my phone faster than openai can run astra, it doesn't mean my model is useful.
replyand frankly for many of the kind of thing people probably want to use this for... you would want to run locally anyways.
replywhy even bother with a network hop? build a specialized engine which does the prefill->measure cycle on local GPU/TPU/NPU with a model fine tuned for your application (e.g. gaming NPCs, autonomous driving, agricultural intelligence, drone.. target... selection, whatever)
the nice thing is that if you're skipping decode you're not as memory bandwidth bound.
There's also prior art. Or probably, anyways.
replyhttps://www.reddit.com/r/LocalLLaMA/comments/1wijo3e/i_liter...
Not only is it replicable as you say, things like it already exist(ed).
The important bit of course is in the actual implementation: a) models fine tuned to produce good results for these types of questions and b) runtimes optimized to do this quickly and at scale