We're on the cusp of Kimi K3 becoming usable on sub-10k hardware.
replyFor values of "usable" that include "14.6 seconds/token". It's a cool accomplishment! And newer hardware would speed it up some. But I think I'd want something a bit faster before declaring it usable in practice.
reply14 seconds per token? Not tokens per second. Seconds per token? That’s nowhere near the cusp!
replyThis is a bit of a strawman. The parent comment wasn't suggesting that the inherent defects magically disappear nor was it suggesting that local inference today is sufficient for all tasks.
replyAt the heart of it, self-hosting liberates your use cases from all the horribly opaque configuration, shadow prompting, etc. And local models are only getting better and more diverse every month.
>The parent comment wasn't suggesting that the inherent defects magically disappear
replythat is almost exactly what they said though..?
"Want it to go away, almost like magic? Local inference. [...] all of the common LLM defects will go away.
> This is a bit of a strawman. The parent comment wasn't suggesting that the inherent defects magically disappear
replyThe parent comment literally said that the common LLM defects would go away.
Direct quote:
> all of the common LLM defects will go away
I get your reading. My interpretation is that they’re claiming enshittification and that the local models don’t have that.
replyIt’s an understandable view, but I’d be astonished by any local model processing very long context better than any frontier model (and now many racks are we talking).
> Want it to go away, almost like magic?
replyTrue, almost like magic and magic are not the same thing, but I'd hesitate to call this a strawman.
[dead]
reply