upvote
I tend to disagree here. Think about how useful it would be to have a nice llm in real-time debugging your applications loaded entirely on your computer. That sounds pretty sweet to me. I don't see how that's not really cool / immediately useful to you?

Maybe your point is the majority of the time that's not needed, fair, then maybe the approach is both?

reply
Well, LLMs handle raw data perhaps a million times slower than conventional algorithms, so in "bytes" and "latency" terms, a dialup modem can handle most AI communication today - a rack of cloud GPUs 100ms away is still faster for a lot of things than $10k of hardware in your GPU, Mac, etc.
reply
> the applications today aren't latency-sensitive at all

This is circular reasoning. We don’t have latency-sensitive applications because sufficiently powerful local AI is not profitable to entertain such applications.

reply
Absolutely, and there are applications like self-driving or robotics or surgery or whatever where it's critical to have the work be local and never have a network hiccup. But these are all in some sense new markets, though I guess these would be growing faster if we had abundant local AI. Right now most of the AI applications that people pay for are ones that make human workers more efficient, and humans are already high latency.
reply
The economics point the other way because capital is aligned against you. Should this be a bubble that pops we will hopefully see prices come down again.

MooresLawIsDead and his spidey web / senses thinks it already popped:

https://youtu.be/YJeK2gumpEI

reply