I don't know. As others have said, the Taalas chip wasn't small, or particularly low power, so it's hard to "imagine" what that tech in an cell phone chip might look like.
replyBut if the basic premise of "good enough LLM at insane throughput" holds, I think it could qualitatively change local uses of LLMs. At a certain speed point, you're able to move from request -> response to a cascade of tool calling and "subagents", which could allow a small model to be much more useful, if provided with a lot of local data and tool calls.
That said, this is assuming you could stuff a "good enough" model into a phone with Taalas-like technology. The Taalas tech demo was an 8B parameter model and required hundreds of watts (IIRC) to run. The efficiency was good given the speed (as I understand), but it's not clear at all that the approach scales small enough to be a sensible coprocessor on an iPhone or whatever.
Box that plugs into my desktop would be fine. Or perhaps in SSF form factor.
replyBut wouldn't higher tps allow for more reasoning or other hidden processes, potententially making a smarter model?
replyThis is my thought as well. Models have to be intentional about which tokens they burn because there's a real lag time. If you can just fork out 10 different reasoning sessions at once with no regard for token waste/lag, you can compensate a smaller model with just doing more at once with it. No idea if this is reasonably true though.
replyI think that only works if you have checkpoints where all that reasoning can be checked against reality. Otherwise you get an army of armchair experts. LLMs are hilariously bad at home improvement advice btw, where reasoning alone won’t get you far.
replyThat order of magnitude could be the difference between "the users wants me to open the notes app, let's open it" and "I've scanned all your notes before you could blink and found what you're looking for".
replyIf Siri is using a 3T model in high reasoning mode to answer your question you will.
reply