It might be fine for one-off applications, but it's incredibly inefficient for something that has to be done over and over.
replyWhy would you use a GPU and gigabytes of VRAM running a non-deterministic process for something a program can do on the CPU in milliseconds with kilobytes of memory?
Probably for the same reasons most of our desktop applications have become slow Chromium instances consuming multiple GBs of RAM, each:
replyIt’s cheaper to hire developers, and quicker to ship software. The user experience be damned.
The logic here is a bit different. Consumer software is slow because it's the user paying for the compute to run those bloated Electron apps. Unless we're all running local inference in the future, it would be the vendor paying for the compute to run the AI-based UI.
replyOf course they would pass that along to the consumer, but nobody is going to pay token-based pricing for something that used to be free and ad-supported, or a cheap monthly subscription.
This demo is already using a local model. Which is why I made the Electron comparison.
replyI don't really disagree, but... why do we use Javascript for so many UIs? Badly designed web pages with ads can easily use gigabytes of RAM.
replyEdit: oh, I saw you wrote a related comment after I posted this. So you already know that direct LLMs for UIs are all but inevitable...