Um, dismissing the tech as "the local LLMs are dumb" seems shortsighted. I can run some pretty impressive models on my local Mac, but it has >64gb of ram and an M3 Max.
replyGiven the privacy benefit I wouldn't dismiss them so fast. I'd suggest picking one or two that your prompts will work well with and treating it as "we let you run with local models too, if you have a computer capable of that." This will (a) quiet the people who complain about everything and (b) get more people to try the cloud model knowing they could move to a local model for real usage.
I'm not dismissing them. I'm saying they're not there yet. As a startup, we have to prioritize. We can't do everything simultaneously, and it would be a substantial engineering effort to have a dual architecture as well as potentially more security holes. And the amount of people that want to run local LLMs is very small. I use local LLMs when I'm on flights, and that is my personal assessment. They are all benchmark-maxed and incapable of reliable tool calling or consistency over meaningfully long conversations.
reply