How refreshing to hear this kind of thinking, in a thread where most people have given up on local computing in exchange for convenience and permanent third-party dependency. It's the old-school way, and with embedded systems affordable and ubiquitous, hopefully a growing segment of the new generation, to push the limit of available hardware and see how far we can take it. Not just to squeeze maximum value out of constrained devices, but as an engineer there's a satisfaction in solving things with what you got.

There's a new technique, 1-bit family of language models that can achieve up to 9x memory efficiency compared to existing models. Still multiple gigabytes for practical use I imagine, but it's great progress toward local AI, which I believe will be common in the near future. https://prismml.com/news/ternary-bonsai