The more reasonable comparison is against rented GPU's, while looking at tradeoffs in latency and upload/download/storage/instance management overhead.
Buying hardware for local models is meeting a wholly different need than buying tokens through OpenRouter or whatever.
All decades prior and up to about a year ago, I would have agreed with you. My Framework Desktop, however has appreciated in value by 75% since I bought it. Will it stay there for a long time? Probably not. But it shows that there are no hard and fast rules about things anymore.
64 can still easily do a Qwen 4.8 model, so I’m relatively happy with my purchase… plus, it’s price change has caused it to quickly appreciate in value… so I could sell it if my situation ever turned dire lol
If I'm a professional photographer chasing the best possible end product, I'm not buying cameras because they're economical. I'm buying the best camera I can get my hands on to get the best product I can produce within reason under the understanding that it doesn't have to equate to the best economic decision to be the _right_ decision.
If you're in a position to be able to take advantage of the local inference - it's a no brainer. If you're not sure how that would be done, then it's not a good move.
For example, I have a small posix-shell-based LLM harness that can SSH into my NAS and run organization tasks using the local DS4Flash that I have right now. It's already been a massive help for me to keep me organized, and that's just 2x DGX Spark's worth of compute.
But things change real fast when you're no longer bound by costs/apis/rate limits. All of a sudden it's not about "how can I do this right and efficiently" and more about "I can poke at and test _all the things_ that might make this better".
I think most people who can't see this value in the local inference approach are likely still copy/pasting from their web LLM ui's or don't even come close to subscription quotas. Meanwhile, 1b tokens a day is a light day for me with 3 $200/m subscriptions + some level of sub at basically every frontier level provider. Had I been less frugal and ponied up for the hardware before things got crazy I wouldn't need 80% of that - just the frontier models for the most complex tasks, the open weight models would handle the rest easily _and_ I'd get to do a lot more exploratory work without concern about quotas.
Employer just sent an email that.. things are changing when it comes to token spend...
What did I do with these?
Setup record/replay for our product using qemu, several variatons thereof including experiments on target hardware. Fixed a tricky bug in qemu that I sadly can't upstream..
Experimented with rr on WSL2 and our target arch. Failed experiment.
Setup mutation testing PoC.
Optimized pipelines
etc. etc. Just contung code its soo much more than I would normally produce, but its also 95% experiments that are still not productized, and much of it never will be.