My solution is to see what four or five-year-old equipment I can buy that will let me run local LLMs. I may only get six or seven tokens per second out of an i7, but it's a start. And best of all, I can turn the machine off when I'm not using it.
IMO, Migrating to small-scale local LLMs would be a significant improvement over using data centers.
LLM serving is most efficient when you batch a lot of parallel requests together. Data center solutions also have the advantage of collecting queries from around the globe, so the hardware can be utilized around the clock.
Having everyone serve their own local LLMs would produce a lot more memory demand. Not less. The same memory would be idle most of the time, and when it was used it would be used for 1 person instead of a batch of requests.
There are other reasons to run local LLMs, but solving hardware demand problems is not one of them.
This shift is probably inevitable, but it will vary significantly by region depending on prices for electricity. Look at, for example, the difference in fundamental homelab build recommendations between Germans and just about anyone else. Electricity prices in Germany are so high that even a now expensive Raspberry Pi or other ARM board is often preferred over Intel/AMD builds due to low power draw (especially low idle power draw), an effect that adds up for a machine running all the time over years.
With local LLMs and the GPUs to run it, especially if you want a model available to you all the time and can remote into your local network to use it whenever you want, there's no escaping much higher power draws, even at idle. Wherever electricity is expensive, the electric bill can be a prohibitive barrier.
Eg: I shaved ~40W off the idle load on a server (250->210W) by doung nothing more than removing the redundant supply
A college buddy used to work at Motorola (I'm naming the company because they wouldn't mind this story being shared) back in the late 90's or early 2000's. They had redundant power to their campus, bought from two different companies, coming in on opposite sides of the campus, so that even if some backhoe operator cut a ground-based power line somewhere, they wouldn't lose power.
And yet, one morning, the power went off all across their campus. After a little investigation, they sent pretty much all their employees home at noon and told them "take the afternoon off, don't come back until tomorrow, you wouldn't be able to do any work anyway". Turns out that although the power lines came in at opposite sides of their campus, somewhere a few miles away both of the power lines feeding their campus ended up running through the same underground conduit. And yes, a backhoe had managed to cut that conduit and break both of the lines they depended on at the same time. They had a single, VERY non-obvious, point of failure, and the backhoe had unerringly homed in on that SPoF.
I'll second this. The combination of Whisper + LLM makes speech recognition fantastic. I occasionally have arm pain from typing, and this is a Godsend.
I don't use it to write code - but in my experience stuff like emails + docs was the greater source of pain (one generally types slower while coding).