What about all the things you don't currently use an LLM for?
If a specialized chip can run a model 100 times faster, you can suddenly use it for a lot of things at sub-second latency. You can write "make white transparent and add a red outline to x.png" instead of the corresponding imagemagick invocation and perceive little to no latency difference. You can hook it up to your browser and have it yank out all advertisements live, or tell it to highlight anything that might interest you, again, live. There's probably thousands of latent use cases nobody has thought of that would be enabled by a truly fast LLM, even a mediocre one.
I don't think an on-device model needs to change much; it's already quite general in its capabilities.
Looking for a lamp in a specific style?
"Make a VR application set inside my apartment (based on all the photos of my apartment from my photos directory) with all lamps under $100 that fit Scandinavian interiors and could be delivered to my house before Friday. Place the lamp on the dining room table, allow us to: cycle through lamps, change time of day, and interact with all the lamps and furniture".
Five seconds later and bam you have this new piece of software that you'll use once and then dispose of.
AI: "I'm sorry, a security guardrail prevents me from performing this operation."
Your company will decide what makes economical sense.