The good thing with most NNs is that you can go with fp32 almost always (I put almost because I'm not 100% sure that this covers all the cases) for almost 0 performance degradation.
For Parakeet I'm 100% sure because I used it yesterday and it was using just 1GB.
replyWhy do you use GLM 5.3 tho? Unless you have an instance running "free" 24/7 you could use a local Gemma model. It should be good enough, no? Or maybe with openrouter it's cheap enough for such simple requests to be considered free?
Yeah maybe I'll play around with different precision, I did see improvement when I went from fp4 to full fidelity though.
replyThis is my setup for Home Assistant voice, parakeet for STT, GLM 5.3 to generate responses, and Kokoro for TTS. I use local routing for simple commands like turn on the lights, but if I want to ask questions or something, it gets routed to GLM 5.3. It is basically free, half a cent per turn maybe, so in a month I may spend a dollar? Not even usually.