If you have a semi-recent MacBook with even 32GB, you can run 20GB models that are pretty damn smart, with room to spare for the rest of your toolchain.
If you’re reasonably connected to the code you’re writing and prompting the AI at the level of the code, not the level of the feature, you can get some fantastic results.
Sure, it’s not the completely automated dreamland that’s been sold, but it’s still a speed-up on par from going from assembler to a higher-level language, which is still immense.
And for effectively free, if you have a machine that would otherwise have been considered “developer-grade” for a lot of tasks anyway.
If you set a large context window you end up using all your memory. Prefill is slow. And the 10-20B models are somewhat poor at understanding intent and tend to think in circles, so if you set a small thinking budget to control context size the model ends up answering based on some half formed thoughts.
Or something like batch document summarization or image classification or whatever yeah it's great. But for "agentic" and coding workflows I think you would need a completely different harness setup and prompting style compared to what we are used to from the big proprietary models. I certainly have not yet figured out what that is.