But I’m having a lot of luck just running things when I’m away from the computer and can leave it plugged in.
It starts going weird (unreliable and slow) with context over 80k so you have to pick tasks one at a time and baby sit a lot more than Claude. But it really is very capable and feels like there’s an intelligence there to talk to. Maybe gpt-4 level clever?
I have an m5 max 64gb and I think anything slower would be quite painful.
Try using Goose instead of Claude's harness? Goose doesn't load as much context in at the start, so it might be more usable. I've definitely been able to get it past 128K, although I typically don't go beyond 70-80k. MoE might also be better at this.
Edited to add: for agentic workflow I’m running omlx which tells me it has about a 90% cache hit rate (tradeoff is some disk and mem space) - that noticeably changes the felt speed.
- find a free GPU droplet on digital ocean
- fire it up
- pull in a snapshot of the model + extra files/packages etc
- set up a ssh tunnel so that the localhost:8000 routes to the above
Then I just configured OpenCode to use the above and was off to the races.
Works out to be about ~$2/hr all said and done which isn't bad as I only pay when I'm using it (but could get expensive with 24/7 running)
What GPU you end up with for that price? Vast.ai (https://cloud.vast.ai/?priceInstanceHourlyMax=2) has a bunch of setups available to reach 192GB VRAM under $2 :) Quick skim showed 4x48, 2x96 and 8x24, all for under 2 buckaroos or around there.
I actually dislike LLMs. But I'm a realist, and on-demand compute like this is massive cost saving measure.
(persistent drives are relatively cheap, compared to a box with several GPUs.. or even one. I find it worth the expense)