upvote
After I stopped letting Claude run tests, my token usage went from "gone by Wednesday" to "what can I automate this Saturday so my tokens won't go to waste".
reply
Thanks this is very helpful, i didn't realize that. But not sure what to do since running tests is how Claude often catches mistakes and iterates. If i run them myself, the cache will still expire.

However where i think you have a point is that it tends to run tests quite casually where a reviewer would have been enough to catch mistakes.

IMHO, the real problem is the 5min cache way too small. They should up it to 1 hour.

reply
What is the reason for this? Surely running tests by itself isn't necessarily token-expensive? And it provides cheap feedback for the agent: to see whether they're breaking things.
reply
My guess is that the LLM (usually Sonnet or Opus in my case) is burning tokens polling to see if the output finished - and there is often a lot of it. My current project is Rust, Python, and Kotlin for what it's worth.

I don't actually know the real mechanism, I came across this serendipitously.

reply
By default Anthropic's token cache TTL is 5 minutes. If your tests take >5m to execute, when the LLM resumes your entire context is a cache miss
reply
There are many reasons why your test suite shouldn't take >5m to execute. This is just one of them.
reply
.NET solution having 120+ projects, with 7000 tests of which maybe 1500 are integration and/or component tests.

Buid alone takes 4 minutes. Test suite heavily parallelized runs in about 2 to 3 minutes, infrastructure dependencies fully containerized using podman, no external API call.

And this is without mentioning the 5000 frontend tests (angular), including Playwright end-to-end tests wired to the backend (about 3 to 4 minutes run without the e2e tests).

And this is running on a dev machine with 64gb of RAM and an NVMe disk.

Some systems are just massive, and even if you run focused tests, it's easy for a change to have a large blast at least requiring a large rebuild, and at times to rerun a larger set of tests.

reply
I've worked on projects where the test suite would take hours.
reply
Do you have a link to the docs for that? Thanks.
reply
How do you do that?
reply
I just told it that I want to run the tests, because so much of the code was vibed at that point and I felt I was losing understanding of the application. A few days of that and I could see that my token use was much lower than usual. Going back to LLM-run testing returned the high token usage.
reply
Yeah, I should see if I can create a similar skill. The tests do spend some time to complete.

Claude: Confirmed: 195M of the 212M cache tokens (92%) come from full re-writes after 5+ minute gaps, accounting for 77% of relative cost. Now I'll check the existing wrappers so the skill builds on them.

reply
Yesterday in my CC there was an update that added a feature to auto compact immediately prior to the cache being invalidated.
reply
And I immediately turned it off because that just invites utter churn and stupidity. Part of that is just CC though, their compaction is miles behind Codex.
reply
Some to think about with both Claude and skills.

Can you invert things. What’s deterministic and what needs “thinking”?

Can you create a script that runs things chaining together the deterministic parts with calling out to Claude for glue that needs “thinking”.

reply
Don't suppose you would be happy to share the skill? Or is it fairly unique to your work flow?

I think the tests problem is really having an impact for me. Hadn't considered it before so thanks for flagging.

reply
How do you keep the caches valid?
reply
Not OP, but I typically ask it to report status of whatever is running; it can run a background task that reports to it every N minutes. With Claude Code in VS Code and a Pro subscription, it seems like the one-hour cache is the default, making it an issue less often.
reply
Ask Claude, it's pretty wild to see.
reply
> including a way to keep the caches valid

Maybe doing bogus small requests to prevent the cache from expiring?

I mean in the API you can set the cache expiration time right? But I guess claude code can't do this

reply
> my old friend claude.

Awfully new old friend.

reply
Google is the ex, Altavista is the high school sweetheart who tragically passed at a young age.
reply