undefined

points

[-]

Thanks this is very helpful, i didn't realize that. But not sure what to do since running tests is how Claude often catches mistakes and iterates. If i run them myself, the cache will still expire.

However where i think you have a point is that it tends to run tests quite casually where a reviewer would have been enough to catch mistakes.

IMHO, the real problem is the 5min cache way too small. They should up it to 1 hour.

by tasuki8 hours ago|

prev|

[-]

What is the reason for this? Surely running tests by itself isn't necessarily token-expensive? And it provides cheap feedback for the agent: to see whether they're breaking things.

by dotancohen7 hours ago|

parent|

[-]

My guess is that the LLM (usually Sonnet or Opus in my case) is burning tokens polling to see if the output finished - and there is often a lot of it. My current project is Rust, Python, and Kotlin for what it's worth.

I don't actually know the real mechanism, I came across this serendipitously.

by erincandescent5 hours ago|

parent|

prev|

[-]

By default Anthropic's token cache TTL is 5 minutes. If your tests take >5m to execute, when the LLM resumes your entire context is a cache miss

by tasuki4 hours ago|

parent|

[-]

There are many reasons why your test suite shouldn't take >5m to execute. This is just one of them.

by rednb1 hours ago|

parent|

[-]

.NET solution having 120+ projects, with 7000 tests of which maybe 1500 are integration and/or component tests.

Buid alone takes 4 minutes. Test suite heavily parallelized runs in about 2 to 3 minutes, infrastructure dependencies fully containerized using podman, no external API call.

And this is without mentioning the 5000 frontend tests (angular), including Playwright end-to-end tests wired to the backend (about 3 to 4 minutes run without the e2e tests).

And this is running on a dev machine with 64gb of RAM and an NVMe disk.

Some systems are just massive, and even if you run focused tests, it's easy for a change to have a large blast at least requiring a large rebuild, and at times to rerun a larger set of tests.

by dotancohen2 hours ago|

parent|

prev|

[-]

I've worked on projects where the test suite would take hours.

by dotancohen2 hours ago|

parent|

prev|

[-]

Do you have a link to the docs for that? Thanks.

by tomrod9 hours ago|

prev|

[-]

How do you do that?

by dotancohen7 hours ago|

parent|

[-]

I just told it that I want to run the tests, because so much of the code was vibed at that point and I felt I was losing understanding of the application. A few days of that and I could see that my token use was much lower than usual. Going back to LLM-run testing returned the high token usage.