upvote

points

by kuboble16 hours ago |
comments
upvoteby a_t4816 hours ago|
next
[-]
I got exactly this warning message yesterday, saying that it could use up a significant amount of my token budget if I resumed the conversation without compaction.
reply
upvoteby jhogendorn10 hours ago|
parent|
next
[-]
Compaction wont save you, in fact calling compaction will eat about 3-5x the cold cache cost in usage ive found.
reply
upvoteby _flux5 hours ago|
parent|
[-]
Wouldn't it help if the system did compaction before the eviction happens? But the problem is that Claude probably don't want to automatically compact all sessions that have been left idle for one hour (and very likely abandoned already), that would probably introduce even more additional costs.

Maybe the UI could do that for sessions that the user hasn't left yet, when the deadline comes near.

reply
upvoteby onemoresoop16 hours ago|
parent|
prev|
next
[-]
Im glad they chose to do that as opposed to hidden behavior changes that only confuse users more.
reply
upvoteby fhub16 hours ago|
parent|
prev|
next
[-]
Really good to know. That should have made it into their update letter in point (2). Empowering the user to choose is the right call.
reply
upvoteby doubleunplussed10 hours ago|
parent|
prev|
[-]
I saw that too, but that's actually even worse on cache - the entire conversation is then a cache miss and needs to be loaded in in order to do the compaction. Then the resulting compacted conversation is also a cache miss.

You ideally want to compact before the conversation is evicted from cache. If you knew you were going to use the conversation again later after cache expiry, you might do this deliberately before leaving a session.

Anthropic could do this automatically before cache expiry, though it would be hard to get right - they'd be wasting a lot of compute compacting conversations that were never going to be resumed anyway.

reply
upvoteby skeledrew15 hours ago|
prev|
[-]
> I think the best option would be tell a user who is about to resurrect a conversation that has been evicted from cache that the session is not cached anymore and the user will have to face a full cost of replaying a session

This feature has been live for a few days/weeks now, and with that knowledge I try remember to a least get a process report written when I'm for example close to the quota limit and the context is reasonably large. Or continue with a /compact, but that tends to lead to be having to repeat some things that didn't get included in the summary. Context management is just hard.

reply
upvoteby Terretta15 hours ago|
parent|
[-]
Right, and reloading that context is the same cost as refilling the cache, so really, they're charging the same, and making it hard.
reply