Compaction wont save you, in fact calling compaction will eat about 3-5x the cold cache cost in usage ive found.
replyWouldn't it help if the system did compaction before the eviction happens? But the problem is that Claude probably don't want to automatically compact all sessions that have been left idle for one hour (and very likely abandoned already), that would probably introduce even more additional costs.
replyMaybe the UI could do that for sessions that the user hasn't left yet, when the deadline comes near.
Im glad they chose to do that as opposed to hidden behavior changes that only confuse users more.
replyReally good to know. That should have made it into their update letter in point (2). Empowering the user to choose is the right call.
replyI saw that too, but that's actually even worse on cache - the entire conversation is then a cache miss and needs to be loaded in in order to do the compaction. Then the resulting compacted conversation is also a cache miss.
replyYou ideally want to compact before the conversation is evicted from cache. If you knew you were going to use the conversation again later after cache expiry, you might do this deliberately before leaving a session.
Anthropic could do this automatically before cache expiry, though it would be hard to get right - they'd be wasting a lot of compute compacting conversations that were never going to be resumed anyway.