[1]: https://www.reddit.com/r/OpenAI/comments/1stsxvc/new_feature...
What you think could be a big chunk, is more likely to be a fraction of a percent of queries.
And what use is similar query caching - so you (very often! if actually cost effective, maybe half the time) get a response to a query that was different from yours. Including for when you have a lot of context input already. You’re going to get trash.
If it were constrained to only very common initial prompts, and somehow the long tail did not actually dominate as it does with Google search (can't find the reference at the moment but it was a famous article some years ago), it also wouldn't account for serious enough cost savings. Long context is what is expensive.
This might only work in constrained domains like customer service where there’s tolerance for generic answers and escalation paths. For technical work? For general purpose use, with secretly canned responses charged at full price?
By contrast, when coding, devs typically have hundreds of thousands of tokens in the context window, and may use many millions of input tokens per day.
Caching requires the full prefix to match exactly. If a single word differs near the beginning of the prompt, nothing after that can share the cache. So this type of caching would save a few queries that cost virtually nothing, but wouldn't help with the stuff where cost matters.
The only optimization that makes sense is per user prefix caching, because you are often sending the same system prompt over and over again or are continuing a conversation.
“ Automatic Prefix Caching (APC in short) caches the KV cache of existing queries, so that a new query can directly reuse the KV cache if it shares the same prefix with one of the existing queries, allowing the new query to skip the computation of the shared part.”
https://docs.vllm.ai/en/latest/features/automatic_prefix_cac...
The technique you linked only makes a substantial difference for particular use cases where you are going to have many LONG CONTEXT queries with the same prefix. For instance, when having a set of documents that commonly get loaded in as context. It's a way for application developers to keep prefixes they manage (or prefixes managed by some set of their users) cached. It has no relevance for long tail general purpose use.
The transform script(s) are cached and can be played back or adjusted. Surely for some breadth of question inputs, they map more often to similar answers--but not static answers; instead, evented edits.
It's nearly untenable for a human to keep private edit scripts to generate code changes. The extra steps for custom regex, essentially one-offs for a shared codebase, is inefficient. But maybe not to an LLM.