That's not exactly the math. Theres also vram needed for context. I operate several 72-128 GB machines and the larger the context the slower they go.

And the context takes space +kv cache. KV cache drives usefulness as your context grows, it needs to pull the kv cache.

Simplified, the context has to be run on every turn, so the KV cache supplies the processed tokens, so it just needs the new inpute.