A tricky LLM interview question:You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces.So you add KV cache compression and evict 90% of the cached tokens.VRAM usage stays as is and GPU still runs out of memory.Why?(answer below)Evicting 90% of the KV cache can free almost none of the memory it was using.This sounds counterintuitive, but it follows directly from how production servers store the cache today.The KV cache grows with every token a model genera