What Is a KV Cache and How Does It Work?

When an LLM generates text, it produces one token at a time, and each new token attends to every token before it. Done naively, that means recomputing the attention keys and values for the entire sequence at every single step, which gets expensive fast. The KV cache avoids this by remembering the keys and values it already computed.

By storing those vectors and reusing them, the model only has to compute the key and value for the newest token at each step. This single optimization is what makes interactive LLM generation fast enough to be practical, at the cost of holding the cache in GPU memory.

This guide explains what a KV cache is, how it works, what it costs, and why it matters for LLM inference.

What is a KV cache?

A KV cache, short for key-value cache, is a store of the key and value vectors that a transformer's attention layers compute for each token in a sequence. During generation, these vectors are kept in GPU memory so that when the model produces the next token, it can attend to all previous tokens without recomputing their keys and values from scratch.

Think of it like keeping notes as you read a long document. Rather than re-reading everything each time you add a sentence, you glance at your notes and add just the new part. The KV cache is the model's notes on what it has already processed, so each new token is cheap to generate.

The key idea is reuse of attention state. The expensive part, processing the whole prior sequence, is done once per token and saved, so generation scales far better than recomputing everything each step.

How does a KV cache work?

For each new token, the model computes only its key and value, appends them to the cache, and attends over the cache.

• Process the prompt. The model computes keys and values for all prompt tokens and stores them.
• Generate a token. For the new token, it computes just that token's key and value.
• Append to cache. The new key and value are added to the stored cache.
• Attend. The token attends over all cached keys and values to produce output.
• Repeat. Each step adds one entry and reuses the rest, keeping per-token work small.

Because only one token's key and value are computed per step, the cost per generated token stays roughly constant instead of growing with the full sequence each time. The trade is that the cache occupies memory that grows as the sequence lengthens.

What does a KV cache cost?

The KV cache is a classic memory-for-compute trade. Without it, each token recomputes attention over the whole sequence, so compute per token grows with length and generation is slow. With it, compute per token stays small, but the cache consumes GPU memory proportional to the sequence length, number of layers, and model size. Managing that memory is a central concern of efficient serving.

This is why long contexts are expensive: the KV cache for a very long sequence can take significant GPU memory, limiting how many requests fit at once. Techniques to compress or page the cache exist precisely to ease this pressure.

Why does the KV cache matter?

• Fast generation. It makes per-token cost roughly constant instead of growing.
• Practical interactivity. Without it, streaming responses would be far too slow.
• Memory pressure. Its size is a main limit on context length and concurrency.
• Serving design. Managing it well is key to high-throughput inference.
• Enables batching. Efficient cache handling underpins continuous batching.

What are the challenges?

• Memory growth. Long sequences produce large caches that strain GPU memory.
• Concurrency limits. Many simultaneous requests multiply total cache size.
• Fragmentation. Naive allocation wastes memory; paging schemes help.
• Long context cost. Very long contexts can make the cache the bottleneck.

When does the KV cache matter to you?

It matters whenever you serve or run autoregressive LLM generation, which is essentially all text generation. Anyone optimizing inference speed, context length, or GPU memory use is, directly or indirectly, working with the KV cache.

For a single short generation it is invisible and simply makes things fast. But at scale, with long contexts and many concurrent requests, the KV cache becomes a primary factor in cost, throughput, and how much context you can afford.

Best practices around the KV cache

• Use serving engines with efficient KV cache management, such as paging.
• Right-size context length to balance capability against cache memory.
• Monitor GPU memory to see how much the cache consumes under load.
• Favor engines that combine KV caching with continuous batching.
• Consider cache compression techniques for very long contexts.
• Plan concurrency limits with total cache memory in mind.

Conclusion

A KV cache stores the key and value vectors for tokens already processed so the model avoids recomputing attention over the whole sequence at every step. It makes autoregressive generation fast by keeping per-token cost roughly constant, in exchange for GPU memory that grows with sequence length. That memory is a central limit on context length and concurrency, which is why managing the KV cache well is core to efficient LLM serving.

TrueFoundry helps teams serve models with engines that manage the KV cache efficiently on their own infrastructure, with GPU memory and throughput visible in one place. As contexts grow longer, handling the KV cache well is key to serving them affordably.