What Is PagedAttention and Why Does It Matter?

When a large language model generates text, it keeps a growing cache of attention keys and values for every token so far, known as the KV cache. On a busy inference server with many simultaneous requests, this cache dominates GPU memory, and how it is managed decides how many requests the server can handle at once. Traditional approaches waste a lot of that memory, which caps throughput.

PagedAttention solves this by borrowing a decades-old idea from operating systems: paging. Instead of reserving one big contiguous chunk of memory per request, it stores the KV cache in small fixed-size blocks that can live anywhere in memory. The result is almost no wasted memory and far higher serving throughput. This guide explains how it works and why it matters.

We will cover the memory problem it solves, how paging applies to the KV cache, how it compares to traditional serving, and when it matters.

What is PagedAttention?

PagedAttention is a technique for managing the attention key-value cache during LLM inference by dividing it into small, fixed-size blocks that need not be stored contiguously in GPU memory. A lookup table maps each request's sequence of tokens to the scattered blocks holding its cache, exactly as an operating system's page table maps a program's memory to physical pages.

Think of GPU memory like a parking lot. Traditional serving reserves a long contiguous row for each car in case it grows, wasting spaces that stay empty. PagedAttention instead parks each car in whatever single spots are free and keeps a map of where each one is, so almost no space is wasted.

The key idea is decoupling the logical sequence from its physical storage. Because blocks can sit anywhere, memory does not have to be pre-reserved for the longest possible output, which is where traditional serving wastes most of its capacity.

How does PagedAttention work?

PagedAttention stores the KV cache in blocks and maps logical positions to them.

• Split the cache. Divide the KV cache into small fixed-size blocks.
• Allocate on demand. Give a request new blocks only as it generates tokens.
• Map with a table. Keep a page table from logical positions to physical blocks.
• Attend across blocks. Compute attention by gathering the needed blocks.
• Free and reuse. Release blocks when a request finishes, for the next one.

Because blocks are allocated only as needed and can be placed anywhere, there is almost no reserved-but-unused memory. Blocks can even be shared between requests that start with the same prompt, saving more memory. The freed capacity lets the server keep many more sequences in flight at once.

PagedAttention vs. traditional KV cache serving

Traditional serving reserves a single contiguous block of memory per request, sized for the longest output it might produce. Most requests never reach that length, so much of the reserved memory sits idle, and fragmentation wastes still more. This sharply limits how many requests fit on a GPU.

PagedAttention allocates small blocks on demand and places them anywhere, so there is almost no wasted or fragmented memory. The same GPU can hold the caches for many more concurrent requests, which raises throughput substantially. The tradeoff is a modest bookkeeping overhead for the page table, which is small next to the memory it saves.

Why does PagedAttention matter?

• Memory efficiency. It almost eliminates wasted KV cache memory.
• Higher throughput. More concurrent requests fit on each GPU.
• Lower cost. More requests per GPU means less hardware per request.
• Prompt sharing. Common prefixes can share cached blocks.
• Scalability. It underpins modern high-throughput serving engines.

What are the tradeoffs?

• Bookkeeping. The page table adds a small amount of overhead.
• Complexity. The serving engine is more complex to implement.
• Kernel support. Attention kernels must handle non-contiguous blocks.
• Tuning. Block size needs to balance overhead against granularity.

When does PagedAttention matter?

PagedAttention matters most when serving large language models to many users at once, where GPU memory for the KV cache is the bottleneck on how many requests you can handle. If you run an inference service and care about throughput and cost per request, it is one of the most impactful optimizations available, which is why it is built into leading serving engines.

For a single request at a time, or a tiny workload, the memory savings matter less. But at any real serving scale, PagedAttention is close to essential for getting the most out of each GPU.

Key points about PagedAttention

• PagedAttention stores the KV cache in small paged blocks.
• It applies operating-system paging to GPU memory.
• It almost eliminates wasted and fragmented memory.
• It lets a GPU serve many more concurrent requests.
• Common prompt prefixes can share blocks.
• It is the core idea behind high-throughput serving engines.

Conclusion

PagedAttention brings the operating-system idea of paging to LLM serving, storing the attention key-value cache in small non-contiguous blocks instead of one reserved chunk per request. This almost eliminates wasted GPU memory, letting a server handle far more concurrent requests and raising throughput. It is the memory-management breakthrough behind today's high-throughput serving engines.

TrueFoundry helps teams serve models efficiently on their own infrastructure, including high-throughput engines that use PagedAttention to pack more requests onto every GPU. When inference cost and throughput matter, efficient KV cache management is where the gains come from.