What Is Continuous Batching and How Does It Work?

LLM inference generates text one token at a time, and different requests finish at different lengths. With traditional static batching, the server groups a fixed set of requests, runs them together, and cannot start new ones until the whole batch is done, so the GPU sits partly idle while it waits for the longest request. Continuous batching removes that waste.

By managing the batch at the level of individual tokens, continuous batching lets requests join and leave a running batch continuously. The GPU stays full of useful work, and throughput rises sharply, which is why it has become a defining technique for serving LLMs efficiently.

This guide explains what continuous batching is, how it works, how it differs from static batching, and why it matters.

What is continuous batching?

Continuous batching, sometimes called in-flight or dynamic batching, is an LLM serving strategy in which the set of requests processed together is updated at every generation step. As soon as a request finishes, it leaves the batch and a waiting request takes its place, so the batch is continuously refilled rather than fixed for its whole run.

Think of it like a ride that lets people get on and off at every stop rather than waiting for everyone to finish a full loop. Seats never sit empty waiting for the slowest rider, so the ride carries far more people per hour. Continuous batching keeps the GPU's seats full the same way.

The key idea is per-token scheduling. Because the batch is reconsidered at each step, the GPU is rarely left waiting on a few long requests while others could run.

How does continuous batching work?

At each generation step, the server advances the batch, retires finished requests, and admits new ones.

• Run a step. The engine generates the next token for every request in the batch.
• Retire finished. Any request that just completed leaves the batch.
• Admit waiting. Queued requests fill the freed slots immediately.
• Repeat. The batch advances token by token, continuously refilled.
• Serve. Each request streams out as it generates, finishing independently.

Because slots free up and refill every step, the GPU processes a nearly full batch almost all the time. That steady utilization, rather than any change to the model, is what delivers the throughput gain.

Continuous vs. static batching

The two approaches differ in when the batch changes. Static batching fixes a group of requests, runs them to completion together, and only then starts the next group, so short requests wait for the longest one and the GPU idles. Continuous batching updates membership every token, so finished requests leave and new ones join immediately, keeping the GPU busy. Dynamic batching, a middle ground, groups requests that arrive close in time but still runs each group to completion.

The practical difference is large. Because request lengths vary so much in LLM workloads, static batching wastes a lot of GPU time, while continuous batching recovers it, often multiplying throughput on the same hardware.

Why does continuous batching matter?

• Higher throughput. Keeping the GPU full serves far more requests per second.
• Better GPU use. Expensive accelerators are not left idling between requests.
• Lower cost per request. More work from the same hardware lowers unit cost.
• Smoother latency. Requests need not wait for an entire batch to finish.
• Scales serving. It is foundational to high-volume LLM inference.

What are the considerations?

• Engine support. It requires a serving engine built for per-token scheduling.
• Memory management. Handling many concurrent requests needs careful KV cache use.
• Fairness. Scheduling must balance throughput against individual request latency.
• Complexity. The dynamic scheduling is more intricate than static batching.

When do you need continuous batching?

You want it whenever you serve LLM inference to many concurrent users and care about GPU efficiency, which is most production deployments. The more variable the request lengths and the higher the volume, the more continuous batching improves throughput and cost.

For a single request at a time or a tiny workload, the benefit is smaller. But any real serving setup with concurrent traffic gains substantially, which is why modern inference engines make continuous batching a default.

Best practices for continuous batching

• Use a serving engine that supports continuous batching natively.
• Pair it with efficient KV cache management for concurrent requests.
• Tune batch size and scheduling for your latency and throughput goals.
• Monitor GPU utilization to confirm the batch stays full.
• Combine with autoscaling so capacity matches demand.
• Balance throughput against per-request latency for interactive apps.

Conclusion

Continuous batching updates an LLM inference batch token by token, letting requests join and leave a running batch instead of waiting for a fixed group to finish. By keeping the GPU busy rather than idling on uneven request lengths, it multiplies throughput and lowers cost per request on the same hardware. It is a core technique behind efficient, high-volume LLM serving.

TrueFoundry helps teams serve models with engines that use continuous batching on their own infrastructure, with GPU utilization and throughput visible in one place. As inference demand grows, continuous batching is central to serving it efficiently.

‍