What Is Load Balancing and How Does It Work?

A single model instance can only handle so much traffic. Push past its capacity and requests queue, latency climbs, and some calls start to fail. Adding more instances helps only if the traffic is spread across them evenly, and that is the job of load balancing.

A load balancer sits in front of a pool of instances and hands each incoming request to one of them. By spreading the work, it keeps any single instance from being swamped, raises the total throughput of the pool, and keeps the service up even when one instance goes down.

This guide explains what load balancing is, how it works, the common strategies, and how it applies to LLM traffic.

What is load balancing?

Load balancing is the practice of distributing incoming requests across multiple instances that can each do the same job. The instances are interchangeable, so it does not matter which one serves a given request. The balancer's only concern is spreading the load so the work is shared and no instance is overloaded.

Think of a bank with one queue and several tellers. A single teller would create a long line, but a host directing each customer to the next free teller keeps the queue short. A load balancer is that host for requests.

This is different from routing. A router picks between different options that vary in cost or quality. A load balancer spreads work across copies of the same option. Many systems use both: route to the right model, then balance across its instances.

How does load balancing work?

A load balancer tracks a pool of instances and decides, per request, which one should handle it.

• Register the pool. The balancer knows the set of instances that can serve requests and keeps their addresses.
• Health check. It regularly checks each instance and marks any that are slow or failing as unavailable.
• Pick an instance. For each request it chooses a healthy instance using its balancing strategy, such as round robin or least latency.
• Forward the request. It sends the request to the chosen instance and returns the response to the caller.
• React to changes. As instances are added, removed, or recover, the balancer updates the pool and rebalances traffic.

Health checks are what make a load balancer more than a splitter. By taking failing instances out of rotation, it routes around problems automatically, which is a large part of how a service stays available.

What are the common load balancing strategies?

• Round robin. Hand each request to the next instance in order. Simple and even when instances are similar in capacity.
• Weighted. Give stronger instances a larger share of traffic by assigning weights, useful when the pool is not uniform.
• Least connections. Send each request to the instance with the fewest in-flight requests, which balances uneven workloads well.
• Least latency. Route to the instance responding fastest right now, which keeps response times low.
• Hash-based. Map a key, such as a user ID, to a fixed instance so related requests land together, useful for cache locality.

Load balancing vs. routing and fallback

These three often work together but do different jobs. Load balancing spreads traffic across interchangeable instances to avoid overload. Routing chooses between different models based on cost, speed, or task. Fallback is what happens on failure, retrying a request on another model or instance.

A request can pass through all three: the router picks the best model, the balancer spreads the load across that model's instances, and fallback steps in if the chosen instance or model fails. Together they cover performance, cost, and reliability.

Load balancing for LLMs

LLM workloads load balance across more than just servers. You can spread traffic across replicas of a self-hosted model to raise throughput, across several provider accounts or keys to get past per-account rate limits, or across regions to keep latency low and meet data residency rules.

Because LLM calls are slow and uneven in cost, strategies that react to real load, like least connections or least latency, often work better than plain round robin. A gateway can combine load balancing with rate limits and routing so the whole pipeline is managed in one place.

Why use load balancing?

• Higher throughput. Spreading work across instances lets the pool handle far more traffic than any single instance.
• Steady latency. Keeping any one instance from overloading holds response times stable under load.
• High availability. Health checks route around failed instances, so the service survives individual outages.
• Scalability. Adding instances to the pool increases capacity without changing application code.
• Past rate limits. Spreading LLM traffic across accounts or keys raises the effective rate ceiling.

Best practices for load balancing

• Use health checks so failing instances are taken out of rotation automatically.
• Prefer load-aware strategies like least connections or least latency for uneven LLM workloads.
• Weight instances by capacity when the pool is not uniform, rather than splitting evenly.
• Combine load balancing with fallback so a failure retries elsewhere instead of erroring.
• Spread across provider accounts or regions to raise rate ceilings and meet residency rules.
• Run balancing in a gateway so it sits alongside routing, rate limits, and observability.

Conclusion

Load balancing spreads requests across a pool of interchangeable instances so none is overwhelmed. It raises throughput, holds latency steady, and keeps a service available by routing around instances that fail, and for LLMs it extends to replicas, provider accounts, and regions.

TrueFoundry's AI Gateway load balances across model instances and providers with latency-based and weighted strategies, alongside routing, fallbacks, and rate limits in one layer. For any LLM system running at scale, load balancing is what turns a group of instances into capacity you can rely on.

‍