LLM Router: The Three Things That Name Actually Means

Why the term is confusing

Ask three teams what their LLM router does and you get three answers that do not overlap. One runs GPT-4o on both Azure and OpenAI and wants traffic to survive an outage. One got their bill and wants the easy 70% of requests served by something cheaper. One has a German subsidiary whose prompts must not leave the EU. All three need different configuration, and two of the three would get nothing from what the others built.

The taxonomy is worth stating precisely, because it is what most llm routing content skips:

The tell is what each reads to decide. Load balancing reads infrastructure state, model selection reads the request body, data routing reads metadata and identity. Three inputs, so three features — and they compose rather than compete.

Kind one: load balancing across interchangeable targets

This is the oldest and least interesting sense of the word, which is why it is also the most reliable. You have several ways to reach the same capability — azure/gpt-4o and openai/gpt-4o, or two deployments in different regions — and you want requests spread across them so no single failure is your failure.

The premise is that every target is equivalent for correctness. Any could serve the request; you are choosing on availability, capacity, and speed. That is what makes llm load balancing safe to automate. And provider latency is not stable enough to ignore — it varies by model, region, provider, and hour:

Chart of measured latency variance across several LLM providers over one month

TrueFoundry configures this on a virtual model — a name your application calls, like my-group/production-chat, with one routing strategy and a list of real targets behind it. Three strategies qualify as load balancing:

‍