Classify each request as simple, medium, or complex and route it to the cheapest model that can handle it.
Most production traffic is not uniformly hard. A greeting, a one-line factual lookup, and a request to design a distributed rate limiter all arrive on the same endpoint — and if every one of them is served by your most capable model, you pay top-tier prices for the easy majority.Complexity-based routing classifies each request into a complexity tier — simple, medium, or complex — and routes it to the target you configured for that tier. Your application keeps calling one virtual model name; the AI Gateway decides which model actually serves each request.Unlike the other routing strategies, the routing decision depends on what is in the request, not on weights, priorities, or measured latency.
Complexity-based routing is available on virtual models only. It is not a valid rule type in the tenant-level Routing Config YAML (gateway-load-balancing-config).
The virtual model must also meet these conditions, or it is rejected when you save it:
Model types must be limited to chat, completion, and responses. Embedding, image, audio, rerank, and moderation are not supported.
Each target serves exactly one tier, and the same model cannot appear under two tiers.
The classifier model (LLM strategy only) must be a catalog chat model, not a virtual model.
If any target defines metadata_match, non-matching targets are dropped before anything else happens. Classification and escalation only ever consider the remaining targets.
2
Check the session pin
If sticky routing is configured and this session already has a pin, the pinned tier and target are reused and classification is skipped entirely.
The targets for the classified tier come first, followed by the escalation chain — higher tiers, then lower tiers.
5
Send the request, falling forward on failure
The first target is attempted with its own retry configuration. If it fails with a fallback status code, the AI Gateway moves to the next target in the chain.
6
Pin the session
Once a target returns a successful response, the tier and target are written to the session pin (when sticky routing is enabled) so subsequent turns skip classification.
Multi-step summaries, drafting, standard code, light reasoning
A balanced mid-tier model
complex
Deep reasoning, long context, hard problem-solving, agentic chains
Your most capable model
These descriptions are the same ones shown next to each tier in the dashboard.
You do not have to configure all three tiers. Configuring only simple and complex, for example, is valid — requests classified as medium escalate to complex. See Escalation and fallback.
Two strategies are available. The heuristic classifier is the default and costs nothing; the LLM classifier is more accurate on ambiguous prompts but adds a model call to every request.
Heuristic (default)
LLM classifier
Added latency
None — runs in-process
One chat completion before the request is forwarded
Both strategies look at the same two pieces of text, extracted from the request body:
The last user message — the current turn.
The last system or developer message — for context. This includes the top-level system field on Anthropic /messages requests and the top-level instructions field on /responses requests.
Earlier turns, assistant replies, and tool messages are ignored. Consistency across a multi-turn conversation comes from sticky routing, not from classifying the whole history.
Classification scans at most 8,000 characters of user text and 2,000 characters of system text. A prompt longer than that is still recognised as a long prompt.
The heuristic classifier reads the prompt text, looks for a fixed set of signals, and picks a tier. It needs no configuration — this is what you get if you do not set classification_strategy at all.
routing_config: type: complexity-based-routing classification_strategy: type: heuristic # this is the default — the block can be omitted
What the heuristic looks for
These signals push a request toward a higher tier, listed from most to least influential:
Signal
What it matches
Code
Programming keywords in the user or system text — function, class, api, docker, sql, language names, and similar.
Explicit reasoning requests
Phrases in the user text that ask the model to reason — step by step, think through, pros and cons, compare and contrast, explain your reasoning, and similar.
Technical vocabulary
Systems and ML terms — architecture, distributed, neural network, concurrency, throughput, and similar.
Prompt length
Long user prompts.
Multi-step structure
Cues such as first … then, step 2, numbered or lettered lists.
Many questions at once
More than three question marks in one request.
These push a request toward a lower tier:
Signal
What it matches
Lookup and small-talk phrasing
what is, who is, how many, define, hello, thanks, and similar.
Short prompts
Very short user prompts.
Reasoning override. Two or more distinct reasoning phrases in the user text always route to complex, whatever the other signals say.Empty prompts. A request with no classifiable text is treated as simple.The signals and keyword lists are fixed and cannot be customised. If they don’t match how difficulty shows up in your traffic, use the LLM classifier instead.
Worked examples
Prompt
Signals matched
Tier
Hi, thanks!
short prompt, small talk
simple
What is the capital of France?
short prompt, lookup phrasing
simple
Write a Python function that reverses a linked list.
code
medium
Design a distributed rate limiter. Walk me through the trade-offs step by step and explain your reasoning.
two reasoning phrases → override
complex
Our gRPC service has high tail latency under concurrency. Analyze this and break down the likely causes.
The AI Gateway classifies the request by calling a fast chat model you nominate. Use this when the heuristic’s keyword-based view is too coarse for your traffic — for example domain-specific prompts where difficulty is not signalled by vocabulary.
routing_config: type: complexity-based-routing classification_strategy: type: llm-classifier model: openai-main/gpt-4o-mini # must be a catalog chat model timeout_ms: 2000 # default: 2000 fallback_strategy: type: heuristic # default: heuristic
Each classifier call is a billable gateway request. It appears in your request logs, and its tokens and cost are attributed to the same tenant and subject as the request that triggered it. Use a small, fast model and keep timeout_ms tight.
How the classifier call works
The classifier is sent the request’s system context and user text, along with a short instruction block describing the three tiers.
Its reply is constrained to a JSON schema ({"tier": "simple" | "medium" | "complex"}) and capped at 256 output tokens. Models that ignore the schema and answer in prose still work — the AI Gateway extracts the first tier word it finds.
The call is abandoned after timeout_ms (default 2000), and the fallback strategy applies.
If the classifier times out, errors, returns a non-2xx status, or returns nothing usable, the configured fallback strategy decides the tier. The classifier failure never fails the request.
classification_strategy: type: llm-classifier model: openai-main/gpt-4o-mini fallback_strategy: type: heuristic # classify with the built-in heuristic instead
classification_strategy: type: llm-classifier model: openai-main/gpt-4o-mini fallback_strategy: type: static default_tier: medium # always use this tier; default: medium
Use heuristic to keep a content-aware decision during an outage, or static to send everything to one known tier. default_tier: complex preserves answer quality at higher spend; default_tier: simple does the opposite.
Complexity-based routing does not just pick one target. It builds a fully ordered chain, so a request still gets served when the tier it classified into has no usable target.For a request classified into tier T, the order is:
Targets in tier T, in the order you declared them.
Targets in higher tiers, working upward one tier at a time.
Targets in lower tiers, working downward one tier at a time.
So a request classified as simple is attempted on simple, then medium, then complex. A request classified as complex is attempted on complex, then medium, then simple. A request fails only after every target has been tried.Each target in the chain uses its own retry_config, fallback_status_codes, and fallback_candidate settings — these behave exactly as they do for the other routing strategies.
Escalation also covers tiers you did not configure. With only simple and complex targets, a request classified as medium is escalated straight to complex and the response header reports the escalation. This is how you run a two-tier setup.
Unhealthy-target cooldown does not apply here. Under the other strategies a failing target is demoted to the end of the list; under complexity-based routing, target order is always tier order. Per-target rate limits and budgets are still enforced, and a rate-limited or over-budget target is skipped.
By default every turn of a conversation is classified independently, so a chat can start on your cheap model, move to your expensive one on a hard follow-up, and move back again. That is the cheapest behaviour, but it can be visibly inconsistent within one conversation.Sticky routing pins a session to the tier and target chosen on its first classified turn. Subsequent turns in the same session skip classification entirely and go straight to the pinned target.
Session identifiers tell the AI Gateway which request fields identify a session. All configured identifiers are combined into one session key, and you can mix headers and metadata:
key — the header or metadata field name to read.
source — headers or metadata.
TTL (ttl_seconds) must be between 300 (5 minutes) and 3600 (1 hour) — a shorter range than weight-based sticky routing allows. The window slides: every turn that reads the pin refreshes its expiry, so an active conversation stays pinned and an abandoned one expires.
What else to know
Redis is required. Pins are stored in Redis and shared across all gateway pods. If Redis is not configured or is unreachable, sticky routing no-ops and every turn is classified independently.
Pins are written only after success. A target that failed is never pinned.
Sessions are isolated. Two different session identifier values are two different pins.
Stale pins are discarded. If the pinned target is no longer an eligible target of the virtual model, the request is reclassified.
Missing identifiers. If none of the configured identifiers resolve to a value on a request, there is no session to pin and the request is classified normally.
Complexity-based routing is configured on a virtual model, alongside the other routing strategies. See Create a virtual model for the full walkthrough.
1
Create the virtual model and choose Complexity
Go to AI Gateway → Models → Virtual Model and add or edit a model. Give it a Name, and under Model Types select only from Chat, Completion, and Responses.Under Balance them across following targets based on, choose Complexity.
Creating a virtual model with Complexity selected as the routing type
2
Pick a classification strategy
Heuristic Classification is selected by default and needs nothing else.Choose LLM Classification instead to classify with a model, then set Classifier Model, Timeout (ms), and Fallback Strategy. Switching Fallback Strategy to Static Fallback adds a Default Tier field.
LLM Classification with a small, fast model as the classifier
3
Set a target for each tier
The form has one Target slot under each of Simple, Medium, and Complex. Pick a model for each tier you want to serve and leave the rest empty — an empty tier is skipped and its traffic escalates.Each target carries its own Fallback status codes, plus optional Retries, Override Params, Override Headers, and Metadata Filters — all behaving as they do for the other routing strategies.
A target for each tier, from lowest to highest cost
Current dashboard limitations. The virtual model form supports one target per tier, and sticky routing is not yet exposed in the UI. To configure multiple targets in the same tier or to enable sticky routing, apply the virtual model manifest with tfy apply or the API instead. Both are fully supported by the AI Gateway.
The common starting point: cheap model for the easy majority, mid-tier for everyday work, top model for the hard tail. No classifier cost, no extra latency.
LLM classifier with a conservative static fallback
Use a small model to classify, and route everything to the complex tier if that classifier is ever unavailable — quality never regresses, spend rises only during the outage.
Per-tier resilience with retries and a second complex target
Multiple targets in the same tier are attempted in declaration order before escalating. This shape requires the YAML manifest or API — the dashboard form allows one target per tier.
Every response carries an x-tfy-applied-rules header. For complexity-based routing it includes a complexity block with the tier that was served and how the tier was decided, plus the full ordered chain in routing_model_order:
Every routing decision increments a counter, so you can see your tier mix and how often escalation happens.
Metric
Type
Labels
ai_gateway_complexity_routing_decisions_total
Counter
tenant_name, model_name, decided_tier, resolved_tier, cause
decided_tier — the tier the request classified into.
resolved_tier — the tier of the first target actually attempted.
When the two differ, the request was escalated because the classified tier had no eligible target. A persistent gap is a signal that a tier is misconfigured or missing. See Prometheus and Grafana integration for scraping setup.
Each LLM classifier call produces a ComplexityBasedRouting: LLMClassifier span, with the classifier’s own chat-completion span nested beneath it — so classifier latency is visible separately from the served model’s latency.
The request span carries tfy.complexity.tier and tfy.complexity.cause.
The gateway logs each decision with the tier, the cause, the signals that matched, and the resolved model — useful for checking why a given request landed on the tier it did.
These configurations are rejected when you save the virtual model, with a 400:
Error
Fix
cannot use complexity-based routing with model_types [embedding]. Complexity-based routing supports only [chat, completion, responses]
Remove the unsupported model types, or serve them from a separate virtual model. Mixed lists are rejected too — declaring both chat and embedding fails.
Duplicate target across tiers
Assign the model to a single tier.
classifier model "…" could not be resolved in this tenant
Use a model FQN that exists in your model catalog.
classifier model "…" cannot be a virtual model
Point the classifier at a catalog model.
classifier model "…" must support [chat]
Pick a model whose model types include chat.
The three classifier checks also run at request time, so a classifier model that is later deleted or changed surfaces the same error on the request rather than misrouting silently.
That depends on your traffic mix, so measure it. Break ai_gateway_complexity_routing_decisions_total down by resolved_tier to see what share of traffic landed on each tier, and compare per-model cost in analytics. Your saving is that share multiplied by the price difference between tiers.
Can I use complexity-based routing in the tenant-level Routing Config YAML?
No. It is a virtual-model-only routing type. Create a virtual model and point your clients at it.
Can I customise how the heuristic classifies requests?
No, its signals are fixed. If the heuristic misclassifies your traffic, switch to the LLM classifier and nominate your own judging model.
Does the LLM classifier call show up in my usage and cost?
Yes. It appears in request logs, and its tokens and cost are attributed to the same tenant and subject as the request that triggered it. Use a small, cheap model and keep timeout_ms tight.
What happens if the classifier is down?
The request is never failed by a classifier problem. On timeout, error, non-2xx, or an unusable reply, the configured fallback_strategy decides the tier — the built-in heuristic by default, or a fixed default_tier if you chose static.
What if the tier a request classified into has no target?
It escalates to the next higher tier, and if there is none, down to lower tiers. The response header reports the tier actually used, and the metric records decided_tier and resolved_tier separately so you can spot it.
Why is a session jumping between models mid-conversation?
Each turn is classified independently unless you enable sticky routing — a follow-up that reads as harder than the opening message routes to a higher tier. To keep one conversation on one target, add sticky_routing with a session identifier your client always sends.
Do unhealthy-target cooldowns apply?
No. Targets are ordered by tier and escalation, never reordered by health. Per-target retries, fallback status codes, rate limits, and budgets all still apply.
Does it work with streaming?
Yes. Classification finishes before the upstream request is made, so streaming responses are unaffected.
Does it work with the Anthropic Messages API and the Responses API?
Yes. A chat virtual model serves both /chat/completions and /messages, and a responses virtual model serves /responses. In both cases the top-level system prompt (system and instructions respectively) is included as classification context.
