Classify each request as simple, medium, or complex and route it to the cheapest model that can handle it.
Requests to the same endpoint vary in difficulty. A greeting, a factual lookup, and a request to design a distributed rate limiter can all arrive at one model. If your most capable model serves all of them, you pay top-tier prices for requests a cheaper model could answer.Auto Routing classifies each request as simple, medium, or complex and sends it to the target you configured for that tier. Your application calls one virtual model name, and the AI Gateway picks the model that serves each request.Other routing strategies decide based on weights, priorities, or measured latency. Auto Routing decides based on the content of the request.
Auto Routing is generally available on all tenants. You do not need to request access. New virtual models still default to weight-based routing, so select Auto Routing when you create or edit one.
Auto Routing is available on virtual models only. The deprecated tenant-level Routing Config YAML (gateway-load-balancing-config) does not support it.
The AI Gateway rejects the virtual model on save unless it meets these conditions:
Model types are limited to chat, completion, and responses. Embedding, image, audio, rerank, and moderation are not supported.
Each model appears only once. A model can be the target of only one tier, and only once within it.
The classifier model is a catalog chat model for LLM classification, or a catalog decision model in a TypeSafe account for decision model classification. A virtual model cannot be a classifier.
If any target defines metadata_match, the AI Gateway drops non-matching targets first. Classification and escalation consider only the remaining targets.
2
Check the conversation pin
For a multi-turn request, the AI Gateway looks up the tier and target that served the conversation’s previous turn. Conversation pinning is automatic and needs no configuration.
3
Classify the request
The heuristic classifier (default), decision model classifier, or LLM classifier assigns the current turn to simple, medium, or complex. If the conversation is pinned to a higher tier, it stays on that tier. A conversation pinned to complex skips classification.
4
Build the target chain
Targets in the selected tier come first. The selected tier is the classified tier, or the higher pinned tier. The escalation chain follows: higher tiers, then lower tiers.
5
Send the request
The AI Gateway tries the first target with that target’s retry settings. If it fails with a fallback status code, the AI Gateway moves to the next target in the chain.
6
Update the conversation pin
After a successful response, the AI Gateway records the tier of the target that served the turn. Classification never moves a conversation down a tier. If a fallback was served by a lower-tier target, the pin records that lower tier.
Multi-step summaries, drafting, standard code, light reasoning
A mid-tier model
complex
Deep reasoning, long context, hard problem-solving, agentic chains
Your most capable model
The dashboard shows the same descriptions next to each tier.
Configure at least one model in each of simple, medium, and complex. The dashboard requires it. Configs applied through the API or tfy apply are not checked, and a request whose tier has no target goes to the next tier up, then down.
In the benchmark, the heuristic saved 78 to 95% of cost and the decision model classifier (Jev) saved 30 to 75%, depending on the workload. Jev passed more medium and hard prompts.
All three strategies read the same two pieces of text from the request body:
The last user message: the current turn.
The last system or developer message: used as context. This includes the top-level system field on Anthropic /messages requests and the top-level instructions field on /responses requests.
The classifier ignores earlier turns, assistant replies, and tool messages. Conversation pinning keeps multi-turn conversations consistent.
Classification reads up to 8,000 characters of user text and 2,000 characters of system text. Longer prompts still count as long prompts.
The heuristic classifier checks the prompt for a fixed set of signals and picks a tier. It needs no configuration and applies when classification_strategy is not set.
routing_config: type: complexity-based-routing classification_strategy: type: heuristic # default; this block can be omitted
Signals the heuristic uses
These signals push a request toward a higher tier, from most to least influential:
Signal
What it matches
Code
Programming keywords in the user or system text, such as function, class, api, docker, sql, and language names.
Explicit reasoning requests
Phrases in the user text that ask for reasoning, such as step by step, think through, pros and cons, compare and contrast, and explain your reasoning.
Technical vocabulary
Systems and ML terms, such as architecture, distributed, neural network, concurrency, and throughput.
Prompt length
Long user prompts.
Multi-step structure
Cues such as first … then, step 2, and numbered or lettered lists.
Many questions
More than three question marks in one request.
These signals push a request toward a lower tier:
Signal
What it matches
Lookup and small-talk phrasing
what is, who is, how many, define, hello, thanks, and similar.
Short prompts
Very short user prompts.
Reasoning override. Two or more distinct reasoning phrases in the user text always route to complex, regardless of other signals.Empty prompts. A request with no classifiable text routes to simple.The signals and keyword lists are fixed. If they do not reflect how difficulty shows up in your traffic, use the decision model classifier.
Examples
Prompt
Signals matched
Tier
Hi, thanks!
short prompt, small talk
simple
What is the capital of France?
short prompt, lookup phrasing
simple
Write a Python function that reverses a singly linked list in place and returns the new head.
code
medium
Design a distributed rate limiter. Walk me through the trade-offs step by step and explain your reasoning.
two reasoning phrases (override)
complex
Our gRPC service has high tail latency under concurrency. Analyze this and break down the likely causes.
two reasoning phrases (override), technical vocabulary
The AI Gateway classifies the request by calling a TypeSafe decision model, such as Jev, with a built-in tier rubric. In the dashboard, this is Decision Model. To set it up, add a native TypeSafe model account and a model with Model Types set to Decision. Reference the model by its catalog FQN, such as my-typesafe-account/jev.
The AI Gateway calls /v1/systemone with the user text in state.request and the system context, if any, in state.context. It sends one choice question named tier with the options simple, medium, and complex, and reads the tier from answers.tier.choice. You do not write question JSON or classifier instructions. The tier rubric is fixed.timeout_ms must be a positive integer. On a timeout, error, non-2xx response, or unusable tier, the configured fallback strategy applies. Conversation pinning and escalation work the same as with the other classifiers.
The classifier must belong to a TypeSafe account and support decision. A decision model such as Jev, accessed through OpenRouter or Custom Endpoints, cannot be the classifier. The virtual model that serves your application still declares chat, completion, or responses. decision applies only to the classifier model.
Each decision model classifier call is a billable gateway request, with its own tokens, cost, and request log under the caller’s tenant and identity. Set the TypeSafe model’s cost to track this usage. Turns pinned to complex skip the classifier call.
The AI Gateway classifies the request by calling a chat model you choose. In the dashboard, this is LLM Based, and the optional Classifier Prompt field sets instructions. LLM Based is listed only if a virtual model in your tenant already uses it.
routing_config: type: complexity-based-routing classification_strategy: type: llm-classifier model: openai-main/gpt-4o-mini # must be a catalog chat model timeout_ms: 2000 # default: 2000 fallback_strategy: # required type: heuristic # instructions: "..." # optional; replaces the built-in classifier rubric
fallback_strategy is required when classification_strategy.type is llm-classifier. The dashboard pre-fills the heuristic fallback, so this affects only configs written by hand or sent through the API. A config without it is rejected with must have required property 'fallback_strategy'.{"type": "heuristic"} matches the previous implicit behavior. Add it to existing configs to keep them working. classification_strategy itself is still optional and defaults to the heuristic strategy.
Each classifier call is a billable gateway request. It appears in your request logs, and its tokens and cost are attributed to the same tenant and subject as the original request. Use a small, fast model and a short timeout_ms.
How the classifier call works
The classifier receives the request’s system context and user text, plus a short instruction block that describes the three tiers (or your instructions override).
The reply is constrained to a JSON schema ({"tier": "simple" | "medium" | "complex"}) and capped at 256 output tokens. If a model ignores the schema and answers in prose, the AI Gateway uses the first tier name it finds.
After timeout_ms (default 2000), the call is abandoned and the fallback strategy applies.
If the classifier times out, errors, returns a non-2xx status, or returns nothing usable, the configured fallback strategy picks the tier and the request continues. Invalid classifier configuration, such as an unresolved model or the wrong provider account, is a separate case. See Validation errors.
classification_strategy: type: decision-classifier model: my-typesafe-account/jev fallback_strategy: type: heuristic # classify with the built-in heuristic
classification_strategy: type: decision-classifier model: my-typesafe-account/jev fallback_strategy: type: static default_tier: medium # required; always use this tier
llm-classifier uses the same fallback_strategy options. The field is required for both model-based strategies, and the dashboard pre-fills Heuristic Fallback. For static, default_tier is required, and the dashboard pre-fills medium.Use heuristic to keep classifying by content during a classifier outage. Use static to send all requests to one tier: default_tier: complex keeps quality at higher cost, and default_tier: simple keeps cost low at lower quality.
Auto Routing builds an ordered chain of targets, so a request is still served when its tier has no usable target.For a request classified into tier T, the order is:
Targets in tier T, in the order you declared them.
Targets in higher tiers, one tier at a time.
Targets in lower tiers, one tier at a time.
A request classified as simple tries simple, then medium, then complex. A request classified as complex tries complex, then medium, then simple. The request fails only after every target fails.Each target uses its own retry_config, fallback_status_codes, and fallback_candidate settings, the same as in the other routing strategies.
Unhealthy-target cooldown does not apply. Other strategies move a failing target to the end of the list. Auto Routing always orders targets by tier. Per-target rate limits and budgets still apply, and a rate-limited or over-budget target is skipped.
Auto Routing keeps a multi-turn conversation on the same tier and target. You do not need to configure sticky_routing, add a conversation header, or set a TTL.
The first turn is classified as usual. After a successful response, the AI Gateway records the tier and target.
Each later user turn is classified again.
If the new turn is harder, the conversation moves up to the higher tier. The pin updates after a successful response.
If the new turn is easier, the conversation stays on its current tier. A short follow-up such as “thanks” does not move it to a cheaper model.
Once a conversation reaches complex, later turns skip classification.
Classification can move a conversation to a more capable model when the task gets harder, but never to a less capable one. The only way down is a fallback: if a lower-tier target serves a turn, the pin records that tier.Your application needs no changes. Send the conversation history as the API already expects. A pin expires after 10 minutes of inactivity and resets on every turn.
Single-turn requests, such as legacy /completions and /responses with a string input, have no conversation, so each one is classified on its own.
A target that failed is never recorded as the conversation’s next target.
We ran the heuristic and the decision model classifier with Jev on the same prompts and compared each one with sending every request to Claude Opus 5.5.
Setup
Tiers
simple: Claude Haiku 5.5, medium: Claude Sonnet 5.5, complex: Claude Opus 5.5
Baseline
Claude Opus 5.5 on every request. Haiku 5.5 and Sonnet 5.5 on every request are shown for comparison.
Classifiers
Heuristic (default) and Decision Model with Jev (jev-1.13.0), shown as Jev below. Jev used a 3,000 ms timeout with heuristic fallback. No request fell back.
Prompts
1,279 across three workloads: 550 graded, 570 RouterArena, 159 production-style traffic
Cost
Tokens each request used, at the list price of the model that served it
Pass rate
Share of prompts answered correctly. Compare each row with Opus 5.5 on every request.
Latency
Time to the full, non-streamed response, measured at the client. Includes the classifier call. p50 is the median and p95 is the slowest 5% of requests.
Date
October 2026
Answers are graded automatically. Generated code runs against unit tests, and math and multiple-choice answers are checked against answer keys.
Cost saved on each workload. Pass rates are in the tables below.
550 prompts from 11 datasets: ARC-Challenge, BBH, CommonsenseQA, GSM8K, HumanEval, MATH-500, MBPP, MMLU, MMLU-Pro, OpenBookQA, and SVAMP.
Classifier
Pass rate
Cost saved
Latency p50 / p95
Tier mix (simple / medium / complex)
Opus 5.5 on every request
97.3%
0%
3.0s / 7.3s
0 / 0 / 100%
Sonnet 5.5 on every request
97.5%
59.6%
2.6s / 4.3s
0 / 100 / 0%
Haiku 5.5 on every request
95.8%
96.6%
2.2s / 5.9s
100 / 0 / 0%
Heuristic
96.0%
89.8%
2.1s / 5.6s
81 / 19 / 0%
Jev
97.3%
57.4%
2.3s / 7.7s
41 / 50 / 9%
Results by dataset
50 prompts per dataset.
Dataset
Opus 5.5 pass rate
Heuristic pass rate / cost saved
Jev pass rate / cost saved
ARC-Challenge
100%
100% / 97%
100% / 97%
BBH
100%
94% / 95%
98% / 47%
CommonsenseQA
96%
98% / 97%
100% / 97%
GSM8K
98%
96% / 97%
98% / 62%
HumanEval
100%
100% / 55%
100% / 49%
MATH-500
98%
96% / 97%
98% / 26%
MBPP
100%
100% / 46%
100% / 37%
MMLU
94%
90% / 97%
90% / 80%
MMLU-Pro
84%
86% / 95%
88% / 59%
OpenBookQA
100%
98% / 96%
98% / 96%
SVAMP
100%
98% / 97%
100% / 56%
570 prompts from the RouterArena routing benchmark, labeled easy, medium, and hard.
Classifier
Pass rate
Cost saved
Latency p50 / p95
Tier mix (simple / medium / complex)
Opus 5.5 on every request
90.4%
0%
3.9s / 9.1s
0 / 0 / 100%
Sonnet 5.5 on every request
88.2%
57.8%
3.0s / 6.0s
0 / 100 / 0%
Haiku 5.5 on every request
83.9%
96.3%
2.8s / 8.0s
100 / 0 / 0%
Heuristic
85.4%
94.7%
2.8s / 8.7s
96 / 4 / 0%
Jev
86.8%
74.6%
3.0s / 7.9s
74 / 22 / 4%
159 real-world prompts: developer chat (DevGPT), coding agent tasks (SWE-bench Lite), and open-ended user chat (WildChat). These prompts have no answer key, so only cost and latency are measured.
Classifier
Cost saved
Latency p50 / p95
Tier mix (simple / medium / complex)
Opus 5.5 on every request
0%
13.7s / 45.3s
0 / 0 / 100%
Heuristic
78.4%
8.2s / 25.4s
65 / 33 / 3%
Jev
29.9%
10.5s / 38.6s
20 / 32 / 48%
Agent and coding prompts are harder than benchmark questions, so Jev sent close to half of them to Opus 5.5.
The two classifiers differ most as prompts get harder. On RouterArena’s easy prompts, both pass about as often as Opus 5.5. On medium and hard prompts, the heuristic still sends almost everything to Haiku 5.5, while Jev sends more of them to Sonnet and Opus. On medium prompts, Jev passes 84% and the heuristic 79%, against 88% for Opus 5.5. Hard prompts are hard for every model: Opus 5.5 passes 57%, Jev 35%, and the heuristic 29%. The hard set has only 49 prompts, so treat the gap there as a direction, not a precise figure.
RouterArena pass rate by difficulty label, with Opus 5.5 on every request for reference.
Saved 78 to 95% of cost with the lowest latency and no classifier call.
Math, code, or other tasks where accuracy matters
Decision model (Jev)
Passed within 4 points of Opus 5.5 (97% graded, 87% RouterArena), and sent more medium and hard prompts to Sonnet and Opus. Sonnet 5.5 on every request passed 88 to 98% at 58 to 60% lower cost, so compare both.
Coding agents and long, open-ended tasks
Decision model (Jev)
Sent close to half of these prompts to Opus 5.5, so savings were lower (30%). The heuristic saved 78% but sent 65% of them to Haiku.
Mostly hard reasoning
Your complex tier model directly
On hard prompts, both classifiers passed 29 to 35%, against 57% for Opus 5.5.
If you are not sure, start with the heuristic (the default). Before you roll out, compare it with a single cheaper model on your own traffic.To track savings on your own traffic, see Cost Savings and Observability.
You configure Auto Routing on a virtual model. See Create a virtual model for the full setup.
1
Create the virtual model and choose Auto Routing
Go to AI Gateway → Model Gateway → Virtual Models. Click Add Virtual Model Group, or open an existing group, and add or edit a model. Enter a Name, and under Model Types select only Chat, Completion, or Responses.For the routing strategy, select Auto Routing.
Creating a virtual model with Auto Routing selected
2
Pick a classification strategy
Heuristic is selected by default and needs no other settings.To classify with a decision model, choose Decision Model, select your TypeSafe Classifier Model, such as Jev, and set Timeout (ms) (default 800) and Fallback Strategy. Choosing Static Fallback adds a Default Tier field. If the model is not listed, add it through the TypeSafe integration first.
Decision Model classifier set to TypeSafe Jev, with an 800 ms timeout, heuristic fallback, and a target for each complexity tier
LLM Based appears only if a virtual model in your tenant already uses it. Otherwise, set llm-classifier through the API or tfy apply. See LLM classification.
3
Add targets and assign tiers
Targets form a numbered list, each with a Tier dropdown. The first three are fixed to Simple, Medium, and Complex. Click + Add Target to add more targets to any tier.Each target has its own Fallback status codes, plus optional Retries, Override Params, Override Headers, and Metadata Filters. These work the same as in the other routing strategies.
A target for each tier, plus a second Complex target added with + Add Target
routing_config: type: complexity-based-routing # How requests are classified. Optional; defaults to { type: heuristic }. classification_strategy: type: heuristic | llm-classifier | decision-classifier # llm-classifier and decision-classifier: model: string # catalog FQN; chat for LLM, TypeSafe decision model for decision-classifier timeout_ms: integer # positive; default: 2000 (LLM), 800 (decision model) fallback_strategy: # required for both model-based strategies type: heuristic | static default_tier: simple | medium | complex # static only; required when type is static instructions: string # llm-classifier only; optional rubric override load_balance_targets: - target: string # catalog model FQN, e.g. openai-main/gpt-4o tier: simple | medium | complex # required; a model can appear in only one tier retry_config: attempts: integer # retries on the SAME target; required when retry_config is set delay: integer # ms between retries; default: 100 on_status_codes: string[] # default: ["429","500","502","503"] fallback_status_codes: string[] # codes that move to the NEXT target in the chain # default: ["401","403","404","408","429","500","502","503"] fallback_candidate: boolean # eligible to receive escalated traffic; default: true metadata_match: # all pairs must match resolved request metadata key: value headers_override: set: header-name: header-value remove: - header-name override_params: temperature: number max_tokens: integer prompt_version_fqn: string
Decision model classifier with a static fallback to complex
A decision model such as Jev classifies requests. If the classifier is unavailable, all requests go to the complex tier, so quality holds and cost rises only during the outage.
Every response includes an x-tfy-applied-rules header. For Auto Routing, it contains a complexity block with the served tier and how it was chosen, and the full target chain in routing_model_order:
The heuristic classifier chose the tier, either as the configured strategy or as the fallback after an LLM or decision model classifier failure.
llm_classifier
The LLM classifier chose the tier.
decision_classifier
The decision model classifier chose the tier.
session_pin
The conversation pin set the tier: the pin was higher than the current turn’s classification, or the same. Classification is skipped only when the pin is complex.
default
The LLM or decision model classifier failed and the static fallback applied its default_tier.
reason shows why each target in routing_model_order is in its position:
reason
Meaning
complexity-tier
The target is in the tier the request was classified into.
complexity-escalation
The target is in a different tier and is in the chain for failover.
complexity-session-pin
The target is first because the conversation pin selected it.
The x-tfy-resolved-modelresponse header also returns the model that served the request.The Cost savings column in Metrics Overview estimates how much Auto Routing saved compared with sending every request to the complex tier. See Cost Savings.
Each routing decision increments a counter that shows your tier mix and how often requests escalate.
Metric
Type
Labels
ai_gateway_complexity_routing_decisions_total
Counter
tenant_name, model_name, decided_tier, resolved_tier, cause
decided_tier: the tier assigned to the current turn, before target fallback. For a complex pin that skips classification, this is complex.
resolved_tier: the tier after the conversation pin is applied. It differs from decided_tier only when cause is session_pin.
Escalation does not change resolved_tier or the complexity.tier value in the response header. When a target is rate-limited, over budget, or fails and the request moves to another tier, only resolved_model in the header and x-tfy-resolved-model show the model that served it. See Prometheus and Grafana integration for scraping setup.
Each LLM classifier call creates a ComplexityBasedRouting: LLMClassifier span, with the classifier’s chat completion span nested under it. This separates classifier latency from the served model’s latency.
Each decision model classifier call creates a ComplexityBasedRouting: DecisionClassifier span, with the System One request nested under it. The span records tfy.complexity.tier and, when returned, tfy.complexity.confidence.
The request span includes tfy.complexity.tier, tfy.complexity.cause, and, for heuristic decisions, tfy.complexity.score.
The AI Gateway logs each decision with the tier, cause, matched signals, and resolved model. Use these logs to see why a request landed on its tier.
The AI Gateway rejects these configurations with a 400 when you save the virtual model:
Error
Fix
Virtual model <name> cannot use complexity-based routing with model_types [embedding]. Complexity-based routing supports only [chat, completion, responses].
Remove the unsupported model types, or serve them from a separate virtual model. Mixed lists, such as chat and embedding, are also rejected.
Each target model may serve only one tier in virtual model <name>, but these appear under more than one tier: <models>.
Assign each model to one tier, once.
classifier model "…" could not be resolved in this tenant
Use a model FQN from your model catalog.
classifier model "…" cannot be a virtual model
Use a catalog model as the classifier.
classifier model "…" must support [chat]
For LLM classification, pick a model whose model types include chat.
classifier model "…" must belong to a typesafe provider account
For decision model classification, pick a model from a native TypeSafe account.
classifier model "…" must support [decision]
For decision model classification, pick a TypeSafe model whose model types include decision.
The AI Gateway also checks the classifier model at request time. If the classifier model is deleted or changed after you save, requests return the same error instead of routing to the wrong tier.
In our benchmark, savings ranged from 30% to 95% depending on the classifier and workload. Your savings depend on your traffic mix. The Cost savings column in Metrics Overview estimates savings against the complex tier. See Cost Savings. To see the share of traffic on each tier, break down ai_gateway_complexity_routing_decisions_total by resolved_tier, and compare per-model cost in Model Metrics.
Can I use Auto Routing in the tenant-level Routing Config YAML?
No. Auto Routing is available on virtual models only. Create a virtual model and point your clients at it.
Can I customize how the heuristic classifies requests?
No. Its signals are fixed. If the heuristic misclassifies your traffic, switch to the decision model classifier.
Do LLM and decision model classifier calls count toward usage and cost?
Yes. Classifier calls appear in request logs, and their tokens and cost are attributed to the same tenant and subject as the original request. Use a small, low-cost model and a short timeout_ms. Conversations pinned to complex make no further classifier calls.
What happens if the classifier is down?
The request still goes through. On a timeout, error, non-2xx response, or unusable reply, the configured fallback_strategy picks the tier: the heuristic, or a fixed default_tier if you chose static.
Can I configure only two tiers, such as simple and complex?
Not in the dashboard, which requires a target in each of simple, medium, and complex. Configs applied through the API or tfy apply are not checked. A request classified into a tier with no target goes to the next tier up, then down. Give every tier a target so cost and quality stay predictable.
What if the classified tier has no usable target?
If all targets in the classified tier are ineligible at request time, for example rate-limited, over budget, or filtered out by metadata, the request escalates to the next higher tier, then to lower tiers. resolved_model in the x-tfy-applied-rules header and x-tfy-resolved-model show the model that served it. The tier fields and metric labels still show the classified tier.
Can a conversation switch models partway through?
Yes, when it needs a more capable model or the current target fails. Conversation pinning stops later turns from moving to a lower tier. A harder follow-up can move the conversation to a higher tier, and a successful fallback can change the target for later turns.
Do unhealthy-target cooldowns apply?
No. Targets are ordered by tier and escalation, not by health. Per-target retries, fallback status codes, rate limits, and budgets still apply.
Does it work with streaming?
Yes. Classification finishes before the upstream request starts, so streaming works as usual.
Does it work with the Anthropic Messages API and the Responses API?
Yes. A chat virtual model serves /chat/completions, /messages, and /responses, and a responses virtual model serves /responses. The top-level system prompt (system or instructions) is used as classification context, and multi-turn conversations are pinned the same way. A /responses follow-up with previous_response_id goes to the backend that served the original response, but the classifier still runs and is billed.
