Introducing Auto Routing on the TrueFoundry AI Gateway

We’ve just added a new capability to the TrueFoundry AI Gateway. The new Auto-Routing feature reads each incoming request, sorts it into one of three complexity tiers, and sends it to the most appropriate model assigned to that tier.

We built it to solve a recurring pattern we see in customer traffic. Typically, most queries are straight-forward and can be handled by simple models and only a few actual complex queries that need to be routed to a more complex model. What ends up happening, however, is that one model is uniformly assigned to answer each query, regardless of the complexity. As a result, companies are faced with an unnecessarily massive bill at the end of each cycle. Hard-coding routing rules is not a solution that scales, as each team eventually applies their own routing logic with no standardization.

That work now belongs to the gateway.

Moving from tokenmaxxing to tokenomics

For most of the last two years, teams measured AI adoption by consumption. Tokens burned became a stand-in for work done. Some companies put up internal leaderboards to celebrate whoever used the most. This may have made sense in the nascent stage of AI adoption, when the goal was to understand how much work could actually be offloaded to models. 

Then the invoices arrived. In one IDC survey of cloud and AI decision-makers, 61% of organizations blew past their 2025 cloud AI budget. Uber's CTO disclosed that his team had spent its entire annual AI budget by mid-April. Reasoning loops got longer, context windows got fuller, and agents started calling models on their own behalf thousands of times a day with nobody watching.

So priorities shifted. Tokenmaxxing has been replaced by tokenomics, a strategy where companies try to cap AI spends without slowing down adoption or experimentation. This means there’s nmore focus on real-time cost observability, chargeback models, per-team attribution, and usage caps. All of these are important features, but they only help monitor or restrict spend, not optimize it.

The decision that actually determines your bill happens much earlier and much faster: which model gets this request. That is the layer Auto Routing operates in.

What Auto Routing actually does

Three tiers behind a single model name

Auto Routing is configured on a virtual model, which is one named entry your application calls. There are three complexity tiers to which you can assign specific models.

Because the tier assignments live on the virtual model, you can swap the model behind a tier whenever a better or cheaper option ships, and no application code changes. The routing policy becomes a configuration object with an owner, a change history, and a single place to edit it.

How is a complexity tier selected?

There are two methods in which each query gets categorized into one of the three tiers:

1. Heuristic classification (the default)

The heuristic classifier reads the prompt, scores it against a fixed set of signals, and picks a tier. It runs inside the gateway process, so it adds no latency and no cost, and it is fully deterministic. 

It determines the complexity of a query based on certain parameters:

• Code, meaning programming keywords in the user or system text such as function, class, api, docker, sql, and language names
• Explicit reasoning requests like "step by step", "think through", "pros and cons", "explain your reasoning"
• Technical vocabulary such as architecture, distributed, concurrency, throughput
• Prompt length, since long prompts tend to carry more work
• Multi-step structure, such as "first, then" phrasing or numbered lists
• Many questions at once, meaning more than three question marks in a single request

2. LLM classification

Here, you choose the classifier model, which must be a catalog chat model rather than another virtual model, and you set a timeout in milliseconds. You must also configure what happens if that call fails. Two fallback options are available: heuristic, which hands the decision back to the built-in signals, or static, which sends everything to a tier you name. A classifier failure never fails the request.

Two tradeoffs come with this. The classifier runs as a model call ahead of the request, so it adds latency. And it is a billable gateway request, attributed to the same tenant and subject as the request that triggered it. 

Stand-out capabilities of Auto Routing

Auto Routing goes further than simple routing rules, giving you additional capabilities to ensure model spends are optimized, while retaining accuracy:

• Escalation and fallback: Auto Routing builds a fully ordered chain where a request classified into a tier tries that tier's targets first, in the order you declared them, then works upward through higher tiers. 
• Conversation pinning: The first turn is classified normally, and once a target answers successfully, the gateway records its tier. Later turns are classified again. If a turn is harder, the conversation moves up. If a turn is easier, the existing higher tier wins.
• Observability: ai_gateway_complexity_routing_decisions_total counts every decision, labeled by both the tier that was decided and the tier that was actually resolved. You can show finance the tier mix, show engineering the escalation rate, and point at a specific request to explain why it landed where it did.

What the benchmarks show

We ran Auto Routing against a baseline that sends every request to Claude Opus 5, across 16 public datasets and roughly 4,700 calls, using the free heuristic classifier and a Haiku, Sonnet, Opus ladder. Grading is deterministic: generated code runs against each benchmark's unit tests in a sandbox, and math and multiple-choice answers are matched to keys.

In each case, there were significant cost savings, with no compromise on quality.

For the full methodology, the per-dataset breakdown, and the production traffic analysis, read the engineering deep dive: How we cut LLM cost by two-thirds without losing quality.

Getting started

Auto Routing lives on a virtual model. Go to AI Gateway → Models → Virtual Model, choose Complexity under routing type, pick a classification strategy, and set a target for each tier you want to serve. Then point your application at the virtual model's name and test it out with prompts of varying complexity in the playground.