How we cut LLM cost by two-thirds without losing quality
Your LLM endpoint sees a mix of work. One-line lookups, routine completions, and every so often a problem that needs a big model, all landing on the same API. Point all of it at your best model and you pay top rates for the easy requests, which are most of the traffic.
TrueFoundry's Auto Routing sorts each request into simple, medium, or complex and sends it to the cheapest model that can handle it. Your app keeps calling one virtual-model name. We ran it against an all Opus 5 setup across 16 public datasets and about 4,700 calls, graded deterministically.
How it works
Auto Routing lives on a virtual model, a single named entry your application calls. You set one target per tier behind that name, and you can swap those targets later without touching code. Every request is classified at the gateway, then handed to its tier.
We used the default heuristic classifier for this benchmark. It scores the prompt on a few signals. Code and systems vocabulary, phrasing like "step by step", length, and multi-step structure push a request up; short lookups and small talk pull it down. If a tier has no healthy target the gateway escalates or falls back, so something always answers. It also reports the tier it picked and the model that replied, which is how we read routing straight from the response rather than inferring it.
How we measured it
Two setups, same prompts. The router arm is one virtual model with the heuristic classifier and a Haiku, Sonnet, Opus ladder. The baseline sends everything to Opus 5.
Grading is deterministic. Generated code runs against each benchmark's unit tests in a sandbox, and math and multiple-choice answers are matched to keys. We measured it three ways: graded academic benchmarks for quality, production-shaped traffic for real cost and routing, and a difficulty-labeled set to test whether the router actually spends more on the harder questions.
Quality holds while cost drops by two thirds
We start on academic benchmarks, the standardised public datasets used to measure model quality, eleven of them across math, code, knowledge, and reasoning, at 50 prompts each. Every question has a known answer a machine can check, and because these tasks are uniformly hard, this is the conservative case for savings.
Bars: cost savings vs Opus 5. Chip: quality retained. Gridline marks 50%.
| Overall, 550 prompts | Baseline | Router | Result |
|---|---|---|---|
| Cost | $3.23 | $0.99 | 69% cheaper |
| Quality (pass rate) | 96.4% | 94.2% | 98% kept |
| Cost per correct answer | $0.0061 | $0.0019 | 3.2x cheaper |
| Mean latency | 7.6s | 4.0s | 1.9x faster |
Production traffic saves more
To see production behavior, we ran four datasets of genuine prompts (consumer chat, developer chat, code-heavy chat, and long agent tasks from real GitHub issues), measured for cost and routing against the same Opus 5 baseline, since open-ended chat has no answer key to grade against.
Cost savings vs Opus 5 on real traffic. Overall: 80% cheaper.
Real traffic saved even more than the academic sets: 80%. Most everyday requests are easy, so they route to the cheap tier, and Opus only showed up on the genuinely hard work, 4% of the agent tasks (real GitHub issues) and about 2% of chat.
Where the free classifier slips, and the knob that fixes it
So far the free heuristic looks great. One of our datasets shows where it does not. Every question in it is labeled easy, medium, or hard, so we can check whether routing follows real difficulty. It does not. The heuristic reads the shape of a prompt, not how hard it is, so a short but hard question looks easy and goes to the cheap tier. On easy questions that is free. On hard ones it costs real accuracy.
Quality retained by difficulty, heuristic classifier.
Two classifiers, two operating points
The fix is a setting, not a rebuild. The LLM-classifier spends a small model call to judge how hard each prompt really is. It sends about a third of these questions up to Opus and wins most of the quality back, for roughly ten points less in savings. So every virtual model has two modes: the free heuristic for maximum savings, or the LLM-classifier when you would rather protect quality on hard prompts.
Heuristic
LLM-classifier
Measured on the 570-question difficulty set.
Cheaper and faster at once
Routing does not cost you speed. It saves it. The classifier runs inside the gateway with no real overhead, and the cheaper models finish faster than Opus 5, so the router was quicker in every run.
Try it on your own traffic
Auto Routing is one setting on a virtual model: pick a model for each tier and point your app at the virtual model's name. Nothing else in your code changes. Your savings depend on your own traffic, so the real test is to switch it on and watch the routing and spend the gateway logs for you.