> ## Content Index > Fetch the complete content index at: https://www.truefoundry.com/blog/engineering/llms.txt > Use this file to discover other available public pages before exploring further. # How we cut LLM cost by two-thirds without losing quality - URL: https://www.truefoundry.com/blog/engineering/llm-cost-routing-benchmark/ - Published: 2026-08-06T11:13:59.000Z - Updated: 2026-08-19T11:16:17.000Z - Author: Shubham Agarwal - Tags: LLM Cost, Benchmarks Your LLM endpoint sees a mix of work. One-line lookups, routine completions, and every so often a problem that needs a big model, all landing on the same API. Point all of it at your best model and you pay top rates for the easy requests, which are most of the traffic. TrueFoundry's **Auto Routing** sorts each request into simple, medium, or complex and sends it to the cheapest model that can handle it. Your app keeps calling one virtual-model name. We ran it against an all Opus 5 setup across **16 public datasets and about 4,700 calls**, graded deterministically. 69% Cheaper graded · $3.23 to $0.99 98% Quality retained deterministic grading 1.9x Faster 7.6s to 4.0s mean ## How it works **Auto Routing** lives on a virtual model, a single named entry your application calls. You set one target per tier behind that name, and you can swap those targets later without touching code. Every request is classified at the gateway, then handed to its tier. We used the default **heuristic classifier** for this benchmark. It scores the prompt on a few signals. Code and systems vocabulary, phrasing like "step by step", length, and multi-step structure push a request up; short lookups and small talk pull it down. If a tier has no healthy target the gateway escalates or falls back, so something always answers. It also reports the tier it picked and the model that replied, which is how we read routing straight from the response rather than inferring it. your appprod/smart-chat → gatewayclassify complexity → Haiku 4.5simple · $1 / $5 Sonnet 4.5medium · $3 / $15 Opus 5complex · $5 / $25 ## How we measured it Two setups, same prompts. The router arm is one virtual model with the heuristic classifier and a **Haiku, Sonnet, Opus ladder**. The baseline sends everything to **Opus 5**. Grading is deterministic. Generated code runs against each benchmark's unit tests in a sandbox, and math and multiple-choice answers are matched to keys. We measured it three ways: graded **academic benchmarks** for quality, **production-shaped traffic** for real cost and routing, and a **difficulty-labeled set** to test whether the router actually spends more on the harder questions. ## Quality holds while cost drops by two thirds We start on academic benchmarks, the standardised public datasets used to measure model quality, eleven of them across math, code, knowledge, and reasoning, at 50 prompts each. Every question has a known answer a machine can check, and because these tasks are uniformly hard, this is the conservative case for savings. GSM8K**79%** 100% MATH-500**78%** 100% SVAMP**76%** 98% BBH**75%** 98% CommonsenseQA**73%** 96% MMLU-Pro**71%** 91% OpenBookQA**68%** 96% MMLU**65%** 98% ARC-Challenge**60%** 100% MBPP**55%** 96% HumanEval**44%** 102% Bars: cost savings vs Opus 5\. Chip: quality retained. Gridline marks 50%. | Overall, 550 prompts | Baseline | Router | Result | | ----------------------- | -------- | ------- | ------------ | | Cost | $3.23 | $0.99 | 69% cheaper | | Quality (pass rate) | 96.4% | 94.2% | 98% kept | | Cost per correct answer | $0.0061 | $0.0019 | 3.2x cheaper | | Mean latency | 7.6s | 4.0s | 1.9x faster | ## Production traffic saves more To see production behavior, we ran four datasets of genuine prompts (consumer chat, developer chat, code-heavy chat, and long agent tasks from real GitHub issues), measured for cost and routing against the same Opus 5 baseline, since open-ended chat has no answer key to grade against. Real user chat**91%** Developer chat**88%** Agent tasks**75%** Code-heavy chat**69%** Cost savings vs Opus 5 on real traffic. Overall: 80% cheaper. Real traffic saved even more than the academic sets: **80%**. Most everyday requests are easy, so they route to the cheap tier, and Opus only showed up on the genuinely hard work, 4% of the agent tasks (real GitHub issues) and about 2% of chat. ### Where the free classifier slips, and the knob that fixes it So far the free heuristic looks great. One of our datasets shows where it does not. Every question in it is labeled easy, medium, or hard, so we can check whether routing follows real difficulty. It does not. The heuristic reads the shape of a prompt, not how hard it is, so a short but hard question looks easy and goes to the cheap tier. On easy questions that is free. On hard ones it costs real accuracy. easy**99%** kept medium**79%** kept hard**35%** kept Quality retained by difficulty, heuristic classifier. ## Two classifiers, two operating points The fix is a setting, not a rebuild. The LLM-classifier spends a small model call to judge how hard each prompt really is. It sends about a third of these questions up to Opus and wins most of the quality back, for roughly ten points less in savings. So every virtual model has two modes: the free **heuristic** for maximum savings, or the **LLM-classifier** when you would rather protect quality on hard prompts. free · in-process #### Heuristic 76% cheaper 90% quality 0% to Opus small classifier call #### LLM-classifier 66% cheaper 95% quality 31% to Opus Measured on the 570-question difficulty set. ## Cheaper and faster at once Routing does not cost you speed. It saves it. The classifier runs inside the gateway with no real overhead, and the cheaper models finish faster than Opus 5, so the router was quicker in every run. Graded benchmarks baseline7.6s router4.0s Router-evaluation set baseline8.1s router4.4s ## Try it on your own traffic Auto Routing is one setting on a virtual model: pick a model for each tier and point your app at the virtual model's name. Nothing else in your code changes. Your savings depend on your own traffic, so the real test is to switch it on and watch the routing and spend the gateway logs for you. [Read the docs](https://www.truefoundry.com/docs/ai-gateway/complexity-based-routing?ref=truefoundry.com)