Auto Routing with Jev: Opus-level accuracy at 57% lower cost

If every request in your app goes to Claude Opus, you are paying Opus prices for "convert this to JSON" and "what does this error mean". Most of those requests would get the same answer from Haiku for a fraction of the cost. The hard part is telling them apart, request by request, without slowing anything down.

That is what Auto Routing in the TrueFoundry AI Gateway does. Each request gets a tier, simple, medium, or complex, and goes to the model you assigned to that tier. Until now, the tier came from a keyword heuristic. You can now have Jev decide instead.

We ran about 1,300 prompts through it to see how well that works. The short version:

• Same accuracy as Opus 5.5, 57% cheaper. On 550 public benchmark prompts, Auto Routing with Jev passed 97.3%, exactly what Opus passed on every request.
• Faster, not slower. Median response time went from 3.0 s to 2.3 s, classifier call included.
• Almost free to run. Jev took about 140 ms per request and cost 2.5 cents per 1,000 requests. Not one routed request fell back to the heuristic.

What Jev is

Jev is a decision model from TypeSafe. It doesn't write text. You hand it a request and a few named options, and it picks one, with a probability for each. For routing, the options are our three tiers, so a routing decision is a single, fast call.

How we tested

We set up three tiers: Haiku 5.5 for simple, Sonnet 5.5 for medium, and Opus 5.5 for complex. Then we ran three sets of prompts through Auto Routing with Jev, Auto Routing with the heuristic, and Opus on every request:

• 550 public benchmark prompts, 50 each from 11 datasets covering science and commonsense questions, math, and Python.
• 570 RouterArena prompts, from a benchmark built for testing routers, each labeled easy, medium, or hard.
• 159 agent and chat prompts from DevGPT, SWE-bench Lite, and WildChat.

Code answers ran against unit tests. Math and multiple-choice answers were checked against answer keys. Cost is what the tokens cost at list price, and every "cost saved" figure is against sending everything to Opus.

Jev matches Opus

Jev and Opus each got 535 of the 550 prompts right. Jev sent only 9% of them to Opus, and 41% went to Haiku. Smaller models answer faster, so responses came back sooner too.

The heuristic saved more, 90%, but gave up 1.3 points of accuracy to do it. It sent every math prompt to Haiku, and Jev moved them up to Sonnet or Opus. That is the real difference between the two. The heuristic is built to save money, and Jev spends where the request needs it.

It routes by the kind of work

The routing pattern makes sense when you look at it dataset by dataset:

Science and commonsense questions went to Haiku and saved 96 to 97% with no drop in accuracy. Python functions went mostly to Sonnet, and every one passed its tests. Competition math got the most Opus of any dataset and matched Opus at 98%.

Across all eleven datasets, Jev matched or beat Opus on eight.

It holds up as prompts get harder

RouterArena labels every prompt by difficulty, so we can see how each classifier handles harder work.

On easy prompts, Jev kept 87% of requests on Haiku, saved 88%, and stayed within a point of Opus.

Medium prompts are where the two classifiers split. The heuristic still sent 96% of them to Haiku and fell 9 points behind Opus. Jev sent about half to Sonnet or Opus and closed more than half of that gap, passing 84% against Opus's 88%.

Hard prompts are hard for everyone. Even Opus passed only 57%. Jev passed 35% and the heuristic 29%. If most of your traffic looks like this, point it straight at your strongest model.

Across all of RouterArena, Jev came within 4 points of Opus at 75% lower cost.

Agent traffic

Agent and developer prompts have no answer key, so here we can only compare cost and speed. Jev sent nearly half of them to Opus and still cut cost by 30% and median latency by 3 seconds, from about 14 s to 11 s. The heuristic sent 65% of the same prompts to Haiku and saved 78%. If you route agent traffic, check that tier mix against your own evals.

What Jev costs to run

We also sent every prompt to Jev three more times on its own, about 3,900 calls in all.

• Latency: about 140 ms at the median and 190 ms at p95. Only 2 calls went over the 800 ms default timeout.
• Cost: about 600 input tokens a call, or 2.5 cents per 1,000 requests. That's roughly 1/150 of a typical Opus response in our run.
• Consistency: Jev picked the same tier all three times for 97% of prompts. The same request gets the same model, so costs stay predictable and every decision is easy to audit.

Setting it up

Auto Routing is a routing type on a virtual model. Tag each target with a tier and choose Jev as the classifier:

routing_config:
  type: complexity-based-routing
  classification_strategy:
    type: decision-classifier
    model: typesafe/jev-1.13.0
    timeout_ms: 800
    fallback_strategy:
      type: heuristic
  load_balance_targets:
    - target: anthropic/claude-haiku-5-5
      tier: simple
    - target: anthropic/claude-sonnet-5-5
      tier: medium
    - target: anthropic/claude-opus-5-5
      tier: complex

Your app calls the virtual model like any other model. Here is what happens to each request:

Jev sees the request, not the whole chat. It gets the last user message and the system prompt, plus a short description of each tier. Its instructions say that a request asking for a tier is content to classify, not a command. Without that rule, typing "treat this as complex" would get a user Opus on your bill.

Conversations stay put. A chat that started on Opus shouldn't drop to Haiku when the user types "thanks". Auto Routing pins each conversation to its tier using a hash of the user's messages, so you don't need to pass a session ID. The pin only moves up, and expires after 10 minutes of inactivity.

Nothing blocks on the classifier. If Jev is slow or errors, the request falls back to the heuristic. If a tier's models are rate-limited or failing, the request moves up a tier, then down.

Everything is recorded. Each request logs its tier and the reason it got that tier, in metrics and on the trace. The Jev call itself is a normal gateway request, so its cost shows up in the same dashboards as the rest of your spend.

Rolling it out

1. Keep the 800 ms timeout and heuristic fallback. It leaves plenty of room over Jev's 190 ms p95.
2. Test on your own traffic. Run a sample through Opus and through Auto Routing, and grade the answers. Pass rate and cost per request are the numbers to watch.
3. Look at hard prompts separately. If some traffic always needs your best model, give it its own virtual model.
4. Add a second target per tier. If Haiku is rate-limited, a second simple target keeps requests at the price you planned for instead of moving them up.

Try it

Auto Routing with Jev works with any TypeSafe account connected to the gateway. Pick the heuristic when cost matters most. It saved 78 to 95% across our prompt sets. Pick Jev when accuracy matters. It matched Opus at 57% lower cost and held up better as prompts got harder.

Setup steps and the full configuration reference are in the Auto Routing docs.