Skip to main content
The AI Gateway estimates how much you saved versus full-price inference. The Metrics Overview reports that total, then splits it into three independent sources: provider prompt caching, gateway semantic/exact caching, and Auto Routing.

​
Where cost savings appear

Open AI Gateway → AI Monitoring → Metrics → Overview. The usage table includes a Cost savings column next to Spend. Each row shows the dollar amount saved for that user, virtual account, or team, plus a stacked bar of the three sources. Hover the cell for the breakdown, the total saved, and the percent off full price.
The Cache Metrics tab also has a Total Cost Saved counter. That counter is gateway cache hits only (semantic and exact-match). The Overview Cost savings figure is the sum of all three sources below.

​
What the breakdown shows

The percent shown next to the total is saved / (spend + saved) — the share of what the same traffic would have cost at full price.

​
How each figure is calculated

In every formula below, A is the actual cost of the request: today’s tfy.model.metric.cost_in_usd.

​
Prompt caching

Provider prompt caching reuses a previously computed prompt prefix. Cached input tokens are billed at a lower rate than uncached input. See Prompt caching. Savings are:
The gateway only records this when the request actually read from or wrote to the provider prompt cache. The value can be negative on a request that mostly writes to the cache, because cache writes are often priced above the normal input rate.

​
Semantic caching

Gateway exact-match and semantic caching return a stored response without calling the model. On a hit, A is $0. Savings are the cost of the original request that populated the cache — the amount you would have paid if that hit had been a miss. The same number is stored as tfy.model.cache.potential_cost_savings_in_usd and aggregated as potentialCostSavings in Cache Metrics. This bucket includes both exact-match and semantic hits. The Overview labels it Semantic caching because semantic cache is a superset of exact-match.

​
Auto Routing

Auto Routing classifies each request and serves it with the cheapest configured model that can handle that complexity. Savings are estimated against the baseline model: the complex tier target — the most capable, and typically most expensive, model in the rule.
A few details:
  • The figure is 0 when the request was already served by the baseline.
  • It is an estimate: the baseline model might have used a different number of tokens for the same prompt.
  • It is omitted when A is $0 (for example a gateway cache hit), so a cached Auto Routing request is not counted twice.
  • It is omitted when the baseline model’s pricing cannot be resolved.

​
How the total is combined

The three numbers are computed independently per request and summed in the Overview. A single request can contribute more than one source — for example Auto Routing plus prompt caching on the cheaper model — but a gateway cache hit does not also claim Auto Routing savings.
Cost savings appear only after the underlying feature is enabled and traffic uses it. Set up prompt caching, gateway caching, or Auto Routing first, then return to Overview to see the breakdown.

Cost tracking

How actual cost is priced from public or private rates.

Metrics dashboard

Overview, cache metrics, and the rest of the dashboard.

Request log attributes

Per-request savings fields on model spans.