Skip to main content
The AI Gateway estimates how much you saved versus full-price inference. The Metrics Overview reports two separate figures. Cost savings covers what the gateway itself saved through semantic/exact caching and Auto Routing. Cache % (Savings) covers what provider prompt caching saved. Prompt caching is not part of Cost savings.

Where cost savings appear

Open AI Gateway → Metrics → Overview. The Usage Directory on the Overview shows savings in the LLM Cost view. Cache % (Savings) shows the average share of input tokens read from the provider prompt cache and the dollars saved by it. Cost savings shows what semantic/exact caching and Auto Routing saved versus full price. Hover a column header to see its definition.
The cache dashboard in the metrics Classic view also has a Total Cost Saved counter. That counter is gateway cache hits only (semantic and exact-match).

What the breakdown shows

How each figure is calculated

In every formula below, A is the actual cost of the request: today’s tfy.model.metric.cost_in_usd.

Prompt caching

Provider prompt caching reuses a previously computed prompt prefix. Cached input tokens are billed at a lower rate than uncached input. See Prompt caching. Savings are:
The gateway only records this when the request actually read from or wrote to the provider prompt cache. The value can be negative on a request that mostly writes to the cache, because cache writes are often priced above the normal input rate.

Semantic caching

Gateway exact-match and semantic caching return a stored response without calling the model. On a hit, A is $0. Savings are the cost of the original request that populated the cache — the amount you would have paid if that hit had been a miss. The same number is stored as tfy.model.cache.potential_cost_savings_in_usd and aggregated as potentialCostSavings in Cache Metrics. This bucket includes both exact-match and semantic hits.

Auto Routing

Auto Routing classifies each request and serves it with the cheapest configured model that can handle that complexity. Savings are estimated against the baseline model: the complex tier target — the most capable, and typically most expensive, model in the rule.
A few details:
  • The figure is 0 when the request was already served by the baseline.
  • It is an estimate: the baseline model might have used a different number of tokens for the same prompt.
  • It is omitted when A is $0 (for example a gateway cache hit), so a cached Auto Routing request is not counted twice.
  • It is omitted when the baseline model’s pricing cannot be resolved.

How Cost savings is combined

Both numbers are computed independently per request and summed in the Cost savings column. A gateway cache hit does not also claim Auto Routing savings. Prompt caching savings are computed the same way per request but are reported only in Cache % (Savings), so a request served by Auto Routing that also reads from the provider prompt cache shows up in both columns without being double counted.
Cost savings appear only after the underlying feature is enabled and traffic uses it. Set up prompt caching, gateway caching, or Auto Routing first, then return to Overview to see the savings.

Cost tracking

How actual cost is priced from public or private rates.

Metrics dashboard

Overview, cache metrics, and the rest of the dashboard.

Request log attributes

Per-request savings fields on model spans.