> ## Documentation Index
> Fetch the complete documentation index at: https://www.truefoundry.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Budget Usage Recovery

> How TrueFoundry keeps budget usage counters durable, detects a loss of the gateway usage store, and automatically restores counters so budget enforcement stays correct.

Budget enforcement depends on counters that live in the AI Gateway's key-value store rather than in a database. TrueFoundry snapshots those counters, detects when the store has been wiped or rebuilt, and restores them automatically. This page describes how that works and what it does and does not recover.

<Info>
  Budget usage recovery runs automatically on the TrueFoundry-managed SaaS platform and on self-hosted installations. There is nothing to configure or enable.
</Info>

## Why budget usage needs recovery

The AI Gateway computes no running spend of its own. This is deliberate: as described in [AI Gateway Plane Architecture](/docs/platform/gateway-plane-architecture), there are no external calls in the path of a request, and every limit check happens in memory. Budget enforcement is therefore split across three pieces of state:

1. **The gateway reports cost.** After each request it publishes the cost of that request, asynchronously, and never blocks on doing so.
2. **The control plane aggregates it into usage counters.** Those counters live in the gateway's NATS JetStream key-value store, one per budget and per tracked entity.
3. **Blocked state is derived from the counters.** The gateway reads back which budgets are currently blocked and enforces that, without ever seeing a usage figure itself.

Because step 3 reads from step 2, losing the counters does not produce an error. It produces silence. Usage reads as zero, every block is dropped, and a budget that was fully exhausted starts allowing traffic again until spend re-accumulates from scratch. Nothing in the request path fails, so nothing surfaces the problem. That silent failure is what this mechanism exists to prevent.

<Note>
  This page is about counters being lost. A blocked user or team becoming unblocked at the next UTC period boundary is not a loss: that is the budget period resetting, which is normal and is covered in [Budget Limiting](/docs/ai-gateway/budget-limiting-v2).
</Note>

## How it works

Two loops run independently. One keeps a recent copy of every budget's usage, and the other watches for the store being wiped.

```mermaid theme={"dark"}
flowchart LR
    Store[Gateway usage store] -->|every hour| Snap[Snapshot to control<br/>plane database]
    Snap --> Keep[(Kept for 7 days)]
    Store -->|every 10 minutes| Check{Store identity<br/>unchanged?}
    Check -->|Yes| Baseline[Record new baseline]
    Check -->|No| Restore[Restore counters]
    Keep -.-> Restore
    Restore --> Store
```

### Hourly snapshot

Every hour, each tenant's budget usage is copied out of the gateway store and written to the control plane database, one record per budget, all sharing a single capture timestamp. Snapshots are kept for **7 days**.

A tenant whose snapshot fails in a given run is excluded from that run's cleanup. Without this, a tenant that is temporarily unreadable could have its last good recovery point deleted by the same run that failed to replace it.

### Loss detection

Every **10 minutes**, the store's identity is compared against the last known good values recorded in the control plane. Two things are checked: when the store was created, and how far its sequence counter has advanced.

A changed creation time, or a sequence counter that has gone backwards, means the store was wiped or rebuilt. Normal operation cannot produce either signal, because deletes, purges and routine cleanup of expired windows are all themselves writes, and every write pushes the sequence forward. The first check after an install records a baseline and makes no judgement.

### Automatic restore

Once a loss is detected, the stored snapshots are merged back into the live counters. The restore waits for the NATS cluster to converge first, meaning a leader has been elected and every replica is current, because a cluster mid-election cannot give an authoritative answer about what it holds. A single-server installation always qualifies.

Two properties make this safe to run unattended:

* **Only windows that are still current are restored.** A snapshot captured before a period rolled over cannot resurrect the previous period's spend. A lifetime budget's single window always counts as current.
* **Values are merged by taking the higher of the two, never overwritten.** Counters only grow within a window, so a restore can raise usage back toward the truth but can never lower it. If live traffic has already rebuilt a counter past the snapshot value, the live value wins.

Blocked state itself is not restored, and does not need to be. It is re-derived from the counters, so once usage is back, enforcement resumes on its own without any further action.

## Recovery states

| State | Meaning | Effect on snapshots |
| - | - | - |
| **Healthy** | No loss detected | Snapshots run normally |
| **Recovering** | A loss was detected; a restore is in progress or being retried | Paused, so a wiped store cannot overwrite good recovery points |
| **Failed** | The restore gave up after **3 attempts** | Still paused; a manual restore is required |

Pausing snapshots during recovery is the important detail. A snapshot taken from a wiped store would record zeros, and within a few hours that would silently replace every genuine recovery point with an empty one. Backups stay paused until the incident is resolved.

A restore fails for one of two reasons, and they point at different problems: either there were no snapshots to restore from, which means the hourly backups had already been failing before the loss, or writing the restored counters back did not succeed.

## What is and is not recovered

Recovery restores most usage, not all of it. The gaps are bounded and always in the same direction.

* **Up to an hour of spend is not recovered.** Anything spent between the last snapshot and the loss is gone. Counters therefore come back low, never high, so a budget may allow somewhat more than its limit for the remainder of that period. It will not block traffic that should have been allowed.
* **An old snapshot is still used.** Snapshot age never blocks a restore, because a stale snapshot is much closer to the truth than the zero left by a wipe.
* **Rate limit counters are out of scope.** This mechanism covers budgets only. See [Rate Limiting](/docs/ai-gateway/ratelimiting-v2) for how throughput limits behave.

<Warning>
  **Budgets split across more than 500 entities recover only their largest spenders.** Snapshots are taken from each budget's leaderboard, which holds its top 500 entries. For a budget using `per-user`, `per-model`, `per-virtual-account`, `per-agent`, or metadata partitioning across more entities than that, the long tail restarts the current period from zero. Aggregate budgets and budgets below 500 entities are unaffected.
</Warning>

## Manual restore

Every retained snapshot stays individually restorable, and a restore can be previewed before it is applied so the effect on each budget is known in advance. This is the path used when automatic restore reaches the **Failed** state, and it can also restore a single budget rather than a whole tenant.

Manual restores are performed by TrueFoundry. Contact [support](mailto:support@truefoundry.com) if you need one.

## FAQ

<AccordionGroup>
  <Accordion title="Does a budget stop enforcing while the counters are missing?">
    Yes, and this is the reason the mechanism exists. Blocked state is derived from usage counters, so when they read as zero every block is dropped and budgets allow traffic until spend re-accumulates. Detection runs every 10 minutes, which bounds how long that window can last before a restore begins.
  </Accordion>

  <Accordion title="Can recovery double-count my spend?">
    No. Restored values are merged by taking the higher of the snapshot and the live counter rather than being added, so applying the same snapshot twice has no additional effect. Only windows that are still current are restored, so a snapshot taken before a period boundary cannot add the previous period's spend to the new one.
  </Accordion>

  <Accordion title="What happens to budget dashboards during and after a loss?">
    During a loss, usage reads as zero. Dashboards surface an error rather than reporting a misleading zero when usage cannot be read at all. After a restore, both the usage figures and the per-entity breakdowns are rebuilt, subject to the top 500 limit described above.
  </Accordion>

  <Accordion title="Do I need to re-create or edit my budgets after a recovery?">
    No. Budget definitions live in the control plane database and are never affected by a loss of the usage store. Only the accumulated usage is at risk, and only the current period's usage at that. Nothing about your rules, limits, scopes, or alerts needs to be touched.
  </Accordion>

  <Accordion title="Does this protect rate limits too?">
    No. Rate limit counters are not snapshotted and are not restored. The exposure is much smaller: rate limit windows run from a minute to a day, so a lost counter rebuilds itself within at most one window of normal traffic. Budget periods run from a day to a lifetime, and a lost lifetime or quarterly counter would otherwise never rebuild. See [Rate Limiting](/docs/ai-gateway/ratelimiting-v2) for how those windows work.
  </Accordion>

  <Accordion title="How do I know a recovery happened?">
    Recovery status is not currently surfaced in the dashboard or in any public API, and no email, Slack, or PagerDuty notification is sent when a loss is detected or a restore completes. If you suspect a loss, contact [support](mailto:support@truefoundry.com).

    This is separate from budget milestone alerts, which notify you when spend crosses a threshold. Those are a different feature and work as described in [Budget Limiting](/docs/ai-gateway/budget-limiting-v2#budget-milestone-alerts).
  </Accordion>
</AccordionGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.