> ## Documentation Index
> Fetch the complete documentation index at: https://www.truefoundry.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Disaster Recovery

> How TrueFoundry recovers from regional failures across the AI Gateway and the control plane, including database failover, using multi-region health checks.

TrueFoundry runs every tier of its hosted platform in more than one region so that a regional failure does not take your traffic, traces, or configuration with it. This page describes what is deployed where, how failure is detected, and what recovery you can expect for each component.

<Info>
  This page describes the **TrueFoundry-managed SaaS platform**. If you self-host the control plane, disaster recovery is governed by your own infrastructure. See [Control Plane Architecture](/docs/platform/control-plane-architecture) for the components you need to make redundant.
</Info>

## Key Components

The platform splits into two tiers, and each gets the recovery posture its state model allows.

**AI Gateway** — a stateless service hosted in 14 locations across AWS, GCP, and Azure. Every location holds its configuration in memory and can serve any request, so the tier runs active-active: all locations are live at once and no location's failure requires recovering anything.

**Control plane** — deployed in two regions, `eu-west-1` (primary) and `us-east-2` (secondary). It is made up of:

* **Stateless services**, running in both regions:
  * **Backend service** — serves the dashboard, the APIs, and the configuration that gateway planes subscribe to. It reads and writes the database.
  * **OpenTelemetry Collector** — receives traces and metrics from the AI Gateway and from your instrumented applications at `<control_plane_url>/api/otel`. It tolerates the database being down, so trace ingest keeps working through a database outage.
* **Stateful services**:
  * **Postgres** — runs on Amazon Aurora, with the primary cluster in `eu-west-1` and a replica in `us-east-2` that receives continuous replication.
  * **Cache** — NATS KV. It holds no durable state: it is rebuilt automatically, and gateway planes fall back to fetching configuration over HTTP if it is unreachable.

The important architectural detail is that **the stateless services in both regions talk to the same database and cache**. The `us-east-2` control plane does not have its own database — it connects across regions to the Aurora primary in `eu-west-1`. This is what makes stateless failover instant: either region can serve, because neither owns the state.

| Tier              | Posture        | Regions                                        |
| ----------------- | -------------- | ---------------------------------------------- |
| **AI Gateway**    | Active-active  | 14 locations across AWS, GCP, and Azure        |
| **Control plane** | Active-passive | `eu-west-1` (primary), `us-east-2` (secondary) |

Both tiers sit behind Cloudflare load balancers that health-check each origin and steer traffic away from unhealthy ones. Failover of the services is automatic, and you do not change endpoints, credentials, or DNS during one. The only manual step in the whole design is promoting the database replica, covered in [Database Down](#database-down). The diagram below shows the full design in one view.

<Frame caption="Active-active gateway planes fail over among themselves; both control plane regions serve against the same Aurora primary, which fails over via a manual master switch">
  <img src="https://mintcdn.com/truefoundry/AQD6952Pt0MYcBJb/images/docs/platform/disaster-recovery-architecture.png?fit=max&auto=format&n=AQD6952Pt0MYcBJb&q=85&s=e8e4c0b5f14b5a66b57d036638a94265" alt="Disaster recovery architecture showing the application reaching 14 active-active AI Gateway locations through a Cloudflare load balancer with an unhealthy location removed from rotation, and the control plane in eu-west-1 and us-east-2 each running a backend service and Otel Collector, both writing to the single Aurora primary in eu-west-1, with a replica in us-east-2 promoted by a manual master switch in under 30 minutes" width="1536" height="1024" data-path="images/docs/platform/disaster-recovery-architecture.png" />
</Frame>

## Recovery Objectives

Two terms are used throughout this page:

* **Recovery Time Objective (RTO)** — how long a tier can be unavailable before it is serving again, measured in time. An RTO of 0 means no interruption to the service.
* **Recovery Point Objective (RPO)** — how much data can be lost, measured in time. An RPO of 0 means no data loss.

| Component                            | RTO              | RPO     | Basis                                                                                                                                                                                  |
| ------------------------------------ | ---------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **AI Gateway**                       | 0                | 0       | Active-active. Unhealthy locations leave rotation while the rest keep serving, so nothing is recovered or promoted.                                                                    |
| **Control plane stateless services** | Near-instant     | 0       | Automatic Cloudflare fallback to `us-east-2`, which serves against the same database. Spans in flight to the Otel Collector are buffered and retried by the gateway across the window. |
| **Control plane database**           | Under 30 minutes | Seconds | Manual switch of the Aurora master to the `us-east-2` replica. Data loss is bounded by replication lag at the moment of failure. See [Database Down](#database-down).                  |

<Note>
  These are architectural design targets, not contractual commitments. For the terms TrueFoundry commits to, see the [Service Level Agreement](https://www.truefoundry.com/service-level-agreement). For live and historical availability, see [status.truefoundry.com](https://status.truefoundry.com).
</Note>

## AI Gateway

The AI Gateway runs in 14 locations across three cloud providers. Every location is live and serving traffic, and each one is registered as an origin in a Cloudflare load balancer with its own health check attached.

When a location stops passing its health check, Cloudflare removes it from rotation and routes traffic to the nearest healthy location. A request that was already in flight to the failing location is retried against another region rather than returned as an error, so no request is dropped. Because the gateway planes are stateless and hold their configuration in memory, a location that drops out takes no state with it. There is nothing to replicate or promote, and the remaining locations keep serving at full capacity rather than waiting for the failed one to recover.

```mermaid theme={"dark"}
flowchart LR
    App[Your application] --> LB[Cloudflare Load Balancer<br/>gateway.truefoundry.ai]
    LB --> Check{Nearest location<br/>passing health check?}

    Check -->|Yes| Nearest[Serve from nearest location]
    Check -->|No| Next[Serve from next-closest<br/>healthy location]

    Nearest --> Provider[LLM / MCP provider]
    Next --> Provider
```

For the full list of locations, the regional and multi-regional endpoints, and the primary and fallback locations per region, see [Globally Distributed SaaS Gateway](/docs/ai-gateway/globally-distributed-saas).

<Warning>
  Single-location endpoints such as `orf.gateway.truefoundry.ai` bypass this steering and **do not fail over**. If that one location becomes unavailable, requests to it fail. Use the global endpoint `https://gateway.truefoundry.ai`, or a multi-regional endpoint such as `eu.gateway.truefoundry.ai`, so that failover applies.
</Warning>

## Control Plane

Each of the two regions runs the full set of stateless services — the backend service and the Otel Collector — behind a Cloudflare load balancer performing TCP health checks against each origin. Under normal operation all traffic lands in `eu-west-1`, and both regions connect to the Aurora primary and the NATS cache there. The `us-east-2` region holds no state of its own, so which region serves a request never matters.

Because state and serving are separated this way, a control plane failure and a database failure are different events with different recovery paths.

### Control Plane Down, Database Up

If the `eu-west-1` control plane services become unavailable while the database is healthy, the Cloudflare health check fails and traffic falls back instantly and automatically to `us-east-2`. The `us-east-2` control plane serves the dashboard, the APIs, and configuration sync exactly as before, writing to the same Aurora primary in `eu-west-1`. There is no data loss and no manual step.

### Database Down

If the Aurora primary in `eu-west-1` fails, the replica in `us-east-2` must be promoted to master. This is a **manual switch** performed by TrueFoundry, and it completes in **under 30 minutes**. Data loss is bounded by the replication lag at the moment of failure — Aurora replicates across regions with latency typically under a second, so the RPO is measured in seconds. See [Using switchover or failover in Amazon Aurora Global Database](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database-disaster-recovery.html) for the underlying mechanism.

While the switch is in progress, most of the platform keeps working:

* **Gateway traffic is unaffected.** Gateway planes serve on their last synced configuration, as described in [Graceful Degradation During Control Plane Outage](#graceful-degradation-during-control-plane-outage).
* **Trace ingest is unaffected.** The Otel Collector tolerates the database being down and keeps receiving spans.
* **Dashboard and API writes are unavailable** until the promotion completes, since there is no writable database in that window.

### Observability

The Otel Collector fails over with the rest of the stateless services: the same Cloudflare TCP health check redirects ingest from `eu-west-1` to `us-east-2` if the primary region becomes unavailable. The AI Gateway buffers spans and retries failed exports across the failover window, so traces emitted while `eu-west-1` is unreachable are delivered once traffic redirects. Clients keep sending to the same `/api/otel` endpoint throughout, with no configuration change needed.

Ingest also continues through a database outage, because the collector does not depend on the database being up. And request serving is never at risk either way — the AI Gateway never fails a request because trace export is failing.

## Graceful Degradation During Control Plane Outage

A control plane outage does not stop the AI Gateway from serving requests. Gateway planes keep the configuration they have already synced and continue to authenticate, authorize, rate limit, and route traffic in memory. When the control plane returns, configuration reconciles automatically.

The [AI Gateway Plane Architecture](/docs/platform/gateway-plane-architecture#faq) FAQ covers this behavior in detail: the HTTP fallback used when the NATS queue is unreachable, the readiness probes that keep traffic away from a pod still syncing, and the periodic configuration republish that reconciles drift.

## Status and Notifications

Live and historical availability for every hosted component, including per-region AI Gateway uptime, is published at [status.truefoundry.com](https://status.truefoundry.com). You can subscribe to updates by email, RSS, or webhook using the **Get Updates** button on the status page.

See [AI Gateway Status Monitoring](/docs/ai-gateway/globally-distributed-saas#ai-gateway-status-monitoring) for the per-region view and subscription steps.

## FAQ

<AccordionGroup>
  <Accordion title="Do I need to change my endpoints or credentials during a failover?">
    No. Both tiers fail over behind their load balancer, so `gateway.truefoundry.ai`, the control plane URL, and the `/api/otel` ingest path stay the same throughout. API keys, JWTs, and SSO configuration are unaffected.
  </Accordion>

  <Accordion title="What happens to requests that are in flight when a region fails?">
    They are retried against a different region. Cloudflare retries a request that fails at an unhealthy origin rather than returning the failure to your client, so an in-flight request is served by another healthy location instead of being dropped. Because the gateway planes are stateless, any location can serve the retry.
  </Accordion>

  <Accordion title="Will I lose traces if the observability region fails?">
    No. Export is asynchronous and retried across the failover window, so spans emitted while `eu-west-1` is unreachable are delivered once traffic redirects to `us-east-2`.

    Your LLM traffic is unaffected either way. The AI Gateway never fails a request because trace export is failing.
  </Accordion>

  <Accordion title="What is affected while the database master switch is in progress?">
    Gateway traffic and trace ingest are not affected: gateway planes serve on their last synced configuration, and the Otel Collector keeps receiving spans without the database. Dashboard and API operations that write to the database are unavailable until the `us-east-2` replica is promoted, which takes under 30 minutes.
  </Accordion>

  <Accordion title="Where is my data during a control plane failover?">
    Within the same two regions described on this page. Aurora replicates only between `eu-west-1` and `us-east-2`, so a failover does not move data outside those regions. If you have specific data residency requirements, see [Data Residency and Sovereignty](/docs/platform/security-and-compliance#data-residency-and-sovereignty).
  </Accordion>

  <Accordion title="Does the AI Gateway keep working if the control plane is down?">
    Yes. Gateway planes serve traffic using their last synced configuration and reconcile automatically when the control plane returns. See [Graceful Degradation During Control Plane Outage](#graceful-degradation-during-control-plane-outage).
  </Accordion>
</AccordionGroup>