Disaster Recovery

TrueFoundry runs every tier of its hosted platform in more than one region so that a regional failure does not take your traffic, traces, or configuration with it. This page describes what is deployed where, how failure is detected, and what recovery you can expect for each component.

​

Key Components

The platform splits into two tiers, and each gets the recovery posture its state model allows. AI Gateway — a stateless service hosted in 14 locations across AWS, GCP, and Azure. Every location holds its configuration in memory and can serve any request, so the tier runs active-active: all locations are live at once and no location’s failure requires recovering anything. Control plane — deployed in two regions, eu-west-1 (primary) and us-east-2 (secondary). It is made up of:

• Stateless services, running in both regions:
  • Backend service — serves the dashboard, the APIs, and the configuration that gateway planes subscribe to. It reads and writes the database.
  • OpenTelemetry Collector — receives traces and metrics from the AI Gateway and from your instrumented applications at <control_plane_url>/api/otel. It tolerates the database being down, so trace ingest keeps working through a database outage.
• Stateful services:
  • Postgres — runs on Amazon Aurora, with the primary cluster in eu-west-1 and a replica in us-east-2 that receives continuous replication.
  • Cache — NATS KV. It holds no durable state: it is rebuilt automatically, and gateway planes fall back to fetching configuration over HTTP if it is unreachable.

The important architectural detail is that the stateless services in both regions talk to the same database and cache. The us-east-2 control plane does not have its own database — it connects across regions to the Aurora primary in eu-west-1. This is what makes stateless failover instant: either region can serve, because neither owns the state. Both tiers sit behind Cloudflare load balancers that health-check each origin and steer traffic away from unhealthy ones. Failover of the services is automatic, and you do not change endpoints, credentials, or DNS during one. The only manual step in the whole design is promoting the database replica, covered in Database Down. The diagram below shows the full design in one view.

​

Recovery Objectives

Two terms are used throughout this page:

• Recovery Time Objective (RTO) — how long a tier can be unavailable before it is serving again, measured in time. An RTO of 0 means no interruption to the service.
• Recovery Point Objective (RPO) — how much data can be lost, measured in time. An RPO of 0 means no data loss.

​

AI Gateway

The AI Gateway runs in 14 locations across three cloud providers. Every location is live and serving traffic, and each one is registered as an origin in a Cloudflare load balancer with its own health check attached. When a location stops passing its health check, Cloudflare removes it from rotation and routes traffic to the nearest healthy location. A request that was already in flight to the failing location is retried against another region rather than returned as an error, so no request is dropped. Because the gateway planes are stateless and hold their configuration in memory, a location that drops out takes no state with it. There is nothing to replicate or promote, and the remaining locations keep serving at full capacity rather than waiting for the failed one to recover. For the full list of locations, the regional and multi-regional endpoints, and the primary and fallback locations per region, see Globally Distributed SaaS Gateway.

​

Control Plane

Each of the two regions runs the full set of stateless services — the backend service and the Otel Collector — behind a Cloudflare load balancer performing TCP health checks against each origin. Under normal operation all traffic lands in eu-west-1, and both regions connect to the Aurora primary and the NATS cache there. The us-east-2 region holds no state of its own, so which region serves a request never matters. Because state and serving are separated this way, a control plane failure and a database failure are different events with different recovery paths.

​

Control Plane Down, Database Up

If the eu-west-1 control plane services become unavailable while the database is healthy, the Cloudflare health check fails and traffic falls back instantly and automatically to us-east-2. The us-east-2 control plane serves the dashboard, the APIs, and configuration sync exactly as before, writing to the same Aurora primary in eu-west-1. There is no data loss and no manual step.

​

Database Down

If the Aurora primary in eu-west-1 fails, the replica in us-east-2 must be promoted to master. This is a manual switch performed by TrueFoundry, and it completes in under 30 minutes. Data loss is bounded by the replication lag at the moment of failure — Aurora replicates across regions with latency typically under a second, so the RPO is measured in seconds. See Using switchover or failover in Amazon Aurora Global Database for the underlying mechanism. While the switch is in progress, most of the platform keeps working:

• Gateway traffic is unaffected. Gateway planes serve on their last synced configuration, as described in Graceful Degradation During Control Plane Outage.
• Trace ingest is unaffected. The Otel Collector tolerates the database being down and keeps receiving spans.
• Dashboard and API writes are unavailable until the promotion completes, since there is no writable database in that window.

​

Observability

The Otel Collector fails over with the rest of the stateless services: the same Cloudflare TCP health check redirects ingest from eu-west-1 to us-east-2 if the primary region becomes unavailable. The AI Gateway buffers spans and retries failed exports across the failover window, so traces emitted while eu-west-1 is unreachable are delivered once traffic redirects. Clients keep sending to the same /api/otel endpoint throughout, with no configuration change needed. Ingest also continues through a database outage, because the collector does not depend on the database being up. And request serving is never at risk either way — the AI Gateway never fails a request because trace export is failing.

​

Graceful Degradation During Control Plane Outage

A control plane outage does not stop the AI Gateway from serving requests. Gateway planes keep the configuration they have already synced and continue to authenticate, authorize, rate limit, and route traffic in memory. When the control plane returns, configuration reconciles automatically. The AI Gateway Plane Architecture FAQ covers this behavior in detail: the HTTP fallback used when the NATS queue is unreachable, the readiness probes that keep traffic away from a pod still syncing, and the periodic configuration republish that reconciles drift.

​

Status and Notifications

Live and historical availability for every hosted component, including per-region AI Gateway uptime, is published at status.truefoundry.com. You can subscribe to updates by email, RSS, or webhook using the Get Updates button on the status page. See AI Gateway Status Monitoring for the per-region view and subscription steps.

​

FAQ