TrueFoundry AI Gateway: Built-in Governance & Monitoring for Enterprises
TrueFoundry AI Gateway is a unified AI gateway to securely manage and govern AI across 2500+ models with policy control, real-time monitoring, and up to 30% cost reduction. It achieves sub-3ms internal latency even under enterprise-scale workloads and offers secure deployment in VPC, on-prem and air-gapped environments.
Free for the first 30k requests each month
All of the TrueFoundry AI Gateway is free for the first 30k requests each month, with all 2,500+ models and no credit card. The free start includes:
- 30k requests/month
- All 2,500+ models
- Observability
- Cost tracking
- Up to 3 seats
Made for Real-World AI at Scale
| Figure | Value | What it covers |
|---|---|---|
| Uptime | 99.99% | Centralized failovers, routing, and guardrails in the AI Gateway ensure AI apps stay online, even when model providers don't. |
| Requests processed/month | 10B+ | The AI Gateway provides scalable, high-throughput inference for production AI. |
| Average cost optimization | 30% | Smart routing, batching, and budget controls in the AI Gateway reduce token waste. |
| Models | 1600+ | 1600+ models are connected through one AI gateway. |
AI Gateway: Unified LLM API Access
TrueFoundry AI Gateway simplifies a GenAI stack with a single AI Gateway that integrates all major models.
- Connect to OpenAI, Claude, Gemini, Groq, Mistral, and 250+ LLMs through one AI Gateway API.
- Use the AI Gateway to support chat, completion, embedding, and reranking model types.
- Centralize API key management and team authentication in one place.
- Orchestrate multi-model workloads through an organization's own infrastructure.
Unified access for APIs, keys, tools and agents
With TrueFoundry AI Gateway, all model providers and tools sit behind a single, unified API. Instead of managing separate SDKs, endpoints and keys for OpenAI, Anthropic, Bedrock, self-hosted models and others, applications talk to one gateway endpoint and use one gateway key. The gateway then routes requests to the right underlying model based on configuration, so teams can swap models or providers without changing application code. This unified access layer also extends to tools via the MCP protocol and to agents via the emerging A2A protocol, so models, tools and agents can all be orchestrated through the same control plane.
For developers, this means simpler integration and a cleaner security model: provider keys are stored once in the gateway, access is governed centrally using RBAC and policies, and teams can standardize on a single client pattern across languages and frameworks. As new models or providers appear, they can be added to the gateway and become immediately available behind the same unified interface.
Documentation: truefoundry.com/docs/ai-gateway/supported-providers
AI Gateway Observability
TrueFoundry AI Gateway Observability tracks AI gateway performance and costs, and ensures compliance across models in real-time.
- Monitor token usage, latency, error rates, and request volumes across the system.
- Store and inspect full request/response logs centrally to ensure compliance and simplify debugging.
- Tag traffic with metadata like user ID, team, or environment to gain granular insights.
- Filter logs and metrics by model, team, or geography to quickly pinpoint root causes and accelerate resolution.
Observability, tracing and debugging
Every request flowing through TrueFoundry AI Gateway is instrumented so teams can see exactly how their GenAI workloads behave. The monitoring views show aggregate metrics such as total requests, input and output tokens, and cost, broken down by model, team, user, customer, environment or any other metadata a team chooses to attach. Performance is tracked using P99, P90 and P50 latency, time-to-first-token and inter-token latency, so teams can quickly identify models or routes that are causing slowdowns or errors.
For deeper debugging, there is a request-level view to inspect individual calls, see the full prompt and response, and understand how routing, fallbacks and guardrails were applied. For agentic workflows using tools and MCP, the gateway can capture traces that show each step an agent took, which tools it called, and how intermediate results flowed through the system. All of these logs and metrics are also exposed via APIs, so platform and observability teams can build custom dashboards and alerts in their existing monitoring stacks.
Documentation: truefoundry.com/docs/ai-gateway/analytics
Quota & Access Control via AI Gateway
TrueFoundry AI Gateway lets teams set budgets and quotas per team, user, model, application or environment, and once usage crosses a set threshold, requests are throttled, downgraded to cheaper models or blocked.
TrueFoundry AI Gateway enforces governance, controls costs, and reduces risk with enterprise AI gateway policy management.
- Apply rate limits per user, service, or endpoint.
- Set cost-based or token-based quotas using metadata filters.
- Use role-based access control (RBAC) to isolate and manage usage.
- Govern service accounts and agent workloads at scale through centralized rules.
Enterprise AI gateway controls ensure predictable usage, strong access boundaries, and scalable team-level governance for GenAI infrastructure.
Policies, rate limits, fallbacks and budgets as configuration
The AI Gateway expresses reliability and governance rules as configuration so they can be applied consistently and automated. Rate limits can be defined per team, user, model, application or environment, ensuring that no single consumer can exhaust capacity or overspend. Budgets and quotas can be set so that when usage crosses certain thresholds, requests are throttled, downgraded to cheaper models or blocked, depending on an organization's business rules. Load-balancing policies can route traffic based on fixed weights, measured latency or priority, while fallback chains describe the sequence of models to try when errors or timeouts occur.
All of these controls can be managed through the UI or declared in YAML and applied via the TrueFoundry CLI, enabling a GitOps workflow where gateway configuration lives alongside application code and infrastructure definitions. Combined with caching, batching and centralized API key management, these features allow platform teams to treat the AI Gateway as the single place where they define how GenAI should be used, how much can be spent, and how applications should behave under failure—without forcing individual application teams to re-implement these concerns over and over again.
Documentation: truefoundry.com/docs/ai-gateway/gateway-access-control
Low-Latency Inference
TrueFoundry AI Gateway runs the most performance-sensitive workloads through a high-speed AI gateway infrastructure.
- Achieve sub-3ms internal latency even under enterprise-scale workloads.
- Scale to manage burst traffic and high-throughput workloads.
- Deliver predictable response times for real-time chat, RAG, and AI assistants.
- Place deployments close to inference layers to minimize latency and eliminate network lag.
The AI Gateway sits directly in the production inference path — its low-latency architecture ensures no performance tradeoffs.
AI Gateway architecture
From the architecture diagram on this page:
| Component | Role in the AI Gateway architecture |
|---|---|
| Developer/Admin | The Developer/Admin adds models and sets the load-balancing configuration through the Control Plane UI. |
| Control Plane | The Control Plane contains the UI, the Backend, NATS, Postgres and ClickHouse. |
| Gateway | The Gateway subscribes to users, models and configuration, and to latency and tokens per model for rate limit and load-balancing decisions. The Gateway publishes data about tokens, cost, user and model in each request. |
| Application | The Application sends the LLM Request to the Gateway and receives the LLM Response from the Gateway. |
| LLM Model | The LLM Model connects to the Gateway. |
Documentation: truefoundry.com/docs/ai-gateway/gateway-architecture
AI Gateway Routing & Fallbacks
When a request fails, TrueFoundry AI Gateway automatically falls back to secondary models, and it supports latency-based and weighted routing.
TrueFoundry AI Gateway ensures reliability, even during model failures, with smart AI Gateway traffic controls.
- Supports latency-based routing to the fastest available LLM.
- Distribute traffic intelligently using weighted load balancing for reliability and scale.
- Automatically fallback to secondary models when a request fails.
- Use geo-aware routing to meet regional compliance and availability needs.
This AI gateway routing system ensures AI apps never go offline, even when individual models face downtime or spike in latency.
Documentation: truefoundry.com/docs/ai-gateway/load-balancing-overview
Serve Self-Hosted Models
TrueFoundry AI Gateway exposes open-source models with full control.
- Deploy LLaMA, Mistral, Falcon, and more with zero SDK changes.
- Full compatibility with vLLM, SGLang, KServe, and Triton.
- Streamline operations with Helm-based management of autoscaling, GPU scheduling, and deployments.
- Run an organization's own models in VPC, hybrid, or air-gapped environments.
Documentation: truefoundry.com/docs/deploying-an-llm-model-from-the-model-catalogue
AI Gateway + MCP Integration
TrueFoundry AI Gateway powers secure agent workflows through the AI Gateway's native MCP support.
- Connect enterprise tools like Slack, GitHub, Confluence, and Datadog.
- Register internal MCP Servers with minimal setup required.
- Apply OAuth2, RBAC, and metadata policies to every tool call.
Documentation: truefoundry.com/docs/ai-gateway/mcp/mcp-overview
AI Gateway Guardrails
TrueFoundry AI Gateway builds secure AI applications with configurable AI gateway guardrails and policy controls.
- Enforce an organization's own safety guardrails, including PII filtering and toxicity detection.
- Customize the AI Gateway with guardrails tailored to an organization's compliance and safety needs.
Guardrails, safety checks and PII controls end-to-end
Guardrails in TrueFoundry AI Gateway operate on both the input and output paths to provide defense-in-depth. Before a request reaches a model, input guardrails can scan it for sensitive data such as PII, prompt injection patterns or disallowed topics, and either block, redact or transform the prompt based on an organization's policies. After the model generates a response, output guardrails evaluate the content again for toxicity, bias, hallucinations, policy violations or accidental data leakage, and decide whether to return, modify or reject the response.
The gateway can plug into existing safety and compliance services such as OpenAI Moderation, AWS Guardrails, Azure Content Safety and Azure PII detection, and it also supports custom rules written as configuration or Python code. Because guardrails are configured centrally and applied consistently across all models and applications going through the AI Gateway, security and compliance teams get a predictable way to enforce organizational policies for GenAI usage, including in regulated environments like healthcare, financial services and insurance.
Documentation: truefoundry.com/docs/ai-gateway/guardrails-overview
Enterprise-Ready
With TrueFoundry, an organization's data and models are securely housed within its own cloud or on-prem infrastructure.
| Area | Detail |
|---|---|
| Compliance & Security | TrueFoundry follows SOC 2, HIPAA, and GDPR standards to ensure data protection. |
| Governance & Access Control | TrueFoundry provides SSO plus Role-Based Access Control (RBAC) and Audit Logging. |
| Enterprise Support & Reliability | TrueFoundry offers 24/7 support with SLA-backed response SLAs. |
Deploy TrueFoundry in any environment
TrueFoundry deploys in VPC, on-prem, air-gapped, or across multiple clouds. No data leaves a team's domain. Teams get complete sovereignty, isolation, and enterprise-grade compliance wherever TrueFoundry runs.
Real Outcomes at TrueFoundry
Why Enterprises Choose TrueFoundry
10+ Fortune 500s run production AI on TrueFoundry. Customers shown on truefoundry.com include Resmed, Automation Anywhere, Siemens Healthineers, Innovaccer, Cox, NetApp and FloQast.
TrueFoundry’s AI Gateway gave us a unified layer to manage model access, routing, guardrails, and cost controls across teams. What earlier required multiple custom integrations and security reviews now happens through a single governed interface. It has accelerated productionization, increased visibility into spend and performance, and enabled us to scale AI experimentation safely across the organization.
Pratik Agarwal, Sr. Director, Data Science & AI Innovation, Resmed
With TrueFoundry’s AI Gateway, we finally have one consistent interface for all model providers, policies, and telemetry. It eliminated the overhead of managing keys, routing logic, and scattered observability. Introducing new models is now just configuration. The Gateway has improved developer velocity, reduced DevOps burden, and helped us operate multimodel systems with real-time insights and governance.
Vibhas Gejji, Staff ML Engineer, Automation Anywhere
TrueFoundry’s AI Gateway standardized how every team interacts with LLMs, embeddings, and RAG components. Instead of scattered integrations, we now control access, routing policies, and safety guardrails centrally. The ability to optimize for cost or latency without changing applications has been a game-changer. It’s made our AI architecture cleaner, more secure, and far easier to scale.
Indroneel G., Intelligent Process Leader, Siemens Healthineers
TrueFoundry’s AI Gateway has become our control layer for safe, governed AI adoption. It consolidates security, observability, and model usage policies into one place, giving us full visibility into performance and spend. Developers get a consistent interface across clouds and models, while leadership gets governance and predictability. It has meaningfully reduced friction in scaling enterprise AI.
Nilav Ghosh, Senior Director, AI, Innovaccer
AI Gateway Playground, prompt management and Agent Apps
The Playground is the interactive UI on top of the TrueFoundry AI Gateway where developers can try out different LLMs, prompts, MCP tools and configurations before wiring them into applications. Developers can select any model that has been onboarded in the “Models” tab, adjust parameters such as temperature, max tokens, streaming and stop sequences, and immediately see the impact on responses, token usage and latency. This makes it easy to experiment with model choices and generation settings without writing code.
Once a setup is ready, the entire configuration—prompt, model, tools, guardrails and structured output schema—can be saved as a reusable template in a shared repository. The Playground also generates ready-to-use code snippets for the OpenAI client, LangChain and other libraries, using the unified AI Gateway API, so teams can take a working experiment and drop it straight into their services with minimal effort.
Prompts, tools and agent configurations are treated as first-class assets in the AI Gateway. In the Playground, teams can define system prompts, user prompts, input variables, MCP tools, guardrails and model settings, and then save them as named templates. Each template can have multiple versions so teams can iterate safely without overwriting each other’s logic, and roll back to previous versions when needed. This effectively becomes a prompt and agent configuration repository for the organization.
When a particular configuration is ready to be shared more broadly, it can be published as an Agent App. Agent Apps are powered by the gateway but exposed through a simple, locked-down interface: business users or internal teams can interact with the agent exactly as it will run in production, while the underlying prompts, tools and guardrails remain immutable. This makes Agent Apps ideal for user acceptance testing, stakeholder demos and internal copilots, because product and platform teams retain control over the configuration while still giving others a safe way to try agentic workflows.