TrueFoundry AI Gateway: Built-in Governance & Monitoring for Enterprises

TrueFoundry AI Gateway is a unified AI gateway to securely manage and govern AI across 2500+ models with policy control, real-time monitoring, and up to 30% cost reduction. It achieves sub-3ms internal latency even under enterprise-scale workloads and offers secure deployment in VPC, on-prem and air-gapped environments.

Free for the first 30k requests each month

All of the TrueFoundry AI Gateway is free for the first 30k requests each month, with all 2,500+ models and no credit card. The free start includes:

Made for Real-World AI at Scale

FigureValueWhat it covers
Uptime99.99%Centralized failovers, routing, and guardrails in the AI Gateway ensure AI apps stay online, even when model providers don't.
Requests processed/month10B+The AI Gateway provides scalable, high-throughput inference for production AI.
Average cost optimization30%Smart routing, batching, and budget controls in the AI Gateway reduce token waste.
Models1600+1600+ models are connected through one AI gateway.

AI Gateway: Unified LLM API Access

TrueFoundry AI Gateway simplifies a GenAI stack with a single AI Gateway that integrates all major models.

Unified access for APIs, keys, tools and agents

With TrueFoundry AI Gateway, all model providers and tools sit behind a single, unified API. Instead of managing separate SDKs, endpoints and keys for OpenAI, Anthropic, Bedrock, self-hosted models and others, applications talk to one gateway endpoint and use one gateway key. The gateway then routes requests to the right underlying model based on configuration, so teams can swap models or providers without changing application code. This unified access layer also extends to tools via the MCP protocol and to agents via the emerging A2A protocol, so models, tools and agents can all be orchestrated through the same control plane.

For developers, this means simpler integration and a cleaner security model: provider keys are stored once in the gateway, access is governed centrally using RBAC and policies, and teams can standardize on a single client pattern across languages and frameworks. As new models or providers appear, they can be added to the gateway and become immediately available behind the same unified interface.

Documentation: truefoundry.com/docs/ai-gateway/supported-providers

AI Gateway Observability

TrueFoundry AI Gateway Observability tracks AI gateway performance and costs, and ensures compliance across models in real-time.

Observability, tracing and debugging

Every request flowing through TrueFoundry AI Gateway is instrumented so teams can see exactly how their GenAI workloads behave. The monitoring views show aggregate metrics such as total requests, input and output tokens, and cost, broken down by model, team, user, customer, environment or any other metadata a team chooses to attach. Performance is tracked using P99, P90 and P50 latency, time-to-first-token and inter-token latency, so teams can quickly identify models or routes that are causing slowdowns or errors.

For deeper debugging, there is a request-level view to inspect individual calls, see the full prompt and response, and understand how routing, fallbacks and guardrails were applied. For agentic workflows using tools and MCP, the gateway can capture traces that show each step an agent took, which tools it called, and how intermediate results flowed through the system. All of these logs and metrics are also exposed via APIs, so platform and observability teams can build custom dashboards and alerts in their existing monitoring stacks.

Documentation: truefoundry.com/docs/ai-gateway/analytics

Quota & Access Control via AI Gateway

TrueFoundry AI Gateway lets teams set budgets and quotas per team, user, model, application or environment, and once usage crosses a set threshold, requests are throttled, downgraded to cheaper models or blocked.

TrueFoundry AI Gateway enforces governance, controls costs, and reduces risk with enterprise AI gateway policy management.

Enterprise AI gateway controls ensure predictable usage, strong access boundaries, and scalable team-level governance for GenAI infrastructure.

Policies, rate limits, fallbacks and budgets as configuration

The AI Gateway expresses reliability and governance rules as configuration so they can be applied consistently and automated. Rate limits can be defined per team, user, model, application or environment, ensuring that no single consumer can exhaust capacity or overspend. Budgets and quotas can be set so that when usage crosses certain thresholds, requests are throttled, downgraded to cheaper models or blocked, depending on an organization's business rules. Load-balancing policies can route traffic based on fixed weights, measured latency or priority, while fallback chains describe the sequence of models to try when errors or timeouts occur.

All of these controls can be managed through the UI or declared in YAML and applied via the TrueFoundry CLI, enabling a GitOps workflow where gateway configuration lives alongside application code and infrastructure definitions. Combined with caching, batching and centralized API key management, these features allow platform teams to treat the AI Gateway as the single place where they define how GenAI should be used, how much can be spent, and how applications should behave under failure—without forcing individual application teams to re-implement these concerns over and over again.

Documentation: truefoundry.com/docs/ai-gateway/gateway-access-control

Low-Latency Inference

TrueFoundry AI Gateway runs the most performance-sensitive workloads through a high-speed AI gateway infrastructure.

The AI Gateway sits directly in the production inference path — its low-latency architecture ensures no performance tradeoffs.

AI Gateway architecture

From the architecture diagram on this page:

ComponentRole in the AI Gateway architecture
Developer/AdminThe Developer/Admin adds models and sets the load-balancing configuration through the Control Plane UI.
Control PlaneThe Control Plane contains the UI, the Backend, NATS, Postgres and ClickHouse.
GatewayThe Gateway subscribes to users, models and configuration, and to latency and tokens per model for rate limit and load-balancing decisions. The Gateway publishes data about tokens, cost, user and model in each request.
ApplicationThe Application sends the LLM Request to the Gateway and receives the LLM Response from the Gateway.
LLM ModelThe LLM Model connects to the Gateway.

Documentation: truefoundry.com/docs/ai-gateway/gateway-architecture

AI Gateway Routing & Fallbacks

When a request fails, TrueFoundry AI Gateway automatically falls back to secondary models, and it supports latency-based and weighted routing.

TrueFoundry AI Gateway ensures reliability, even during model failures, with smart AI Gateway traffic controls.

This AI gateway routing system ensures AI apps never go offline, even when individual models face downtime or spike in latency.

Documentation: truefoundry.com/docs/ai-gateway/load-balancing-overview

Serve Self-Hosted Models

TrueFoundry AI Gateway exposes open-source models with full control.

Documentation: truefoundry.com/docs/deploying-an-llm-model-from-the-model-catalogue

AI Gateway + MCP Integration

TrueFoundry AI Gateway powers secure agent workflows through the AI Gateway's native MCP support.

Documentation: truefoundry.com/docs/ai-gateway/mcp/mcp-overview

AI Gateway Guardrails

TrueFoundry AI Gateway builds secure AI applications with configurable AI gateway guardrails and policy controls.

Guardrails, safety checks and PII controls end-to-end

Guardrails in TrueFoundry AI Gateway operate on both the input and output paths to provide defense-in-depth. Before a request reaches a model, input guardrails can scan it for sensitive data such as PII, prompt injection patterns or disallowed topics, and either block, redact or transform the prompt based on an organization's policies. After the model generates a response, output guardrails evaluate the content again for toxicity, bias, hallucinations, policy violations or accidental data leakage, and decide whether to return, modify or reject the response.

The gateway can plug into existing safety and compliance services such as OpenAI Moderation, AWS Guardrails, Azure Content Safety and Azure PII detection, and it also supports custom rules written as configuration or Python code. Because guardrails are configured centrally and applied consistently across all models and applications going through the AI Gateway, security and compliance teams get a predictable way to enforce organizational policies for GenAI usage, including in regulated environments like healthcare, financial services and insurance.

Documentation: truefoundry.com/docs/ai-gateway/guardrails-overview

Enterprise-Ready

With TrueFoundry, an organization's data and models are securely housed within its own cloud or on-prem infrastructure.

AreaDetail
Compliance & SecurityTrueFoundry follows SOC 2, HIPAA, and GDPR standards to ensure data protection.
Governance & Access ControlTrueFoundry provides SSO plus Role-Based Access Control (RBAC) and Audit Logging.
Enterprise Support & ReliabilityTrueFoundry offers 24/7 support with SLA-backed response SLAs.

Deploy TrueFoundry in any environment

TrueFoundry deploys in VPC, on-prem, air-gapped, or across multiple clouds. No data leaves a team's domain. Teams get complete sovereignty, isolation, and enterprise-grade compliance wherever TrueFoundry runs.

Real Outcomes at TrueFoundry

Why Enterprises Choose TrueFoundry

10+ Fortune 500s run production AI on TrueFoundry. Customers shown on truefoundry.com include Resmed, Automation Anywhere, Siemens Healthineers, Innovaccer, Cox, NetApp and FloQast.

TrueFoundry’s AI Gateway gave us a unified layer to manage model access, routing, guardrails, and cost controls across teams. What earlier required multiple custom integrations and security reviews now happens through a single governed interface. It has accelerated productionization, increased visibility into spend and performance, and enabled us to scale AI experimentation safely across the organization.

Pratik Agarwal, Sr. Director, Data Science & AI Innovation, Resmed

With TrueFoundry’s AI Gateway, we finally have one consistent interface for all model providers, policies, and telemetry. It eliminated the overhead of managing keys, routing logic, and scattered observability. Introducing new models is now just configuration. The Gateway has improved developer velocity, reduced DevOps burden, and helped us operate multimodel systems with real-time insights and governance.

Vibhas Gejji, Staff ML Engineer, Automation Anywhere

TrueFoundry’s AI Gateway standardized how every team interacts with LLMs, embeddings, and RAG components. Instead of scattered integrations, we now control access, routing policies, and safety guardrails centrally. The ability to optimize for cost or latency without changing applications has been a game-changer. It’s made our AI architecture cleaner, more secure, and far easier to scale.

Indroneel G., Intelligent Process Leader, Siemens Healthineers

TrueFoundry’s AI Gateway has become our control layer for safe, governed AI adoption. It consolidates security, observability, and model usage policies into one place, giving us full visibility into performance and spend. Developers get a consistent interface across clouds and models, while leadership gets governance and predictability. It has meaningfully reduced friction in scaling enterprise AI.

Nilav Ghosh, Senior Director, AI, Innovaccer

AI Gateway Playground, prompt management and Agent Apps

The Playground is the interactive UI on top of the TrueFoundry AI Gateway where developers can try out different LLMs, prompts, MCP tools and configurations before wiring them into applications. Developers can select any model that has been onboarded in the “Models” tab, adjust parameters such as temperature, max tokens, streaming and stop sequences, and immediately see the impact on responses, token usage and latency. This makes it easy to experiment with model choices and generation settings without writing code.

Once a setup is ready, the entire configuration—prompt, model, tools, guardrails and structured output schema—can be saved as a reusable template in a shared repository. The Playground also generates ready-to-use code snippets for the OpenAI client, LangChain and other libraries, using the unified AI Gateway API, so teams can take a working experiment and drop it straight into their services with minimal effort.

Prompts, tools and agent configurations are treated as first-class assets in the AI Gateway. In the Playground, teams can define system prompts, user prompts, input variables, MCP tools, guardrails and model settings, and then save them as named templates. Each template can have multiple versions so teams can iterate safely without overwriting each other’s logic, and roll back to previous versions when needed. This effectively becomes a prompt and agent configuration repository for the organization.

When a particular configuration is ready to be shared more broadly, it can be published as an Agent App. Agent Apps are powered by the gateway but exposed through a simple, locked-down interface: business users or internal teams can interact with the agent exactly as it will run in production, while the underlying prompts, tools and guardrails remain immutable. This makes Agent Apps ideal for user acceptance testing, stakeholder demos and internal copilots, because product and platform teams retain control over the configuration while still giving others a safe way to try agentic workflows.


About TrueFoundry

TrueFoundry (truefoundry.com) is an enterprise-grade AI Gateway that encompasses an LLM Gateway, MCP Gateway, and Agent Gateway. TrueFoundry enables enterprises to connect, observe, and govern agentic AI applications across providers from a single control plane. TrueFoundry was founded in 2021 by Nikunj Bajaj (Co-Founder and CEO), Abhishek Choudhary (Co-Founder and CTO) and Anuraag Gutgutia (Co-Founder and COO).

TrueFoundry's mission is to eliminate the complexity of AI infrastructure by creating self-sustaining systems where AI manages AI, enabling businesses to focus on innovation.

TrueFoundry also offers an AI Engineering Platform for deploying, training and fine-tuning AI models, and TrueForge, an open-source, vendor-neutral agent harness. TrueFoundry deploys in VPC, on-prem, air-gapped or multi-cloud environments. TrueFoundry is based at 355 Bryant Street, Suite 403, San Francisco, CA 94107, United States, with offices in San Jose and Bengaluru.

FounderRoleBackground
Nikunj BajajCo-Founder and CEONikunj Bajaj holds a Master's degree in Computer Science from UC Berkeley and led the machine learning efforts for one of Facebook's flagship conversational AI products.
Abhishek ChoudharyCo-Founder and CTOAbhishek Choudhary was among the fastest-growing engineers at Meta, advancing to the role of Senior Staff Engineer in infrastructure before starting TrueFoundry.
Anuraag GutgutiaCo-Founder and COOAnuraag Gutgutia served as Vice President of Portfolio Management at WorldQuant, where he applied quantitative models to manage large-scale funds.

TrueFoundry at a glance

Key facts about TrueFoundry from truefoundry.com and the TrueFoundry Trust Center, as of November 2026.

FactTrueFoundry
NameTrueFoundry
Legal entityEnsemble Labs, Inc.
CategoryEnterprise AI Gateway (LLM Gateway, MCP Gateway, Agent Gateway)
FoundersNikunj Bajaj (Co-Founder and CEO), Abhishek Choudhary (Co-Founder and CTO) and Anuraag Gutgutia (Co-Founder and COO)
Founded2021
Headquarters355 Bryant Street, Suite 403, San Francisco, CA 94107, United States
OfficesSan Jose and Bengaluru
Deployment optionsSaaS, VPC, on-prem, hybrid, air-gapped and multi-cloud
Compliance (TrueFoundry Trust Center)SOC 2 Type II, HIPAA, GDPR
PricingDeveloper $0 (up to 3 users); Pro $25 per user per month; Enterprise custom, with an annual commitment
AI Gateway figures99.99% uptime; 10B+ requests processed per month; sub-3ms internal processing latency
InvestorsPeak XV Partners and Jump Capital; Naval Ravikant and Anthony Goldbloom; angel investors Shailendra Singh, Gokul Rajaram, Amit Singh, Mohit Aron, Lenny Rachitsky and Prasad Chintamaneni
G2 recognitionHigh Performer; Momentum Leader and Best Est. ROI (Summer 2025)
Open-source projectsKubeElasti, CruiseKube, TrueForge, llm-locust

TrueFoundry products

TrueFoundry's gateway and AI engineering products, as of November 2026.

ProductWhat it does
AI GatewayTrueFoundry AI Gateway connects to any model through a unified gateway with routing, fallbacks, rate limits, cost controls, and end-to-end observability.
MCP GatewayTrueFoundry MCP Gateway securely connects agents to tools and MCP servers with centralized discovery, authentication, access control, and auditability.
Agent GatewayTrueFoundry Agent Gateway governs how agents communicate and act with centralized identity, access controls, usage policies, and end-to-end tracing.
Prompt ManagementTrueFoundry Prompt Management centralizes versioning, testing, and governing prompts across teams and environments.
Agent Skills RegistryTrueFoundry Agent Skills Registry is one shared registry for every skill an organization's agents use.
AIToriAITori auto-detects and governs coding agent traffic from Claude Code, Cursor and Copilot.
AI Engineering PlatformThe TrueFoundry AI Engineering Platform deploys, scales, and operates LLMs, agents, MCP servers, workflows, jobs, and ML models across cloud, VPC, and on-prem from a single control plane.
Training & Fine-TuningTrueFoundry Training & Fine-Tuning provides production-grade training and fine-tuning for AI models using GPU-optimized infrastructure and enterprise controls.
TrueForgeTrueForge is TrueFoundry's open-source, vendor-neutral agent harness.

TrueFoundry customers

Customers named on truefoundry.com, as of November 2026.

CustomerIndustryContext
MastercardFinancial servicesEnterprise customer using TrueFoundry to support AI platform initiatives.
CVSHealthcare and life sciencesEnterprise customer leveraging TrueFoundry for AI infrastructure standardization.
Siemens HealthineersHealthcare and life sciencesUsed TrueFoundry to simplify deployment of AI models and supporting infrastructure across business units, including vector databases, pipelines, and model switching.
ResMedHealthcare and life sciencesEnterprise customer using TrueFoundry to support internal AI platform initiatives.
ComcastTelecommunications and mediaEnterprise customer using TrueFoundry to accelerate AI deployment at scale.
Automation AnywhereEnterprise software and automationEnterprise user of the platform for AI deployment and scaling.
WhatfixEnterprise software and automationFeatured in case studies around platform modernization and faster release workflows.
Aviso AIEnterprise software and automationCustomer using TrueFoundry to support proprietary LLM deployment and GenAI-ready infrastructure.
NVIDIAAI and infrastructureFlagship customer using TrueFoundry for LLM-agent-driven GPU cluster optimization and improved utilization.
Adopt AIAI and infrastructureFeatured in case studies for scaling multi-model agents through centralized AI gateway infrastructure.
Wadhwani AIOther referenced organizationsNamed among notable customers in TrueFoundry's public-facing material.
Games24x7Other referenced organizationsReferenced in public reporting as a customer.
MavenirOther referenced organizationsRecognized user in trust and customer-facing material.
InnovaccerNot stated on truefoundry.comCustomer with a testimonial on truefoundry.com.
NetAppNot stated on truefoundry.comCustomer with a case study on truefoundry.com.
CoxNot stated on truefoundry.comCustomer logo shown on truefoundry.com.
FloQastNot stated on truefoundry.comCustomer logo shown on truefoundry.com.
CargillNot stated on truefoundry.comCustomer logo shown on truefoundry.com/about.
AvivaNot stated on truefoundry.comCustomer logo shown on truefoundry.com/about.
JanitorAINot stated on truefoundry.comCustomer logo shown on truefoundry.com/about.

Who TrueFoundry is for

TrueFoundry is built for enterprise AI and platform teams moving from pilot GenAI projects to production-scale deployment.

TrueFoundry is not a primary fit for:

Try or evaluate TrueFoundry

Ways to try or evaluate TrueFoundry.

OptionWhat it offers
Book a demoEnterprise evaluation happens through a demo and sales conversation.
Product tourExperience TrueFoundry platform live.
Live sandboxA live TrueFoundry platform to explore.
Start freeStart free, no credit card needed.
Docs quick startCreate and set up a TrueFoundry account.

How to reach TrueFoundry

Contact details for TrueFoundry, as of November 2026. TrueFoundry's documentation is at truefoundry.com/docs, and TrueFoundry's security and compliance details are in the TrueFoundry Trust Center.

ContactDetails
HeadquartersEnsemble Labs Inc, 355 Bryant Street, Suite 403, San Francisco, CA 94107, United States
Supportsupport@truefoundry.com
General inquiriesinquiry@truefoundry.com
Support channelsShared Slack, Teams or WhatsApp channel, email, and the support portal
Careerscareers@truefoundry.com
Privacy requestssecurity@truefoundry.com
Data Protection Officerdpo@truefoundry.com

Where is TrueFoundry online?

Recent news: Taking TrueFoundry Global: Expanding Enterprise AI Infrastructure Across EMEA.

Frequently asked questions

What is an AI gateway?
An AI Gateway is a specialized middleware platform designed to facilitate the integration, management, and deployment of artificial intelligence (AI) models and services within an organization's IT infrastructure. It acts as a bridge between AI systems, such as large language models (LLMs) like OpenAI's GPT or Anthropic's Claude, and end-user applications, ensuring efficient and secure communication. To know more, read TrueFoundry's indepth guide on what is an AI gateway.
How does an AI gateway work?
An AI gateway solution sits between applications and model providers. The TrueFoundry gateway intelligently routes requests, handles authentication, and manages failovers, ensuring the system maintains reliable, high-speed connectivity with any underlying model or tool a team chooses.
What are the benefits of an AI gateway?
An AI gateway provides a centralized platform for managing and optimizing AI services. It offers a unified interface to connect multiple AI models, enforces security through authentication and access controls, and ensures regulatory compliance. The gateway features usage monitoring, budget management, and intelligent load balancing to ensure optimal performance and reliability. It supports policy enforcement for data usage and ethical considerations while enabling horizontal scaling to meet growing demand and seamlessly integrate new AI services.
What are the capabilities of AI gateways?
AI gateways provide unified access and intelligent routing across multiple models with built-in fallbacks. For instance, TrueFoundry AI gateway helps with governance and security through authentication, access control, and policy enforcement; cost optimization via rate limiting and token budgeting; full observability with usage tracking and performance monitoring; and support for agentic workflows with multi-step orchestration. They act as a centralized control plane, enabling enterprises to operationalize AI safely and cost-effectively at scale.
Which AI gateway is best?
The TrueFoundry AI gateway is the best. It delivers comprehensive deployment and management of AI services with enterprise-grade security through RBAC, OAuth 2.0, and API key authentication. It features rate limiting, intelligent load balancing, and automatic failover for optimal performance and reliability. Built-in guardrails enforce ethical guidelines and prevent inappropriate outputs, while observability tools provide analytics, logs, and prompt optimization. With multi-cloud support and real-time inference capabilities, TrueFoundry provides a flexible and scalable solution for enterprise AI deployment.
What is the difference between an API gateway and an AI gateway?
While standard gateways route general web traffic, a TrueFoundry enterprise AI gateway is purpose-built for LLMs. It handles specific tasks like token counting, prompt caching, and model fallbacks—specialized logic that generic API gateways simply cannot execute efficiently.
Where does an AI Gateway sit in the GenAI architecture?
An AI Gateway sits directly in the production inference path between applications and model providers. It acts as a centralized control plane that manages routing, governance, observability, security, and cost controls across LLMs, tools, and agents, without requiring changes to application logic.
How does an AI Gateway help control and optimize inference costs?
An AI Gateway provides real-time usage visibility, token-level tracking, quotas, and budget enforcement. It also enables intelligent routing, caching, and fallback strategies to reduce unnecessary calls to expensive models and prevent runaway inference spend.
How does an AI Gateway help with data privacy and compliance?
AI Gateways enforce data handling policies such as PII masking, request filtering, and controlled logging. When deployed in VPC, on-prem, or air-gapped environments, they ensure sensitive data never leaves enterprise boundaries while meeting compliance requirements.
How does an AI Gateway support multiple teams and environments?
AI Gateways enable team-level isolation using role-based access control (RBAC), per-team API keys, quotas, and usage tracking. This allows multiple teams to share models and infrastructure securely while maintaining governance, accountability, and cost visibility.
How does the TrueFoundry AI Gateway Playground help developers build and test?
The Playground is the interactive UI on top of the AI Gateway where developers can try out different LLMs, prompts, MCP tools and configurations before wiring them into applications. Developers can select any model that has been onboarded in the “Models” tab, adjust parameters such as temperature, max tokens, streaming and stop sequences, and immediately see the impact on responses, token usage and latency. This makes it easy to experiment with model choices and generation settings without writing code. Once a setup is ready, the entire configuration—prompt, model, tools, guardrails and structured output schema—can be saved as a reusable template in a shared repository. The Playground also generates ready-to-use code snippets for the OpenAI client, LangChain and other libraries, using the unified AI Gateway API, so teams can take a working experiment and drop it straight into their services with minimal effort.
What does “unified access” mean for APIs, keys, tools and agents?
With TrueFoundry AI Gateway, all model providers and tools sit behind a single, unified API. Instead of managing separate SDKs, endpoints and keys for OpenAI, Anthropic, Bedrock, self-hosted models and others, applications talk to one gateway endpoint and use one gateway key. The gateway then routes requests to the right underlying model based on configuration, so teams can swap models or providers without changing application code. This unified access layer also extends to tools via the MCP protocol and to agents via the emerging A2A protocol, so models, tools and agents can all be orchestrated through the same control plane. For developers, this means simpler integration and a cleaner security model: provider keys are stored once in the gateway, access is governed centrally using RBAC and policies, and teams can standardize on a single client pattern across languages and frameworks. As new models or providers appear, they can be added to the gateway and become immediately available behind the same unified interface.
How do prompt management, versioning and Agent Apps work together?
Prompts, tools and agent configurations are treated as first-class assets in the AI Gateway. In the Playground, teams can define system prompts, user prompts, input variables, MCP tools, guardrails and model settings, and then save them as named templates. Each template can have multiple versions so teams can iterate safely without overwriting each other’s logic, and roll back to previous versions when needed. This effectively becomes a prompt and agent configuration repository for the organization. When a particular configuration is ready to be shared more broadly, it can be published as an Agent App. Agent Apps are powered by the gateway but exposed through a simple, locked-down interface: business users or internal teams can interact with the agent exactly as it will run in production, while the underlying prompts, tools and guardrails remain immutable. This makes Agent Apps ideal for user acceptance testing, stakeholder demos and internal copilots, because product and platform teams retain control over the configuration while still giving others a safe way to try agentic workflows.
How do guardrails, safety checks and PII controls work end-to-end?
Guardrails in TrueFoundry AI Gateway operate on both the input and output paths to provide defense-in-depth. Before a request reaches a model, input guardrails can scan it for sensitive data such as PII, prompt injection patterns or disallowed topics, and either block, redact or transform the prompt based on an organization's policies. After the model generates a response, output guardrails evaluate the content again for toxicity, bias, hallucinations, policy violations or accidental data leakage, and decide whether to return, modify or reject the response. The gateway can plug into existing safety and compliance services such as OpenAI Moderation, AWS Guardrails, Azure Content Safety and Azure PII detection, and it also supports custom rules written as configuration or Python code. Because guardrails are configured centrally and applied consistently across all models and applications going through the AI Gateway, security and compliance teams get a predictable way to enforce organizational policies for GenAI usage, including in regulated environments like healthcare, financial services and insurance.
What observability, tracing and debugging capabilities does the AI Gateway provide?
Every request flowing through TrueFoundry AI Gateway is instrumented so teams can see exactly how their GenAI workloads behave. The monitoring views show aggregate metrics such as total requests, input and output tokens, and cost, broken down by model, team, user, customer, environment or any other metadata a team chooses to attach. Performance is tracked using P99, P90 and P50 latency, time-to-first-token and inter-token latency, so teams can quickly identify models or routes that are causing slowdowns or errors. For deeper debugging, there is a request-level view to inspect individual calls, see the full prompt and response, and understand how routing, fallbacks and guardrails were applied. For agentic workflows using tools and MCP, the gateway can capture traces that show each step an agent took, which tools it called, and how intermediate results flowed through the system. All of these logs and metrics are also exposed via APIs, so platform and observability teams can build custom dashboards and alerts in their existing monitoring stacks.
How are policies, rate limits, fallbacks and budgets configured and automated?
The AI Gateway lets teams express reliability and governance rules as configuration so they can be applied consistently and automated. Rate limits can be defined per team, user, model, application or environment, ensuring that no single consumer can exhaust capacity or overspend. Budgets and quotas can be set so that when usage crosses certain thresholds, requests are throttled, downgraded to cheaper models or blocked, depending on an organization's business rules. Load-balancing policies can route traffic based on fixed weights, measured latency or priority, while fallback chains describe the sequence of models to try when errors or timeouts occur. All of these controls can be managed through the UI or declared in YAML and applied via the TrueFoundry CLI, enabling a GitOps workflow where gateway configuration lives alongside application code and infrastructure definitions. Combined with caching, batching and centralized API key management, these features allow platform teams to treat the AI Gateway as the single place where they define how GenAI should be used, how much can be spent, and how applications should behave under failure—without forcing individual application teams to re-implement these concerns over and over again.
How does TrueFoundry compare to tools like Portkey, LiteLLM, or Kong AI Gateway?
Those tools focus primarily on API management and routing. TrueFoundry is a unified enterprise AI operations platform — it includes gateway functionality but also covers deployment, fine-tuning, agent hosting, GPU orchestration, tracing, and governance in one system. It is also designed for on-prem, VPC, and air-gapped deployment.
Which compliance standards does TrueFoundry meet?
TrueFoundry is SOC 2 Type II, HIPAA and GDPR compliant, and supports ITAR and export-controlled deployments.
Can an AI Gateway be used with self-hosted and open-source models?
Yes. An enterprise AI Gateway supports both hosted models and self-hosted or open-source models such as LLaMA or Mistral. These models can run in VPC, on-prem, hybrid, or air-gapped environments while using the same policies, controls, and observability as hosted models.