AI Gateway: Connect, observe, and control agentic AI applications.

The Playground is the interactive UI on top of the AI Gateway where developers can try out different LLMs, prompts, MCP tools and configurations before wiring them into applications. You can select any model that has been onboarded in the “Models” tab, adjust parameters such as temperature, max tokens, streaming and stop sequences, and immediately see the impact on responses, token usage and latency. This makes it easy to experiment with model choices and generation settings without writing code.
‍
Once you are happy with a setup, the entire configuration—prompt, model, tools, guardrails and structured output schema—can be saved as a reusable template in a shared repository. The Playground also generates ready-to-use code snippets for the OpenAI client, LangChain and other libraries, using the unified AI Gateway API, so teams can take a working experiment and drop it straight into their services with minimal effort.

With TrueFoundry AI Gateway, all model providers and tools sit behind a single, unified API. Instead of managing separate SDKs, endpoints and keys for OpenAI, Anthropic, Bedrock, self-hosted models and others, applications talk to one gateway endpoint and use one gateway key. The gateway then routes requests to the right underlying model based on configuration, so you can swap models or providers without changing your application code. This unified access layer also extends to tools via the MCP protocol and to agents via the emerging A2A protocol, so models, tools and agents can all be orchestrated through the same control plane.
‍
For developers, this means simpler integration and a cleaner security model: provider keys are stored once in the gateway, access is governed centrally using RBAC and policies, and teams can standardize on a single client pattern across languages and frameworks. As new models or providers appear, they can be added to the gateway and become immediately available behind the same unified interface.

Prompts, tools and agent configurations are treated as first-class assets in the AI Gateway. In the Playground you can define system prompts, user prompts, input variables, MCP tools, guardrails and model settings, and then save them as named templates. Each template can have multiple versions so teams can iterate safely without overwriting each other’s logic, and roll back to previous versions when needed. This effectively becomes a prompt and agent configuration repository for your organization.
‍
When a particular configuration is ready to be shared more broadly, it can be published as an Agent App. Agent Apps are powered by the gateway but exposed through a simple, locked-down interface: business users or internal teams can interact with the agent exactly as it will run in production, while the underlying prompts, tools and guardrails remain immutable. This makes Agent Apps ideal for user acceptance testing, stakeholder demos and internal copilots, because product and platform teams retain control over the configuration while still giving others a safe way to try agentic workflows.

Guardrails in TrueFoundry AI Gateway operate on both the input and output paths to provide defense-in-depth. Before a request reaches a model, input guardrails can scan it for sensitive data such as PII, prompt injection patterns or disallowed topics, and either block, redact or transform the prompt based on your policies. After the model generates a response, output guardrails evaluate the content again for toxicity, bias, hallucinations, policy violations or accidental data leakage, and decide whether to return, modify or reject the response.
‍
The gateway can plug into existing safety and compliance services such as OpenAI Moderation, AWS Guardrails, Azure Content Safety and Azure PII detection, and it also supports custom rules written as configuration or Python code. Because guardrails are configured centrally and applied consistently across all models and applications going through the AI Gateway, security and compliance teams get a predictable way to enforce organizational policies for GenAI usage, including in regulated environments like healthcare, financial services and insurance.

Every request flowing through TrueFoundry AI Gateway is instrumented so you can see exactly how your GenAI workloads behave. The monitoring views show aggregate metrics such as total requests, input and output tokens, and cost, broken down by model, team, user, customer, environment or any other metadata you choose to attach. Performance is tracked using P99, P90 and P50 latency, time-to-first-token and inter-token latency, so you can quickly identify models or routes that are causing slowdowns or errors.
‍
For deeper debugging, there is a request-level view that lets you inspect individual calls, see the full prompt and response, and understand how routing, fallbacks and guardrails were applied. For agentic workflows using tools and MCP, the gateway can capture traces that show each step an agent took, which tools it called, and how intermediate results flowed through the system. All of these logs and metrics are also exposed via APIs, so platform and observability teams can build custom dashboards and alerts in their existing monitoring stacks.

The AI Gateway lets you express reliability and governance rules as configuration so they can be applied consistently and automated. Rate limits can be defined per team, user, model, application or environment, ensuring that no single consumer can exhaust capacity or overspend. Budgets and quotas can be set so that when usage crosses certain thresholds, requests are throttled, downgraded to cheaper models or blocked, depending on your business rules. Load-balancing policies can route traffic based on fixed weights, measured latency or priority, while fallback chains describe the sequence of models to try when errors or timeouts occur.
‍
All of these controls can be managed through the UI or declared in YAML and applied via the TrueFoundry CLI, enabling a GitOps workflow where gateway configuration lives alongside application code and infrastructure definitions. Combined with caching, batching and centralized API key management, these features allow platform teams to treat the AI Gateway as the single place where they define how GenAI should be used, how much can be spent, and how applications should behave under failure—without forcing individual application teams to re-implement these concerns over and over again.