What the checklist covers:
56
Evaluation criteria across model, tool, and agent traffic
4
Areas: LLM gateway, MCP gateway, agent gateway, and platform
49
Must-have capabilities, so you know what a production gateway can't skip
<5 ms
Target p95 gateway overhead for time-to-first-token
Inside the checklist
Download the checklist to get:
An open reference your platform and engineering teams can use to compare AI gateways side by side, or to plan your own rollout.
Download the Checklist✓
The full 56-point AI gateway framework
Every criterion grouped by area, with the question to ask and a must-have or nice-to-have priority.
✓
LLM routing, resilience, and guardrail criteria
Virtual models, weighted load balancing, failover, retries, health checks, input and output guardrails, and budgets.
✓
MCP and agent gateway criteria
MCP registries, 2- and 3-legged OAuth, tool-level RBAC, agent identity, and human-in-the-loop approvals.
✓
Performance and compliance benchmarks
Latency and throughput targets, multi-region deployment, SSO, SOC 2 Type II, GDPR, and encryption requirements.
Why it matters
The AI Gateway Has Grown from a Router into Core AI Infrastructure
The first AI gateways were simple routing layers: one API key, a few providers, maybe a retry. That's no longer enough. Teams now run dozens of models, connect agents to MCP servers with access to systems of record, and roll out coding agents like Claude Code and Cursor to every engineer.
All of that traffic needs the same things: one endpoint, resilience when a provider fails, guardrails on what goes in and comes out, access control, cost tracking, and a full audit trail. When each team builds those pieces separately, coverage is inconsistent and nobody can see the whole picture.
One gateway, three kinds of traffic, one set of controls. The checklist covers each layer.
A modern AI gateway covers three kinds of traffic. The LLM gateway governs model calls. The MCP gateway governs the tools agents use. The agent gateway governs the agents themselves. Underneath all three sit the platform controls: SSO, RBAC, logging, dashboards, rate limits, and compliance.
What a production-grade AI gateway looks like
Developers use one OpenAI-compatible endpoint, or their native provider SDKs, with zero code changes. Virtual model names decouple apps from model choice, and traffic is load-balanced, retried, and failed over automatically when a provider throttles or goes down. Guardrails inspect prompts and responses for PII and prompt injection, and every request is costed and held to a budget, including traffic from coding agents.
MCP servers sit in one registry behind one endpoint, with OAuth for users and services, tool-level RBAC, tool guardrails, and a full audit trail. Agents are registered, access-controlled, and given their own scoped identity, with human approval for sensitive actions. The whole gateway adds under 5 ms p95, scales past 100 concurrent requests per second, and runs across regions with SOC 2 Type II, GDPR, and encryption in place.
An open framework for an evolving category
TrueFoundry built this checklist from work with hundreds of engineering and platform teams, and made it public and vendor-neutral so any organization can use it to evaluate, implement, and scale an AI gateway with confidence.
Preview
The four areas in the AI gateway evaluation checklist
A sample of the criteria in each area. The full checklist includes all 56, with priorities and evaluation questions.
Area 1 · 18 criteria
LLM gateway
An OpenAI-compatible endpoint lets one SDK reach every model
A primary model fails over automatically to secondary targets
Traffic from coding agents is attributed, budgeted, and access-controlled
+15 more criteria
Area 2 · 16 criteria
MCP gateway
MCP servers are registered in one central catalog
Per-user OAuth 2.0 with inline consent during a chat
Tool-level RBAC by user, team, and service account
+13 more criteria
Area 3 · 8 criteria
Agent gateway
All agents are registered and governed in one catalog
Each agent has its own scoped service identity
Sensitive tool calls pause for human approval
+5 more criteria
Area 4 · 14 criteria
Platform and cross-cutting
SSO via SAML 2.0 and OIDC with your enterprise IdP
Every request logged with prompt, response, tokens, cost, and identity
Under 5 ms p95 overhead at 100+ concurrent RPS
+11 more criteria
4.7
★★★★★
Gartner Peer Insights
Rated 4.7 on Gartner Peer Insights
Enterprises run TrueFoundry's AI Gateway, spanning LLM, MCP, and Agent Gateway, to connect, observe, and govern models, tools, and agents from one control plane, as SaaS, on-premises, or air-gapped.
96%
of reviewers likely to recommend
4.9
ease of deployment and admin
3–4 ms
added gateway latency
