What security teams actually need from an AI gateway?

Ask ten people in security what an AI gateway does and you'll get ten answers: a proxy that swaps API keys, a place to log prompts, a firewall for chatbots. None of that is wrong exactly, but none is the full picture, and the gap between "partly right" and "actually governs AI usage" is where breaches happen.

IBM's 2025 Cost of a Data Breach Report found that 20% of breached organizations were compromised through shadow AI, meaning employees using generative AI tools security never approved or knew about, adding roughly $670,000 to the average breach cost. Among those organizations, 97% lacked proper access controls, and a Ponemon Institute survey of 600 organizations found 63% had no AI governance policy at all. [1]

Cost visibility and governance is not a problem that can be solved by buying one more tool. This post walks through it the way a CISO tends to ask: can you see and govern all AI usage in your org, what can you do once you see it, and can it work with the stack you already run.

Question 1: Can you see and govern all the AI usage in your org?

Ask most security leaders to list every AI tool used inside their company, and they can't. AI usage arrives through at least four doors, and each needs a different answer.

Apps and agents your own developers build. The easiest case, since your teams own the code. Route calls through a scoped, revocable credential instead of a raw API key. TrueFoundry does this with a Virtual Account Token: tied to one application, visible in logs as that application, killable on its own.

Desktop and CLI tools employees already use. Cursor, Claude Code, and Codex CLI typically let you set which endpoint they talk to. That makes enforcement a configuration problem, not an interception problem: push TrueFoundry's gateway endpoint fleet-wide through your MDM (Jamf, Intune).

Consumer web apps. ChatGPT and Claude's web interface are the hard case: no config file, no code to change. TrueFoundry solves this with aitori, an on-device proxy that works like a VPN client, intercepting traffic at the OS level and redirecting it to the gateway via a device certificate. It's open source: the code is on GitHub.

AI embedded inside SaaS tools you already pay for. Salesforce's Einstein, Slack's AI summaries, and similar features are the honest exception. The model call never leaves the vendor's servers, so there's nothing to intercept. No gateway, including TrueFoundry's, solves this today; the fallback is CASB-style controls or the vendor's own admin settings.

Three of four doors are solvable today. The fourth isn't, and any vendor claiming otherwise is worth a second look.

Question 2: Once you can see the traffic, what can you actually do about it?

Visibility answers "what's happening." Acting on it breaks into three areas.

Access management

Most people frame this around one scenario: if someone leaves, how fast does access die. TrueFoundry handles that through SSO with SCIM, JIT, or invite-only provisioning, so an IdP group change becomes an access change automatically. But it also covers authentication (tokens tied to Okta, Entra, or Google Workspace, dying the moment the IdP account is deactivated), least-privilege role-based access, and ongoing oversight of who holds privileged access right now.

The piece most vendors skip is non-human identity. Teams build agents on any framework (LangGraph, CrewAI, Bedrock, a homegrown script), and nothing forces anyone to declare one exists before it starts calling internal tools. That's shadow IT, except the thing hiding can take actions, not just read data. TrueFoundry's fix separates an agent's identity (who is this agent, required for every agent) from its registration (what it can do, on whose behalf). For a chain like planning agent to research agent to retrieval agent to tool calls, TrueFoundry uses token exchange: each hop gets a fresh token, the human stays the accountable subject, and every acting agent gets recorded rather than disappearing behind the human's name. Full breakdown of agent identity and multi-agent governance here, including how it maps to Okta's Cross-App Access and Microsoft Entra's On-Behalf-Of.

Token exchange at a single gateway: the subject (user) is preserved, the acting agent is recorded and the scope is narrowed to the next callee

MCP governance and guardrails

MCP tools take real actions (writing to a database, sending an email), a bigger blast radius than a chat reply. TrueFoundry's MCP Gateway separates how a caller authenticates in, which user or app can reach which specific server and tools (tool-level, not whole-server), and how the gateway authenticates outbound to the server, independent of inbound. That lets a private, VPC-hosted server be reached through a tunnel with zero inbound ports, registered once centrally instead of per developer.

Before vs. After: The MCP Gateway Difference

None of that tells you what's inside a call. Guardrails do, reading and redacting the payload itself, in both directions and for MCP tool calls too. TrueFoundry's AI Gateway ships built-in guardrails (PII/PHI detection, prompt injection defense, content moderation, secrets detection, custom policy via Cedar or OPA) and also plugs into whatever guardrail vendor you already use. More on MCP governance and runtime guardrails here, and more on the identity side here.

LLM Request Guardrail Flow

Audit trail, which covers three needs

"We log everything" usually means one of three things: a live dashboard for ongoing visibility, automatic capture of every prompt, response, identity, and cost for forensic reconstruction, or a clean SIEM export (Splunk, Datadog, Grafana) for compliance evidence. TrueFoundry covers all three from the same log, which is the actual proof behind a SOC 2 or HIPAA posture. Full audit-trail checklist here.

Question 3: Does this replace your security stack, or work with it?

The real objection here is usually change management, not technology: nobody wants to rip out tools they trust. A well-built AI gateway sits downstream of what you already run.

Point AI-bound traffic from your SWG (Zscaler, Netskope) at TrueFoundry's gateway with the user's identity in a header, and it governs from there without displacing the SWG's job. Jamf and Intune deploy aitori fleet-wide like any other agent. CrowdStrike Falcon plays a different role, a backstop blocking unsanctioned AI apps outright at the endpoint. None of this asks you to choose between your stack and TrueFoundry; it asks TrueFoundry to plug into it.

Can you trust the vendor running all of this?

Worth verifying rather than assuming. Look for SOC 2 Type II, HIPAA, and GDPR coverage with published attestations (TrueFoundry's are at trust.truefoundry.com), a documented access-review schedule, annual penetration testing with tracked remediation, and a short, disclosed subprocessor list.

Related reading

• Multi-agent A2A governance and the agent gateway: agent identity, token exchange, and delegation
• MCP Gateway, Cross-App Access, and identity-governed agent access: MCP-specific identity via Okta and Entra
• Enterprise AI security with MCP Gateway runtime guardrails: guardrail enforcement for MCP tool calls
• Shadow AI detection tools: finding AI usage you haven't sanctioned
• AI audit checklist: building out the audit piece above

Conclusion

An AI gateway is worth evaluating on three questions: can it see and govern every kind of AI traffic, does it give you real control once that traffic is visible, and does it slot into the stack you've already built.

If you're working through this for your own org, see how TrueFoundry's AI Gateway handles the full picture.

‍