AI Red Teaming for Agents: Attacks, Campaigns, and Runtime Defense

What AI red teaming actually means

Traditional red teaming borrows from military exercise: a team is chartered to attack your systems as an adversary would, then report what worked. AI red teaming keeps the charter and changes the target — you probe the model’s instruction-following, the trust boundary around retrieved content, and the actions the surrounding agent may take.

You are testing a system, not a model. A model that refuses a harmful request in isolation is a weak result if the agent around it will fetch a webpage, read hidden instructions and call a tool. The failures live in the seams.

The attack classes

Five families cover most of what a serious campaign produces. They do not share a control.

Indirect prompt injection changes the threat model. The canonical example is the 2025 Comet browser incident, cited in TrueFoundry’s agent-guardrails docs: a webpage carried hidden instructions written for the agent summarising it, and the agent followed them. No user tricked, no credential stolen — the content was the attack. Once an agent reads untrusted text, all of it is an untrusted instruction stream.

Sponge attacks get filed as billing and they are security — OWASP renamed “Model Denial of Service” to LLM10 Unbounded Consumption for exactly that reason. We mapped the full list in OWASP LLM Top 10: which risks a gateway actually fixes; four of the ten are request-path problems, and those four are what red teaming a live endpoint surfaces.

How agent red teaming differs from pen testing

Same discipline, different physics.

Offline campaigns vs runtime enforcement

Two products, two problems.

Campaigns tell you what to enforce; enforcement makes the finding stop mattering. A campaign producing 40 pages and no change to the request path has not reduced risk. Enforcement without campaigns defends only against attacks someone already wrote a detector for. Run one pre-launch against the assembled system, on every model change including provider checkpoint updates, and on every tool-surface change — one new MCP server can expand blast radius more than a model swap.

Where teams get this wrong

Red teaming the model instead of the system. A team tests the base model, gets a clean result, then ships an agent with a database tool and a web fetcher. The model never failed; the system fails on the first page carrying hidden instructions.

Treating detection as prevention. Every injection detector is a classifier; classifiers have false negatives and attackers iterate cheaply. Assume injection occasionally succeeds and narrow what a compromised agent can do. Against a read-only agent that is an incident report; against an agent with a shell it is a breach.

Running the campaign once. A dated report describes a system that no longer exists: different checkpoint, new MCP servers, a changed system prompt. And guardrails rarely see the whole request — exclusions are platform-specific, so find yours before drawing the control matrix.

Want to see what your traffic is actually carrying?

Attach a guardrail group in Audit mode and read a week of traces before enforcing anything.

Get started free

How this works in TrueFoundry

First, the boundary, plainly. TrueFoundry’s guardrails are runtime, inline enforcement: they validate or mutate content in the request path and return a decision. TrueFoundry does not ship or document offline red-team campaign tooling — no attack generation, no adversarial corpus, no scheduled scans, no red-team report. Across the AI Gateway guardrail docs “red teaming” appears essentially once, in OWASP LLM04 guidance recommending you “test your models against certain evaluation frameworks and red teaming” — an external recommendation, not a feature. DeepKeep is the partner page where red teaming surfaces as a capability, and it is DeepKeep’s, delivered vendor-side [VERIFY]. If you need campaigns, buy campaigns. What follows is the enforcement half.

Four hooks, and an execution order that saves you money

Guardrails attach to four hooks: LLM Input, LLM Output, MCP Tool Pre-Invoke and MCP Tool Post-Invoke.

The LLM path runs in seven documented steps, and step four is the interesting one. Input mutation runs first and blocks until done. Input validation then kicks off in the background, in parallel with the model request, so it does not sit on your time to first token — and if it fails mid-flight the gateway cancels the in-flight model request so you are not billed for it. Output mutation and validation run synchronously after the response arrives, so an output block still costs you the model call.

Validate vs Mutate, Enforce vs Enforce But Ignore On Error

Two independent settings per guardrail. Operation Mode is Validate or Mutate: Validate blocks without touching content, Mutate rewrites content and can still block, running by priority with the lower number first. Enforcement Strategy decides what happens on a violation, and separately on a guardrail error:

Enforce But Ignore On Error is the fail-open/fail-closed switch, and the most consequential dropdown on the page: full Enforce means a vendor outage takes your application down with it. The documented ladder is Audit → Enforce But Ignore On Error → Enforce.

Built-ins and partner integrations

Ten guardrails are native, per the canonical truefoundry-guardrails page: Secrets Detection, Code Safety Linter, SQL Sanitizer, Regex Pattern Matching, Prompt Injection, PII Detection, Content Moderation, Metadata Validation, Cedar and OPA Guardrails. One doc inconsistency: the overview page lists nine, omitting Metadata Validation. Use the ten-item list.

Three run on managed Azure services — Content Moderation, PII/PHI Detection and Prompt Injection — and work only when TrueFoundry hosts the gateway, not on self-hosted or hybrid “Gateway Plane only” deployments. Bring-your-own-key alternatives are documented for each: Prisma AIRS for injection, CrowdStrike for PII, Bedrock Guardrails for moderation.

The adversarial-relevant partners, named from their own doc pages:

That is thirteen security-focused partners. Adding the judge and moderation integrations — Patronus, NVIDIA NeMo, Guardrails AI, Arthur AI, Verra, Azure PII, Azure Prompt Shield, OpenAI Moderations, AWS Bedrock Guardrails — brings the roster to roughly twenty as of September 2026, plus a custom-guardrail contract for anything else.

Guardrails on MCP tool calls

Pre-invoke runs synchronously on the tool arguments; on failure the tool does not run. Post-invoke runs on the result; on failure the result is withheld from the model. Pre-invoke is where tool abuse dies — a SQL sanitizer reading the argument string, a code-safety linter on an execution tool. Post-invoke is where indirect injection dies, because a tool result is untrusted content.

Guardrails run on every tool call separately — five tools in a row means five full sets of checks. And X-TFY-GUARDRAILS-SCOPE, which narrows LLM guardrails to the last N messages, does not apply to MCP hooks. MCP tool poisoning and gateway defense covers the tool-description side.

Policies decide where a guardrail applies

Registering a guardrail does nothing to live traffic; a policy binds it. Path: AI Gateway → Policies → Guardrails → Add Rule, with four sections — When Request Goes To (models, MCP servers, optionally specific tools), From Subjects (users, teams, virtual accounts), With Metadata and Apply on Hooks.

All rules are evaluated on every request and matching ones are unioned per hook, so a permissive baseline and a strict production rule both apply. The X-TFY-GUARDRAILS header is the one bypass — it skips policies entirely, so treat it as privileged.

One scope limit as of September 2026: policy subjects are users, teams and virtual accounts — not agents. Policies keyed on agent identity, chain-aware conditions using delegation context, and hooks on the Agent Gateway itself are all documented as coming soon.

Metrics turn enforcement into evidence

The Metrics Dashboard’s guardrails view counts Requests evaluated, Mutated and Flagged (blocked), and charts requests per second by result — allowed, blocked, mutated, audit_mode_blocked — block and mutate rate per guardrail, and latency at P50 / P75 / P90 / P99, pivotable by Guardrails, Users, Virtual Accounts or Teams.

audit_mode_blocked is what makes an Audit-first rollout work: it counts what would have been blocked, so you can size the blast radius before enforcing.

A worked example: hardening an agent after a campaign

A campaign against a customer-support agent returns three findings: an indirect injection in a knowledge-base article makes the agent call an internal tool it should not; the agent echoes a customer phone number into a summary; one crafted question drives a twelve-tool loop.

Map findings to hooks. The documented baseline for agent traffic is the starting shape: PII redaction (Mutate) and prompt-injection detection (Validate) on input; secrets detection on output; SQL sanitizer and code-safety linter pre-tool; PII and secrets redaction post-tool. For the injection finding the load-bearing hook is post-tool, where the poisoned article enters the context.

Test in the Playground first. All four hooks are testable at AI Gateway → Playground; the Code button generates a snippet with X-TFY-GUARDRAILS already populated.

Bind a policy in Audit, scoped to this agent’s models and MCP servers, and leave it a week — a detector firing on 8% of legitimate tickets is one you cannot enforce yet. Then promote to Enforce But Ignore On Error and set a Custom Error Message, behind the Hide advanced fields toggle, supporting {{guardrail_message}} and {{failed_guardrails}}.

Verify in traces. Every guardrail is its own span in Monitor → Request Traces, with execution time, result, scope and mutations. Blocks return HTTP 400 with error.type = "guardrail_checks_failed".

Close the sponge finding somewhere else. The twelve-tool loop is a rate-limit and budget problem. No guardrail saves you from a loop that is well-formed on every iteration.

Ready to put enforcement behind your red-team findings?

Register a guardrail group, bind one policy, and watch it in traces.

Start free

Gotchas worth knowing

Rail ordering can convert a block into an allow. DeepKeep applies first-listed precedence among rails that fired — not the most severe action. Order PII Detector before Adversarial Prompt Defense and a jailbreak that also contains PII gets redacted and allowed rather than blocked: the redaction wins because it was listed first, and the jailbreak sails through with a phone number starred out. The documented fix is the recommended order — Credentials Leakage → PII → Adversarial → Toxic — and better still the two-firewall split: Adversarial Prompt Defense on the pre firewall only, credentials and PII on the post.

Output guardrails are skipped on streaming responses. With stream: true they do not run at all, because evaluating a response needs the complete text. Input guardrails always run. Since nearly every chat surface streams by default, this disables output-side defences where users see them.

System prompts are never inspected. The gateway strips them before sending content to any guardrail; CrowdStrike AIDR is the one documented exception, and it never modifies them. “We scan the whole prompt” is not a claim you can make.

If you write your own detector, return 2xx. The custom-guardrail contract is explicit: HTTP 2xx means the guardrail ran to completion, with the outcome in the body as verdict: false. Signalling a block with a 400 is the documented failure mode — the gateway may read it as an error and pass the request.

Related reading

• OWASP LLM Top 10 — the taxonomy your findings map to
• Prompt injection defense in the LLM gateway — the highest-yield attack class
• TrueFoundry AI Gateway guardrails explained — the configuration walkthrough
• Benchmarking LLM guardrail providers — how the detectors compare
• Agentic AI security — the wider control model

Conclusion

Two halves that need each other. Campaigns find: episodic, expensive, the only way to discover a failure class nobody wrote a detector for. Enforcement holds: continuous, cheap per request, worthless against an attack nobody described to it. A programme with only the first produces reports; one with only the second is confident about the wrong things.

TrueFoundry sits squarely in the second half, and we would rather say so than imply otherwise. The gateway gives you one place to enforce the findings across every model, team and MCP tool call. Where it stops is at generating the attacks. If a vendor says their firewall makes red teaming unnecessary, ask which attack classes their detectors were trained on, and when — then run the campaign anyway, and enforce what it finds.

Put runtime enforcement behind your red-team findings

Start free