Agent Gatewayシリーズ（全7回中7回目） | Agent DevOps：CI/CD、評価、カナリアデプロイ

In the world of microservices, we have perfected the art of CI/CD. Unit tests are deterministic: assert(2 + 2 == 4). If the tests pass, the code is safe to deploy.

In the world of Agentic AI, "Unit Tests" don't exist in the same way.

• Code: 2 + 2 is always 4.
• Agent: "Be helpful" might mean "Write a poem" today and "Delete the database" tomorrow, depending on the model version or a slight change in the prompt.

You cannot simply deploy an agent because the code compiles. The prompt is a Hyperparameter of Behavior. A minor tweak to the system prompt ("Be more concise") can cause a massive regression in reasoning capability ("The agent stopped checking for errors because it wanted to be concise").

To solve this, the TrueFoundry Agent Gateway supports Agent DevOps—a specialized lifecycle management layer that brings "Shadow Mode," "Online Evals," and "Canary Rollouts" to the cognitive stack.

The Problem: The "Tone Shift" Incident

Let’s look at a concrete example of why standard CI/CD fails for agents.

The Scenario: You have a Customer Support Agent in production. It’s polite and helpful. The Product Manager wants it to be more efficient. The Change: You update the System Prompt from "You are a helpful assistant" to "You are a concise, direct assistant. Do not waste words."

The Standard Deployment:

1. You commit the prompt change to Git.
2. The pipeline builds the container.
3. kubectl apply updates the service.

The Catastrophe: The agent interprets "direct" as "rude."

• Customer: "My package is lost, I'm so worried!"
• Agent (v2): "Tracking says delivered. Check your porch. Goodbye."

The Customer Satisfaction (CSAT) score crashes. You have tainted your brand because you treated a cognitive change like a code change.

The Solution: Shadow Mode (Dark Launching)

The TrueFoundry Gateway supports Traffic Mirroring (Shadow Mode). Instead of replacing v1 with v2, we deploy v2 alongside v1.

• Real User Traffic: Goes to v1. The user sees v1's response.
• Shadow Traffic: The Gateway duplicates the request to v2 invisibly.

The Gateway then compares the outputs asynchronously. You can run an "Auto-Eval" (using a Judge Model) to score the difference.

• v1 Output: "I understand your concern. Let me check the tracking..." (Empathy Score: 9/10)
• v2 Output: "Tracking says delivered." (Empathy Score: 2/10)

The dashboard alerts you: "v2 Empathy Regression Detected." You revert the deployment before a single customer sees the rude message.

‍

The "Eval" Gate: CI for Cognition

Before an agent even reaches Shadow Mode, it must pass the Evaluation Pipeline. Just as you run pytest for code, you must run deepeval or ragas for cognition.

The TrueFoundry Registry treats "Evaluation Datasets" as first-class citizens.

• Dataset: 500 historic customer queries + "Golden Answers."
• Metric: "Faithfulness," "Answer Relevancy," "Tool Usage Accuracy."

When you push a Pull Request, the CI system spins up the agent and runs the 500 queries. Pass Criteria:

• Faithfulness > 0.9
• Latency < 2s
• Regression: Score cannot be >5% lower than main branch.

If the "Concise Prompt" causes the "Faithfulness" score to drop by 10%, the build fails. "Merge Blocked: Agent creates hallucinations."

‍

Canary Rollouts: Progressive Trust

Once the agent passes CI and Shadow Mode, you are ready for the real world. But you don't flip the switch to 100%. You use Canary Routing.

The Gateway creates a "Virtual Service" that splits traffic based on weights.

• Phase 1: 1% of users get v2. (Internal Employees or Beta users).
• Monitor: Check "Feedback Thumbs Up/Down" rate.
• Phase 2: 10% of users.
• Monitor: Check "Tool Error Rate" (Did the new prompt break the JSON output?).
• Phase 3: 50%... 100%.

The Gateway automates this. If the "Error Rate" spikes at the 10% stage, the Gateway can help  automatically roll back to v1 and pages the on-call engineer.

‍

Fig 1: A Canary Rollout Example

Prompt Versioning vs. Code Versioning

Agent DevOpsにおける主要な課題の1つは、プロンプトとコードが異なる場所に存在することが多い点です。

• コード： main.py (Git)
• プロンプト： system_prompt.txt (多くの場合、DBまたはUI内)

TrueFoundryの Agent Registry は 不変のアーティファクトを強制します。デプロイ時には、以下をバンドルします。アーティファクトID = ハッシュ(コード + プロンプト + モデル設定 + 依存関係)

本番環境でv1のプロンプトを変更することはできません。v1.1を新たに作成する必要があります。この厳格なバージョン管理により、 再現性が確保されます。インシデントが発生した場合、どのコードとプロンプトの組み合わせが原因であったかを正確に把握できます。

まとめ

Agent DevOpsは、確率的ソフトウェアにエンジニアリングの厳密さを適用する規律です。「感覚ベースのデプロイ」（速くなった気がする）から「メトリクスベースのデプロイ」（シャドウモードで5%の精度向上を確認）へと移行することで、TrueFoundryは、ユーザーの信頼を損なうことなく、企業がプロンプトを迅速に革新することを可能にします。

‍