Blank white background with no objects or features visible.

EMEAの皆様へ。TrueFoundryは、ヨーロッパ、中東、アフリカ全域で事業を拡大しています。詳細はこちら →

At a glance

  • 1 billion+ prescriptions filled each year
  • 400 million+ patient calls a year fielded by agentic AI
  • 6 trillion tokens processed each month on CVS's own infrastructure, using open-source models
  • 100+ agentic workflows running in production
  • 9 months to 3 weeks — the drop in time to build a new IVR application

About CVS Health

CVS Pharmacy fills more than a billion prescriptions a year and keeps roughly 85% of the U.S. population within about 16 km of a store. That footprint generates more than 400 million inbound patient calls a year, now handled by agentic AI, and more than six trillion tokens of model traffic a month running on CVS’s own infrastructure.

Yogesh Bhagat leads the agentic AI effort behind that scale — in his words, the team “that keeps your prescription bottles filled.” More than 100 production workflows, from prescription editing to patient conversations, run on the platform his team built. He is now part of Health 100, CVS Health’s initiative to build a single front door to healthcare.

The challenge: safety-critical AI, at national scale

CVS started its agentic journey in 2024, building its first IVR call flow with agentic AI at a time when most of the industry was still talking about prompts, not agents. The goal was to replace a rigid “press one, press two” phone tree — the kind where callers forget which key they needed by the time they reach their question — with a system patients could simply talk to.

Three problems stood in the way, and each was a blocker on its own:

  1. Cost. The early proof of concept, migrated across the full IVR footprint, carried a nine-figure annual cost. “Nobody would have allowed us to go live,” Bhagat said.
  2. Latency. Early responses took 50 to 90 seconds. No patient waits that long for an answer.
  3. Reliability. This is a system patients depend on every day. Frontier models can degrade or go down, and traffic can spike — yet the application cannot break, because there is a person on the other end waiting on a prescription.

The solution: open-source models, owned GPUs, and a gateway

CVS solved cost and latency together, then made a contrarian infrastructure bet that compounded both.

First, the team moved off frontier APIs and fine-tuned open-source models to match frontier quality on its specific use cases. That cut response times from 50–90 seconds down to three to four seconds and took the first large bite out of cost.

Then, in 2024, came the contrarian call: CVS stood up its own active-active data center and ran the models on its own GPUs, estimating a further five-to-six-fold reduction in cost. Committing to open-source models was one bet; buying the hardware to run them was a much bigger one — a wager that open models would keep closing the gap with frontier models and could be tuned to win on CVS’s own workloads. Two years on, that bet is paying off in production.

Reliability came from layering, not from any single model. The team routes each request to the cheapest component that can answer it well — smaller models, classic ML models, caching, and deterministic rules — and defines fallbacks so that if one agent stops responding, another model takes over. As Bhagat put it, reliability was “not just one layer, it was three to four different layers working together.”

Guardrails and governance: earning the right to be autonomous

In healthcare, a wrong answer is not a bad user experience — it touches a patient’s health. CVS built its guardrails around one governing principle:

Nobody earns a right to be autonomous. You earn it over time.

Autonomy is graduated, not granted. Smaller models learn on the job; larger “judge” models grade their output in a feedback loop; and a human stays in the loop until the system has proven itself deterministic enough to be trusted. Only then does a workflow graduate to acting on its own.

On top of that sit hard rules and platform-level guardrails. CVS’s agents never give clinical advice — “that’s period, no more questions asked.” The team guards against bias across factors like age, defends against prompt injection, and ensures no PII ever reaches a frontier model or surfaces in a response.

To enforce all of this consistently, CVS put a “traffic cop” in front of every agent — an AI gateway. Just as teams once standardized API traffic on an Apigee or a Kong, Bhagat argued, a fleet of AI applications needs the same kind of control point, so that policy is applied to every call that reaches an agent. “This is not designed by process,” he said. “This is designed by rigor.”

The direction has since been echoed by the wider industry: analysts now recommend that enterprises invest in an AI gateway for exactly these reasons — more control and more confidence in the AI systems they ship.

Driving platform adoption

Building a platform is one problem; getting the rest of the enterprise to adopt it is another. Teams often see a central platform as a rate-limiter that takes away their control, rather than an accelerant. CVS solved that on two fronts.

It made governance invisible but unavoidable. Instead of every application team queuing for a governance body — presenting an app, waiting weeks, being sent back with five more questions — CVS worked with its governance body once to encode every policy into the platform. After that, the governance body only has to certify the platform. Application teams don’t sit in review meetings; the guardrails are simply enforced on every request, with no bypass. Engineers get the autonomy they wanted, and the organization gets control it can rely on.

It made engineers co-owners. Rather than let the platform become “a cool product built by a few cool engineers,” the team open-sourced it internally. Any engineer can contribute; principal engineers review and merge. “There is no cool dudes here,” Bhagat said — everyone building on the platform feels they are building it.

Results: from nine months to three weeks

The first IVR was a hard, nine-month build that went through three rewrites. But it left behind a gateway and a developer platform — and every application after it got dramatically faster to ship.

IVR application Time to build
First 9 months (3 rewrites)
Second 4 months
Third 6 weeks
Latest 3 weeks

Prescription-editing agents are now built in days rather than weeks, and changes can be tested and shipped in hours. One hard-won IVR turned into a platform that now carries more than 100 production-grade applications. “This is no longer a fantasy,” Bhagat said. “It’s a real, simple software-shipping problem for us now.”

The platform is what made that possible. Rather than train every engineer in agent registries, prompt versioning, gateways, and how to test non-deterministic applications, CVS invested in a developer platform that abstracts those complexities — with observability deep enough to trace a single question across the five LLMs and sixty deterministic rules that might answer it. The gateway handles what belongs at the gateway; the platform handles agent creation, multi-tenancy, and observability. Teams register model capacity against a project and org structure, every request flows through the gateway, and usage, TPS, and token counts map cleanly to the right business unit for chargeback.

What's next: agent harness, MCP, and Health 100

The shape of the problem keeps moving. Two years ago, the drag-and-drop agent builder was the coolest thing the team had — today Bhagat calls it close to obsolete. Engineers now write agents through an “agent harness”: give it a prompt, a skill set, and a set of tools, and the agent is registered in the agent registry with guardrails already built in.

From there, the team is focused on the security of those agents and on new standards. A2A and MCP have become the way components talk to each other, so the questions now are how to govern MCP as a tool and how to expose different data assets safely as MCP tools — again split cleanly between what belongs in the gateway and what belongs in the platform.

The biggest bet is Health 100, CVS Health’s initiative to build a single front door to healthcare. Patients today navigate seven or eight entities that barely talk to each other — doctors, insurers, prior authorization, pharmacies, PBMs — while someone they love is in pain. Health 100 aims to bring those assets together in one place: booking an in- or out-of-network appointment, viewing health records, asking health questions against something more reliable than a general chatbot, managing prescriptions, and seeing real cost and affordability — all built on the same gateway-plus-platform foundation.

Key takeaways for teams building agentic AI

  • Define ROI and KPIs before you build, not after. CVS sized the cost and the metrics up front, launched small, and measured against clear KPIs — ready to scrap a project or iterate. “Fail fast” still applies.
  • Frontier models are not the only answer. Fine-tuned open-source models, run on your own infrastructure, can match frontier quality and win decisively on cost. Trust your subject-matter experts to make them work.
  • Reliability is a system, not a model. Route to the cheapest component that can answer well — smaller models, caching, deterministic rules — and define fallbacks across three or four layers.
  • No workflow is autonomous by default. Let agents earn autonomy through judge-model feedback loops and human-in-the-loop review before they act alone.
  • Enforce governance in the platform, not in meetings. Encode policy once so guardrails apply to every request with no bypass — engineers get autonomy, the enterprise keeps control.
  • The two must-haves: an AI gateway and your own developer platform. Build or buy, but the combination is non-negotiable. Early slowness is fine; it is what makes the acceleration possible later.

What an AI gateway actually is. As Nikunj Bajaj framed it, the category unifies what used to be separate control points: every organizational token flows through one common pipe, whether the call goes to a model, an MCP, or another agent. Model gateway, MCP gateway, and agent gateway combine into a single AI gateway — with shared platform capabilities like observability and guardrails cutting across all three.

The fastest way to build, govern and scale your AI

MLパイプラインを初日から運用

パイプライン