
CVS Pharmacy fills more than a billion prescriptions a year and keeps roughly 85% of the U.S. population within about 16 km of a store. That footprint generates more than 400 million inbound patient calls a year, now handled by agentic AI, and more than six trillion tokens of model traffic a month running on CVS’s own infrastructure.
Yogesh Bhagat leads the agentic AI effort behind that scale — in his words, the team “that keeps your prescription bottles filled.” More than 100 production workflows, from prescription editing to patient conversations, run on the platform his team built. He is now part of Health 100, CVS Health’s initiative to build a single front door to healthcare.
CVS started its agentic journey in 2024, building its first IVR call flow with agentic AI at a time when most of the industry was still talking about prompts, not agents. The goal was to replace a rigid “press one, press two” phone tree — the kind where callers forget which key they needed by the time they reach their question — with a system patients could simply talk to.
Three problems stood in the way, and each was a blocker on its own:
CVS solved cost and latency together, then made a contrarian infrastructure bet that compounded both.
First, the team moved off frontier APIs and fine-tuned open-source models to match frontier quality on its specific use cases. That cut response times from 50–90 seconds down to three to four seconds and took the first large bite out of cost.
Then, in 2024, came the contrarian call: CVS stood up its own active-active data center and ran the models on its own GPUs, estimating a further five-to-six-fold reduction in cost. Committing to open-source models was one bet; buying the hardware to run them was a much bigger one — a wager that open models would keep closing the gap with frontier models and could be tuned to win on CVS’s own workloads. Two years on, that bet is paying off in production.
Reliability came from layering, not from any single model. The team routes each request to the cheapest component that can answer it well — smaller models, classic ML models, caching, and deterministic rules — and defines fallbacks so that if one agent stops responding, another model takes over. As Bhagat put it, reliability was “not just one layer, it was three to four different layers working together.”
In healthcare, a wrong answer is not a bad user experience — it touches a patient’s health. CVS built its guardrails around one governing principle:
Nobody earns a right to be autonomous. You earn it over time.
Autonomy is graduated, not granted. Smaller models learn on the job; larger “judge” models grade their output in a feedback loop; and a human stays in the loop until the system has proven itself deterministic enough to be trusted. Only then does a workflow graduate to acting on its own.
On top of that sit hard rules and platform-level guardrails. CVS’s agents never give clinical advice — “that’s period, no more questions asked.” The team guards against bias across factors like age, defends against prompt injection, and ensures no PII ever reaches a frontier model or surfaces in a response.
To enforce all of this consistently, CVS put a “traffic cop” in front of every agent — an AI gateway. Just as teams once standardized API traffic on an Apigee or a Kong, Bhagat argued, a fleet of AI applications needs the same kind of control point, so that policy is applied to every call that reaches an agent. “This is not designed by process,” he said. “This is designed by rigor.”
The direction has since been echoed by the wider industry: analysts now recommend that enterprises invest in an AI gateway for exactly these reasons — more control and more confidence in the AI systems they ship.
Building a platform is one problem; getting the rest of the enterprise to adopt it is another. Teams often see a central platform as a rate-limiter that takes away their control, rather than an accelerant. CVS solved that on two fronts.
It made governance invisible but unavoidable. Instead of every application team queuing for a governance body — presenting an app, waiting weeks, being sent back with five more questions — CVS worked with its governance body once to encode every policy into the platform. After that, the governance body only has to certify the platform. Application teams don’t sit in review meetings; the guardrails are simply enforced on every request, with no bypass. Engineers get the autonomy they wanted, and the organization gets control it can rely on.
It made engineers co-owners. Rather than let the platform become “a cool product built by a few cool engineers,” the team open-sourced it internally. Any engineer can contribute; principal engineers review and merge. “There is no cool dudes here,” Bhagat said — everyone building on the platform feels they are building it.
The first IVR was a hard, nine-month build that went through three rewrites. But it left behind a gateway and a developer platform — and every application after it got dramatically faster to ship.
Prescription-editing agents are now built in days rather than weeks, and changes can be tested and shipped in hours. One hard-won IVR turned into a platform that now carries more than 100 production-grade applications. “This is no longer a fantasy,” Bhagat said. “It’s a real, simple software-shipping problem for us now.”
The platform is what made that possible. Rather than train every engineer in agent registries, prompt versioning, gateways, and how to test non-deterministic applications, CVS invested in a developer platform that abstracts those complexities — with observability deep enough to trace a single question across the five LLMs and sixty deterministic rules that might answer it. The gateway handles what belongs at the gateway; the platform handles agent creation, multi-tenancy, and observability. Teams register model capacity against a project and org structure, every request flows through the gateway, and usage, TPS, and token counts map cleanly to the right business unit for chargeback.
The shape of the problem keeps moving. Two years ago, the drag-and-drop agent builder was the coolest thing the team had — today Bhagat calls it close to obsolete. Engineers now write agents through an “agent harness”: give it a prompt, a skill set, and a set of tools, and the agent is registered in the agent registry with guardrails already built in.
From there, the team is focused on the security of those agents and on new standards. A2A and MCP have become the way components talk to each other, so the questions now are how to govern MCP as a tool and how to expose different data assets safely as MCP tools — again split cleanly between what belongs in the gateway and what belongs in the platform.
The biggest bet is Health 100, CVS Health’s initiative to build a single front door to healthcare. Patients today navigate seven or eight entities that barely talk to each other — doctors, insurers, prior authorization, pharmacies, PBMs — while someone they love is in pain. Health 100 aims to bring those assets together in one place: booking an in- or out-of-network appointment, viewing health records, asking health questions against something more reliable than a general chatbot, managing prescriptions, and seeing real cost and affordability — all built on the same gateway-plus-platform foundation.
What an AI gateway actually is. As Nikunj Bajaj framed it, the category unifies what used to be separate control points: every organizational token flows through one common pipe, whether the call goes to a model, an MCP, or another agent. Model gateway, MCP gateway, and agent gateway combine into a single AI gateway — with shared platform capabilities like observability and guardrails cutting across all three.
