TrueFoundry is recognized in the 2025 Gartner® Market Guide for AI Gateways! Read the full report

Case Study

Summary

Innovaccer is a healthcare intelligence cloud operating in highly regulated environments concerning protected health information (PHI). Innovaccer uses AI to improve clinical efficiency, care management, and operational decision-making across its healthcare platform. AI powers use cases such as clinical summarization, care gap identification, risk stratification, quality and coding support, and natural-language insights over healthcare data, while operating in PHI-heavy, regulated environments.

In this journey of GenAI adoption across clinical and operational applications, Innovaccer needed a centralized way to govern, observe, and scale usage, without fragmenting access or compromising compliance. This surfaced challenges around PII-safe observability, auditability, model access control, and cost governance across multiple LLMs and embedding models.

By partnering with TrueFoundry, Innovaccer standardized all GenAI traffic through TrueFoundry’s AI Gateway, establishing a unified control plane for healthcare-grade governance at scale. Today, Innovaccer routes ~17 million inference requests per month, processing ~34 billion input tokens and 3.4 billion output tokens across 40+ models—including OpenAI, AWS Bedrock, Gemini, and self-hosted deployments—powering 25+ healthcare applications. With centralized logging, PII redaction, cost controls, and policy enforcement built in by default, Innovaccer has embedded GenAI deeply into production workflows while maintaining enterprise-grade observability, compliance, and governance across all major LLM hyperscalers.

A focused engagement benchmarked TrueFoundry against alternate model hosting platforms and showed autoscaling time reduced from ~8 minutes to ~5 minutes (a 37.5% decrease) in addition to faster infrastructure setup, richer observability, and better cost characteristics.

About Innovaccer

Innovaccer activates the flow of healthcare data, empowering providers, payers, and government organizations to deliver intelligent and connected experiences that advance health outcomes. The Healthcare Intelligence Cloud equips every stakeholder in the patient journey to turn fragmented data into proactive, coordinated actions that elevate the quality of care and drive operational performance. Leading healthcare organizations like Orlando Health, Adventist Healthcare, and Banner Health trust Innovaccer to integrate a system of intelligence into their existing infrastructure, extending the human touch in healthcare. Innovaccer manages patient data of millions of patients with billions of data points across them.

Context

“Powering Innovaccer’s AI/ML Innovation” is not just a tagline, it reflects how Innovaccer is scaling AI across healthcare organizations, with TrueFoundry as the enabling infrastructure partner. Innovaccer is automating knowledge work across RCM, patient access, provider copilots, clinical coding, and data mapping. To support this at scale, Innovaccer follows a multi-model strategy spanning Azure, AWS Bedrock, OpenAI, and self-hosted models — with TrueFoundry providing the governance, orchestration, and deployment backbone behind it.

To sustain this growth, Innovaccer needed:

  • A single AI entry point for experimentation and production.
  • Tight observability on token usage, performance, and cost.
  • Self-serve model hosting with strong autoscaling and no DevOps bottleneck.
  • A path to govern PHI/PII-sensitive workloads and future agentic use cases.

The Challenge

Prior to centralizing on TrueFoundry, Innovaccer’s generative AI infrastructure utilized direct,
point-to-point connections between production apps and various providers like OpenAI, Azure,
and Bedrock. 

While functional, this fragmented approach lacked the unified gateway necessary for the high-level traceability and fiscal oversight essential in a healthcare environment. Consolidating these workflows was a strategic move to ensure the reliability required for enterprise-grade
clinical operations.

The Evolution of Healthcare-Grade GenAI

By centralizing its GenAI infrastructure through TrueFoundry, Innovaccer moved from a fragmented model to a unified AI backbone designed for the complexities of healthcare.

  • Reliability & Patient-Centric Workflows: By implementing centralized fallback mechanisms and traffic control, we ensure that critical admin workflows—which providers and patients depend on—remain resilient and performant even during provider outages.
  • Traceability & Clinical Compliance: A centralized layer provides the rigorous audit trails and traceability essential for healthcare data governance. Innovaccer can now monitor how models interact with sensitive data, ensuring every output is accountable.
  • Scale & Cost Stewardship: Managing cost-to-serve is vital for healthcare efficiency. This centralized framework allows Innovaccer to measure and optimize cost across the platform, ensuring that scaling AI does not lead to unpredictable administrative overhead.
  • Developer Velocity through Configuration: Using TrueFoundry’s orchestration layer, Innovaccer decoupled application logic from the underlying model and accelerated value delivery. Development teams can now test and switch between various foundation models purely through configuration, requiring zero code changes. This "pluggable" architecture allows us to adopt the latest clinical LLMs the moment they are available.

For care teams, physicians, and patients who rely on these applications for timely insights and decision support, this created potential risks around consistency of experience, service availability during peak clinical moments, and confidence in how sensitive health data was handled. 

Additionally, TrueFoundry compared its deployment and autoscaling experience with alternate model hosting platforms on popular cloud vendors. They required manual configuration for invocation counts, relied on log-based tracking via CloudWatch to understand autoscaling timing, and added ~25% markup on instance pricing. Visibility into pod-level events and autoscaling behavior was limited, making tuning slower and less transparent.


Solution: TrueFoundry as the Central AI Orchestration Platform

TrueFoundry was adopted as the DevX and orchestration layer for both LLM traffic (AI Gateway) and AI Deployment Platform.

1. AI Gateway: A Single Control Plane for LLMs

On average in a month, the AI Gateway serves:

  • ~17 million inference requests.
  • ~34 billion input tokens and 3.4 billion output tokens.
  • 25+ healthcare applications onboarded.
  • ~40 different models, spanning OpenAI, AWS Bedrock, Azure, Gemini, and self-hosted Llama.
The Gateway provides:
  • Central routing across providers and models.
  • Unified metrics such as time to first token and inter-token latency.
  • Token and cost tracking broken down by teams, users, environments, and models.
  • OpenTelemetry-compatible metrics that flow directly into Innovaccer’s existing Grafana stack for dashboards and alerts.
This centralized AI Gateway turned Innovaccer’s LLM usage from fragmented per-app integrations into a single, observable control plane.

2. Reliability: Protecting Clinical and Care Delivery Workflows with Centralized Fallbacks

Innovaccer uses GenAI across care management, clinical intelligence, and operational workflows that support physicians, care managers, and population health teams. These applications surface patient summaries, risk insights, care gaps, and next-best actions at the point of decision-making

On June 10, when OpenAI experienced elevated error rates, Innovaccer’s AI Gateway automatically rerouted traffic to Azure based on preconfigured fallback rules. This ensured that care teams continued to receive timely insights without disruption, even as underlying model providers experienced instability.

By configuring failover centrally at the AI Gateway rather than within individual applications, Innovaccer ensured consistent reliability across its healthcare platform. This approach reduced variability in clinician and care team experience, while allowing product teams to focus on improving care workflows instead of managing provider-specific failure scenarios.

3. Fast-Track Access to Advanced AI Capabilities

TrueFoundry also accelerated access to newer OpenAI APIs through the Gateway:

  • Responses API: enabling tool-use workflows such as internet search.
  • Codex integration: unlocking code-generation capabilities.
  • OpenAI Batch: supporting asynchronous, high-volume inference workflows.
Instead of each Innovaccer team implementing these capabilities separately, they are exposed centrally through the AI Gateway, enabling consistent governance and monitoring.

4. Faster clinical intelligence workflows with latency-aware routing

Innovaccer’s GenAI is used in care management and clinical intelligence workflows where response time directly affects usability for physicians and care teams. To support this, TrueFoundry implemented latency-aware routing at the AI Gateway, dynamically directing live traffic to the fastest available model endpoint without requiring application changes.
In addition, centralized prompt management allowed Innovaccer teams to safely version and roll out prompt updates across applications, ensuring consistent and reliable AI behavior in clinical and operational workflows.

5. Data Sovereignty and Regulated Deployments (GovCloud)

For compliance-sensitive healthcare use cases, Innovaccer required GenAI infrastructure that could operate entirely within regulated, sovereign environments. TrueFoundry was deployed in AWS GovCloud (US), enabling Innovaccer to run GenAI workloads in regions designed for strict data residency, access control, and audit requirements.

This allows Innovaccer to use the same AI Gateway and orchestration layer for HIPAA-aligned, PHI-heavy workloads, while ensuring sensitive health data remains within approved sovereign boundaries and compliance frameworks.

Impact on Infrastructure Response & Scaling Orchestration

1. Accelerated Service Readiness & Latency Reduction

The implementation of TrueFoundry (TF) introduced a more deterministic lifecycle for model deployment. In performance benchmarking, the "trigger-to-operational" timeline was reduced to a consistent ~5-minute window, representing a 37.5% optimization over previous infrastructure baselines.

  • Provisioning Velocity: The interval from pod nomination to container initialization was stabilized at approximately 2 minutes.
  • Integrated Telemetry: Unlike legacy systems where scaling events must be inferred from external log streams, TF provides native, platform-level visibility into the deployment state. This eliminates the "observability gap" during critical scaling windows.

2. Request-Centric Elasticity (RPS-Based Scaling)

Standard resource-based scaling (CPU/RAM) often lags behind the bursty nature of GenAI traffic. Innovaccer adopted Request-Per-Second-based scaling through TrueFoundry as the primary scaling metric to better handle bursty GenAI traffic

  • Dynamic Load Handling: By scaling on RPS, the infrastructure preemptively adjusts to traffic spikes before compute saturation occurs, ensuring consistent API response times for provider-facing copilots.
  • Hybrid Scaling Logic: TrueFoundry’s scaling system integrates RPS-based triggers with time-based heuristics. This allows for "warm-up" periods during peak clinical hours, ensuring high availability without the fiscal waste of 24/7 over-provisioning.

3. Unified Governance and Control Plane

By consolidating GenAI traffic onto TrueFoundry’s centralized gateway, Innovaccer established the technical “equilibrium” required for enterprise healthcare operations:

  • Programmatic Traceability: Scaling behavior and performance metrics are accessible via a unified API and UI, allowing for automated auditing of system health.
  • Fiscal Oversight: Centralized management enables granular cost-tracking across disparate model providers, ensuring that administrative and clinical workflows remain within budgetary guardrails without manual intervention.

4. Platform Value Observed

The partnership highlighted several advantages of TrueFoundry’s Kubernetes-based platform:

  • Quick infrastructure setup: Azure control and compute plane setup was completed within a day.
  • Developer experience: The data scientist leading the engagement quickly learned the platform and independently executed workflows like deployment and autoscaling. Features such as file-system versioning, model caching, runtime visualizations during builds, and RPS-based autoscaling were called out as standouts.
  • Better observability: TrueFoundry exposes logs, metrics, and Kubernetes events directly, providing deeper debuggability compared with alternate model hosting platforms’ more opaque managed experience.
  • Fractional GPUs and spot instances: The platform supports fractional GPU allocation and spot instances across workflows, adding more levers for cost optimization.
  • Cost model: While SageMaker adds ~25% markup on instance pricing, TrueFoundry uses Kubernetes atop raw instances, allowing it to pass infrastructure savings to users. The document notes that customers have seen at least 30% cost savings relative to SageMaker, characterizing the platform’s potential cost advantage.

Outcomes So Far

From the combined AI Gateway and DLOps initiatives, Innovaccer has achieved:
  • Production-scale GenAI across the healthcare platform: ~17 million monthly inference requests and 37+ billion tokens (~34B input, 3.4B output) routed through a single AI Gateway spanning 40+ models and 25+ healthcare applications. This scale reflects GenAI embedded into core workflows such as clinical summarization, care gap identification, risk stratification, coding support, and operational intelligence — not isolated pilots.

  • Healthcare-grade observability and cost governance: All LLM traffic now flows through a unified control plane with token usage, latency (time-to-first-token, inter-token latency), and cost metrics integrated directly into Innovaccer’s Grafana stack. This enables centralized oversight across teams, environments, and model providers in PHI-heavy, regulated environments.

  • Resilience during provider instability: During elevated OpenAI error rates, traffic was automatically rerouted to Azure via preconfigured fallback rules, maintaining continuity for dependent healthcare applications without requiring changes at the application layer.

  • Faster and more transparent autoscaling for ML workloads: Benchmarking against alternate model hosting platforms showed autoscaling trigger-to-operational time reduced from ~8 minutes to ~5 minutes (37.5% faster), with deeper platform-level visibility into scaling events and deployment states.

  • Regulated deployment readiness: TrueFoundry deployed in AWS GovCloud enables Innovaccer to operate GenAI workloads in compliance-sensitive, sovereign environments while using the same governance and orchestration framework.