Every Request, Evaluated: Confident AI and TrueFoundry AI Gateway

Most AI teams can say how many tokens their applications consumed last week. Far fewer can say how many of those answers were actually good. Tracing records what happened; evaluation judges whether it was right. In an organisation where dozens of services call models through different SDKs and agent frameworks, wiring both into every codebase is the project that never quite finishes.

The integration between TrueFoundry AI Gateway and Confident AI moves that work to the infrastructure layer. Every request that passes through the gateway can arrive in Confident AI as a trace, ready to be scored by evaluation metrics, without a line of tracing code in the application.

The Power of TrueFoundry AI Gateway

TrueFoundry AI Gateway sits between applications and the models they call. It exposes one OpenAI-compatible API across providers and handles authentication, routing across virtual models, rate limiting and guardrails in a single execution layer. Configuration is held in memory on stateless gateway pods, so the gateway adds roughly 3 ms of latency while doing all of this.

The property that matters here is that the gateway is OpenTelemetry-compliant. Each request produces spans for the inbound handler, authentication, model resolution, the outbound provider call and the streaming response. Those spans carry standard gen_ai.* attributes such as token counts and finish reasons, along with tfy.input and tfy.output for the request and response payloads. Traces are published asynchronously after the request completes and can be exported to any OTLP endpoint, in addition to TrueFoundry's own trace storage.

Confident AI: Evaluation That Runs on Production Traffic

Confident AI is an LLM evaluation and observability platform from the team behind the open-source DeepEval framework. Its tracing is OpenTelemetry-native: every trace enters through the OTEL server, an OTLP/HTTP ingestion endpoint that authenticates each batch with a project API key and reassembles traces from their span IDs. Incoming spans are typed as LLM, tool, agent or retriever spans, with model, token usage and cost filled in from the OpenTelemetry GenAI semantic conventions.

What turns a trace store into a quality system is online evaluation. Teams group metrics such as answer relevancy, faithfulness, toxicity and PII leakage into metric collections, then attach them to live traffic through Evaluation Rules that filter by span type and sample at a chosen rate, for example 5% of production and all of staging. Evaluation runs server-side as traces are ingested, so a quality score sits on the trace next to its latency and cost.

One Pipeline, Full Visibility: Confident AI and TrueFoundry

Each side holds half of the answer. The gateway sees every model call in the organisation, regardless of which language or framework made it, but grading answer quality is not its job. Confident AI can type and score any span it receives, but it only receives what something sends it. Pointing the gateway's OTEL Traces Exporter at Confident AI's OTEL server makes the gateway the single instrumentation point for every application behind it.

Gateway traces leave the request path asynchronously and reach Confident AI's OTEL server over OTLP/HTTP, where they are typed and scored.

Configuration lives on the TrueFoundry side, under AI Gateway → Controls → Settings → OTEL Config. The exporter uses HTTP with JSON encoding and authenticates with the Confident AI project key, so traces land in the project that issued it. Confident AI accepts OTLP over HTTP only, which is why the gateway's gRPC option is not used here.

The exporter needs only four settings:

Traces endpoint: https://otel.confident-ai.com/v1/traces (EU: https://eu.otel.confident-ai.com/v1/traces)

Protocol: HTTP

Encoding: JSON

Headers: Content-Type: application/json and x-confident-api-key: <project API key>

One toggle deserves a deliberate choice. Exclude Request Data decides whether prompts and completions travel with the trace. Left off, tfy.input and tfy.output are exported, giving content-based metrics something to judge. Switched on, those attributes are stripped before export, which suits sensitive workloads that still want latency, token and status data in Confident AI.

How the Evaluation Loop Works

The integration is built so that evaluation never sits between a user and their answer. A single request moves through two distinct phases:

1. An application sends an OpenAI-compatible request to the gateway. The gateway validates the caller, resolves the model and forwards the call to the provider.
2. The response streams back to the application. Nothing in this path waits on Confident AI.
3. Once the request completes, the gateway publishes its trace asynchronously, and the OTEL Traces Exporter posts it to Confident AI's OTEL server over OTLP/HTTP with the x-confident-api-key header.
4. Confident AI authenticates the batch, reads the gen_ai.* attributes and rebuilds the trace with model, token usage and cost populated.
5. Evaluation Rules that match the trace or its spans run their metric collections server-side. Only referenceless metrics run online, meaning metrics that can judge an answer without an expected output to compare against.
6. Scores appear on the trace under Observability → Traces, alongside latency, environment and status.

The request completes before export begins, so evaluation adds no latency to the user-facing call.

Because export happens off the request path, the gateway never fails a request when an external OTEL endpoint is unreachable. The worst case is a missing trace, not a missing answer. And because Evaluation Rules are configured in Confident AI rather than in code, a team can change which metrics run on gateway traffic, or how much of it is sampled, without redeploying anything.

Get Started with Production-Ready Evaluation

Setup takes a Confident AI project API key and a handful of fields in the gateway's OTEL Config. US and EU endpoints are both available, and self-hosted Confident AI deployments expose their own traces endpoint. Send a few requests, then confirm they appear in Confident AI's traces view before adding Evaluation Rules.

Full steps are in the TrueFoundry integration guide. Confident AI's OpenTelemetry docs cover the OTEL server, and its online evaluation guide explains metric collections and Evaluation Rules. Teams that also want user, thread and business context can add Confident AI's confident-trace SDK in code, as described on its TrueFoundry integration page.

Two Vantage Points, One Evaluation Layer

The integration stays clean because execution and evaluation meet at a standard rather than an SDK. The gateway is where traffic converges, so it is the natural place to instrument; Confident AI reads the OpenTelemetry GenAI conventions, so it can type and score spans it never instrumented itself. The two views also complement each other: gateway traces record how each request was resolved and served, which an application's own spans cannot see, while in-code tracing adds the user and conversation context only the application knows. TrueFoundry governs and routes the call, Confident AI judges the answer, and OpenTelemetry carries the evidence between them.