Export TrueFoundry AI Gateway traces to Opik with OpenTelemetry

When an LLM call is slow, expensive, or wrong, you want to see the whole request: the prompt, the model, the tokens, the tool calls, the latency. If your traces live in one tool and your evaluation lives in another, that picture is split in two. This guide shows how to send TrueFoundry AI Gateway traces to Comet Opik over OpenTelemetry, so every request the gateway handles shows up in Opik as a full trace, with no changes to your application code.

Why export gateway traces to Opik

The TrueFoundry AI Gateway already sits in front of every model call, routing 1,000+ LLMs through one OpenAI-compatible API at roughly 3-4 ms of overhead and 350+ RPS on a single vCPU. Because it is on the path for every request, it is the natural place to emit traces from. The gateway is OpenTelemetry-compliant out of the box, so exporting to another backend is configuration, not engineering.

Opik is an LLM observability and evaluation platform built by Comet. It ingests OpenTelemetry spans and renders them as LLM traces, with per-call inputs, outputs, token usage, latency, and cost, and it lets you run evaluations against that traffic. Pointing the gateway at Opik means your production requests become traces you can inspect and score, in the same tool. Traces still land in TrueFoundry by default; the export mirrors them to Opik.

Here is how to set it up.

Prerequisites

• A TrueFoundry account with the AI Gateway configured and traffic flowing through it.
• A Comet account for Opik (cloud, self-hosted, or enterprise), with your API key, workspace name, and a target project name.
• Access to the AI Gateway settings in the TrueFoundry dashboard.

Step-by-step integration guide

Opik ingests traces only, and it accepts OTLP over HTTP, so the setup is one exporter with three headers. There is no code to write.

Step 1: Get your Opik key and workspace

In Opik, open your account settings and copy your API key. Note your workspace name from the workspace switcher, and decide the project name you want traces to land in. The project is created on first ingest if it does not exist.

Step 2: Configure the OTEL traces exporter

In the dashboard, go to AI Gateway, then Settings, then OTEL Config. Enable the traces exporter and fill in the Opik endpoint and headers.

To export traces to Opik, configure your OpenTelemetry exporter to use the HTTP protocol with Protobuf (Proto) encoding, pointing to the endpoint https://www.comet.com/opik/api/v1/private/otel/v1/traces. Authentication and routing are handled through three request headers: the Authorization header must contain your Opik API key as a raw value, note that no Bearer prefix should be included, while the Comet-Workspace and projectName headers should be set to your workspace name and project name, respectively. These headers ensure that incoming traces are authenticated and logged to the correct workspace and project within Opik.

Leave the metrics exporter off. Opik ingests traces, not OTLP metrics. Save, and new requests start mirroring to Opik.

Step 3: Verify the traces land

Send a request through the gateway, then open the Traces view in Opik for your project. You should see spans from the gateway with inputs, outputs, token counts, and latency populated. A model call shows up as a chat-completion trace; agent runs show the nested tree of model and tool-call spans.

What you unlock: full LLM observability in Opik

Distributed tracing across the whole request

An agent run comes through as one trace with the model calls and tool calls nested underneath, so you can follow a multi-step request end to end rather than reading disconnected log lines.

Evaluation on real production traffic

Because your live requests are now traced in Opik, you can build datasets from them and run evaluations against the traffic you actually serve, instead of a synthetic sample.

Cost and latency visibility

Each trace carries token usage, latency, and cost, so you can find the slow or expensive calls without stitching data together from separate tools.

No application code change

The export is set once in the gateway. Every service that routes through the gateway gets traced, and you do not add an SDK or edit any application to get there.

FAQ

Q: What is LLM observability through an AI gateway? A: It means every model call your applications make is traced at the gateway, with the prompt, response, tokens, latency, and cost captured for each request. On the TrueFoundry AI Gateway this is OpenTelemetry-based, so you can export those traces to a backend like Comet Opik without instrumenting your code.

Q: Does exporting to Opik stop traces from going to TrueFoundry? A: No. Traces are still stored in TrueFoundry by default. The OTEL export mirrors them to Opik in addition, so you keep both.

Q: Can I export metrics to Opik as well as traces? A: No. Opik ingests traces only, so you configure the traces exporter and leave metrics off. For gateway metrics you would use a metrics backend such as Grafana, Datadog, or Prometheus.

Q: How many LLMs does TrueFoundry support? A: 1,000+ LLMs through a single OpenAI-compatible API. You switch models by changing the model name in the request, with the same URL and credentials.

Q: Does it integrate with my existing observability stack? A: Yes. The gateway is OpenTelemetry-compliant and plugs into Grafana, Datadog, Prometheus, or, as here, Comet Opik, tracing each request from prompt to tool and model execution.

Related reading

• What is an LLM Gateway? - the architecture behind this integration
• LLM Cost Tracking Solution - going deeper on spend visibility
• Observability in AI Gateway - how gateway-level tracing fits the broader observability story

Conclusion

Observability should not require re-instrumenting every service. With the TrueFoundry AI Gateway exporting OpenTelemetry traces to Comet Opik, every request the gateway handles becomes a full trace you can inspect and evaluate, set up once and applied everywhere. Start on the TrueFoundry AI Gateway.