Exporting TrueFoundry AI Gateway Traces to OpenLIT via OTLP

The TrueFoundry AI Gateway generates OpenTelemetry spans for every LLM request and publishes them asynchronously over NATS. OpenLIT is a self-hosted observability platform that accepts OTLP traces and metrics over HTTP or gRPC and stores them in ClickHouse. Connecting the two requires configuring the gateway's OTEL exporter to point at the OpenLIT collector endpoint. No code changes are required in any application.

This post covers the OTEL trace generation path inside the TrueFoundry AI Gateway and the ClickHouse storage model that OpenLIT uses and the configuration surface for wiring the two systems together. It is not a step by step setup guide. It is an explanation of how the integration works at the architecture level.

How the TrueFoundry AI Gateway Generates and Exports Traces

The TrueFoundry AI Gateway is built on the Hono framework and runs as a stateless pod. A single pod on 1 vCPU and 1 GB RAM handles over 250 requests per second with approximately 3 ms of added latency. That performance profile is only possible because the gateway performs zero external calls in the request path. Authentication runs against cached public keys. Authorization checks run against an in-memory map of users to models. Model routing logic runs entirely in memory.

OTEL trace generation follows this same principle. When a request completes the gateway publishes the span data asynchronously to NATS. The OTEL exporter reads from this async path and forwards the span to the configured external endpoint. The exporter never touches the request path. A slow or unreachable OTEL backend never stalls a request.

The spans the gateway emits carry a specific set of attributes. The tfy.input attribute contains the full request body sent to the LLM. The tfy.output attribute contains the full response. The tfy.input_short_hand attribute contains a condensed summary of the input including boolean flags for whether the request contained files or images or audio. The tfy.span_type attribute identifies the operation type such as ChatCompletion or AgentResponse. Standard gen_ai.* semantic conventions are also included covering token counts and model identifiers and finish reasons.

The gateway also exposes an Exclude Request Data toggle. When enabled the exporter strips tfy.input and tfy.output and tfy.input_short_hand from spans before forwarding them. This is relevant for teams that need to route telemetry to an external platform without sending prompt or response content outside their infrastructure.

Export is additive. Enabling an external OTEL destination does not replace or interrupt TrueFoundry's own internal trace storage. Both paths receive the same spans.

What OpenLIT Does with the Traces

OpenLIT is an OpenTelemetry-native observability platform. It does not require a sidecar or an SDK wrapper or any modification to the application layer. It accepts spans over the standard OTLP protocol on port 4318 for HTTP and port 4317 for gRPC and writes them into ClickHouse.

ClickHouse stores traces in the otel_traces table and metrics in the otel_metrics table. The schema follows the OpenTelemetry data model directly. Every span is stored as a row with a SpanAttributes column of type Map(String, String) that holds all the key-value attributes the gateway emitted. Querying a specific attribute uses the map accessor syntax:

sql

SELECT
   Timestamp,
   SpanAttributes['tfy.span_type'] AS span_type,
   SpanAttributes['tfy.data_routing.destination'] AS destination,
   StatusCode,
   Duration / 1e9 AS duration_sec
FROM openlit.otel_traces
WHERE SpanAttributes['tfy.span_type'] = 'ChatCompletion'
ORDER BY Timestamp DESC
LIMIT 20;

The OpenLIT dashboard reads from these ClickHouse tables and renders request timelines and latency distributions and error rates. Because the storage layer is a standard ClickHouse instance teams can also run arbitrary SQL against it directly for custom reporting or alerting pipelines.

OpenLIT bundles its own OTel Collector alongside the dashboard and the database. The collector is the entry point for all incoming spans and metrics. It receives data over OTLP and writes it to ClickHouse using the ClickHouse exporter. The collector configuration defines separate pipelines for traces and metrics and logs each backed by a batch processor and a memory limiter.

The three components ship as a single Helm chart. The collector and the dashboard run inside the same Kubernetes pod. ClickHouse runs as a separate StatefulSet with a persistent volume claim. An init container in the OpenLIT pod waits for ClickHouse to respond on port 8123 before starting the collector and dashboard processes. This sequencing means the Helm release is self-contained and the deployment order is handled automatically.

Once deployed TrueFoundry shows both pods in a Running state: openlit-0 running the dashboard and OTel Collector and openlit-db-0 running ClickHouse.

The OpenLIT dashboard is accessible at the exposed cluster domain URL. The login page lists the platform's core capabilities including end-to-end tracing and cost and token analytics and LLM-as-a-Judge evaluation.

The Integration Surface

The TrueFoundry AI Gateway connects to OpenLIT through two OTEL exporter configurations: one for traces and one for metrics. Both use HTTP and Proto encoding. No authentication headers are required because the OpenLIT collector endpoint sits inside the same Kubernetes cluster as the gateway. Traffic between the two stays on the internal cluster network.

OTEL Traces Exporter Configuration

FieldValueProtocolHTTPEndpointhttp://openlit.NAMESPACE.svc.cluster.local:4318/v1/tracesEncodingProtoHeadersNone

OTEL Metrics Exporter Configuration

FieldValueProtocolHTTPEndpointhttp://openlit.NAMESPACE.svc.cluster.local:4318/v1/metricsEncodingProtoHeadersNone

The openlit Kubernetes service exposes three ports. Port 3000 serves the dashboard. Port 4317 accepts OTLP over gRPC. Port 4318 accepts OTLP over HTTP. All three are on the same service name and DNS entry. The NATS-based async export path in the gateway means that even if the OpenLIT collector is temporarily unavailable the gateway continues processing requests normally. Spans that cannot be delivered are dropped at the exporter level and logged without surfacing errors to the caller.

The OpenLIT Helm chart is hosted at https://openlit.github.io/helm/ under the chart name openlit. The chart accepts a values block that sets the ClickHouse connection parameters for the dashboard and the collector and the persistence configuration for the database. The dashboard connects to ClickHouse using the INIT_DB_HOST environment variable which should point to the ClickHouse StatefulSet service using its internal DNS name openlit-db.NAMESPACE.svc.cluster.local.

The collector configuration is stored in a ConfigMap named openlit-otel-config. It defines three pipelines. The traces pipeline runs through a memory limiter set to 1500 MiB with a 512 MiB spike limit and a batch processor before forwarding to the ClickHouse exporter. The metrics pipeline follows the same processor chain. The logs pipeline omits the memory limiter and uses only the batch processor. All three pipelines write to the same ClickHouse instance using the native TCP protocol on port 9000.

The VirtualService for the dashboard uses the TrueFoundry Istio gateway istio-system/tfy-wildcard and routes traffic from the cluster domain to the internal openlit service on port 3000. This is the only component that requires external exposure. The collector endpoints stay internal.

Internal Service DNS Reference

ComponentDNSPortOTel Collector HTTPopenlit.NAMESPACE.svc.cluster.local4318OTel Collector gRPCopenlit.NAMESPACE.svc.cluster.local4317ClickHouse HTTPopenlit-db.NAMESPACE.svc.cluster.local8123ClickHouse Native TCPopenlit-db.NAMESPACE.svc.cluster.local9000Dashboardopenlit.NAMESPACE.svc.cluster.local3000

Architecture Summary

A request enters the TrueFoundry AI Gateway and is processed entirely in memory against cached authentication state and routing configuration. The gateway forwards the request to the LLM provider and streams the response back to the client. After the response completes the gateway publishes a span to NATS containing the full request and response attributes and token counts and latency. The OTEL exporter picks up the span from NATS asynchronously and forwards it over OTLP/HTTP to the OpenLIT collector running at port 4318 inside the cluster. The collector writes the span to ClickHouse. The OpenLIT dashboard reads from ClickHouse and makes the trace available for inspection.

The Metrics tab surfaces aggregated time-series data from the otel_metrics table. Gateway request volume and latency trends are available over selectable time windows from 24 hours to 3 months.

No sidecars are required. No SDK changes are required. No instrumentation code is added to any application. The only configuration change is setting the OTEL exporter endpoints in the TrueFoundry AI Gateway settings. The gateway already generates and publishes the spans. The exporter configuration determines where they go.

The architectural principle that makes this integration work is the separation between the request path and the telemetry path. Because spans are published to NATS after the response is complete the reliability of the observability backend has no effect on gateway latency or availability. OpenLIT receives what the gateway publishes. If OpenLIT is unavailable the gateway continues operating and resumes exporting when connectivity is restored.

‍