LLM Monitoring with Bifrost: Prometheus and OpenTelemetry in Practice

Bifrost, Maxim AI’s open-source gateway, ships a surprisingly complete LLM monitoring stack in its free tier: a request log, a Prometheus /metrics endpoint and OpenTelemetry trace export. Getting from “it exposes metrics” to dashboards and alerts you can trust takes a few decisions the docs scatter across several pages. This guide puts them in one place: what to turn on, how to scrape it, which labels matter, the queries worth saving, and the settings that quietly leak prompt content or break your counters.

This guide is built from Bifrost’s documentation, plus a few things we saw first-hand running Bifrost OSS v2.0.0, which we point out where they come up.

Gateway latency is out of scope here; our Bifrost review covers performance ([USER TO FILL: link to the review post when live]).

One request, three planes

A single chat request through Bifrost lands in three places at once, all available in OSS (built-in observability):

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-bf-vk: sk-bf-team-search" \
  -H "x-bf-dim-environment: production" \
  -d '{"model":"openai/gpt-4o-mini","messages":[{"role":"user","content":"Summarize Q3 revenue"}]}'

1. ‍‍Logs. The always-on logging plugin writes a row asynchronously: input and output messages, model, token counts, calculated USD cost, latency, status and any logging headers. The docs put its overhead under 0.1 ms. You can see it in the dashboard on port 8080 or query it through GET /api/logs.‍
2. Metrics. Counters such as bifrost_upstream_requests_total, bifrost_input_tokens_total and bifrost_cost_total increment with labels for provider, model and virtual key, plus environment="production" from the dimension header.‍
3. Traces. If the OTel plugin is enabled, a span goes to your collector in the trace format you chose, carrying the same dimension so it can sit next to your application’s traces.

The rest of this guide is about making each plane production-ready.

LLM metrics with Prometheus

Turning it on and scraping it

The telemetry plugin is enabled by default and serves Prometheus exposition format at /metrics on the gateway port (Prometheus). For a single node, a plain scrape job is enough:‍


scrape_configs:
  - job_name: bifrost
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets: ["bifrost:8080"]
    # Needed once Bifrost auth is on (see below)
    basic_auth:
      username: prometheus
      password_file: /etc/prometheus/bifrost-password

Two things trip people up here.

Auth changes the scrape. When auth_config.is_enabled is true, /metrics requires Basic auth, and a scrape job without matching credentials fails with a 401. Prometheus shows the target as down, so you notice, but only if you are watching target health. The reverse problem is worse, and we hit it in our own testing: a fresh OSS install starts with no authentication at all, so /metrics, the dashboard and the admin API are open to anyone who can reach the port until you configure an admin and a setup token. Don’t expose port 8080 beyond the scraper’s network.

Don’t scrape through a load balancer. Every replica keeps its own counters. If Prometheus scrapes a load-balanced address, each scrape lands on a different pod and your counters jump backwards and forwards. Scrape each pod directly (Kubernetes service discovery on the pod IPs), or use the Push Gateway mode, which Bifrost’s docs recommend for multi-node clusters behind a load balancer. This matters more on OSS, where replicas don’t share any state: budgets and counters live per pod, so per-pod metrics are the only honest view.

What Bifrost exposes

The metric catalog, grouped by what you would use it for:

The HTTP metrics label path with the matched route template, not the raw URL, which keeps cardinality bounded. One thing to remember about bifrost_cost_total: it is a cost Bifrost calculates from its pricing catalog and the usage fields providers return. It is good for budgets and showback, but it is not an invoice, and negotiated discounts or credits won’t show up in it.

The labels that make it useful

The default labels carry the same identity spine Bifrost uses for governance, which is what turns raw counters into answers:

• provider, model and alias
• virtual_key_id / virtual_key_name, team_id / team_name, customer_id / customer_name. These are empty when you don’t use governance.
• routing_engine_used, routing_rule_id / routing_rule_name, selected_key_id / selected_key_name and fallback_index, which together explain why a request went where it did.

Per-user attribution is not a default metric label. You get it through logging headers (X-User-ID or any x-bf-lh-* header), which land in the request log, or through dimensions.

Custom dimensions, and the cardinality trap

Any request header starting with x-bf-dim- becomes a dimension that is written to the request log, the OTel span and Prometheus as a custom label at the same time (shared observability dimensions). That is the cleanest way to tag traffic by environment, feature or tenant without new virtual keys.

It is also the easiest way to break Prometheus. Every distinct value of a label creates a new time series for every metric that carries it. A dimension like environment with three values is fine. A dimension like request_id, session_id or user_id creates a series per value across dozens of metrics. Keep high-cardinality identifiers in logging headers (x-bf-lh-*), which only go to the request log, and reserve x-bf-dim-* for values you would happily put on a dashboard dropdown.

PromQL worth saving

These use the documented metric and label names. Confirm the metric types against your build’s /metrics output first; the TTFT query assumes the latency metrics are histograms.‍

# Spend per team over the last 24 hours (calculated cost, USD)
sum by (team_name) (increase(bifrost_cost_total[24h]))

# Token burn per virtual key, per minute
sum by (virtual_key_name) (rate(bifrost_input_tokens_total[5m]) + rate(bifrost_output_tokens_total[5m])) * 60

# Error ratio per provider
sum by (provider) (rate(bifrost_error_requests_total[5m]))
  / sum by (provider) (rate(bifrost_upstream_requests_total[5m]))

# p95 time to first token per model
histogram_quantile(0.95,
  sum by (le, model) (rate(bifrost_stream_first_token_latency_seconds_bucket[5m])))

# Provider keys currently marked down
bifrost_provider_key_up == 0

# Cache hit ratio (hits never reach the upstream)
sum(rate(bifrost_cache_hits_total[1h]))
  / (sum(rate(bifrost_cache_hits_total[1h])) + sum(rate(bifrost_upstream_requests_total[1h])))

# How often requests fall back off the primary target
sum by (model) (rate(bifrost_upstream_requests_total{fallback_index!="0"}[15m]))

‍Alerting you will have to bring yourself

Bifrost’s built-in alerting engine is Enterprise only, and it evaluates budget and rate-limit metrics (budget percent used, spend, request and token usage), not error rates or latency (alerting). For reliability alerts on either tier, point Alertmanager at the metrics above:

groups:
  - name: bifrost
    rules:
      - alert: BifrostProviderErrorRateHigh
        expr: |
          sum by (provider) (rate(bifrost_error_requests_total[10m]))
            / sum by (provider) (rate(bifrost_upstream_requests_total[10m])) > 0.05
        for: 10m
      - alert: BifrostProviderKeyDown
        expr: bifrost_provider_key_up == 0
        for: 5m
      - alert: BifrostTTFTSlow
        expr: |
          histogram_quantile(0.95, sum by (le, model)
            (rate(bifrost_stream_first_token_latency_seconds_bucket[10m]))) > 3
        for: 15m
      - alert: BifrostSpendSpike
        expr: sum(increase(bifrost_cost_total[1h])) > 2 * sum(increase(bifrost_cost_total[1h] offset 1d))
        for: 30m

‍‍Bifrost doesn’t ship a Grafana dashboard or an alert-rule pack in the docs we reviewed, so these become your team’s to own.

LLM tracing with OpenTelemetry

Enabling the OTel plugin

Trace export lives in the otel plugin, which is part of OSS (plugin configuration). It is off by default in the Helm chart; once you add it, trace export is on by default. The settings you will touch:

On the receiving side, a minimal OpenTelemetry Collector that accepts both protocols and forwards to your tracing backend looks like this:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch: {}

exporters:
  otlphttp/tempo:
    endpoint: http://tempo:4318

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/tempo]

‍Keep a collector between Bifrost and any SaaS backend, even if the backend accepts OTLP directly. It gives you one place to batch, filter attributes and switch vendors without touching gateway config.

Choosing a trace format: OTel GenAI, OpenInference or Vercel

trace_type decides how downstream tools interpret Bifrost’s spans. A rough guide to picking one:

• genai_extension follows the OpenTelemetry GenAI semantic conventions (the gen_ai.* attributes, sometimes shortened to OTel GenAI). Pick it for general-purpose backends such as Grafana Tempo, Honeycomb, Jaeger or Datadog over OTLP.
• open_inference follows Arize’s OpenInference conventions. Pick it if your trace viewer is built around OpenInference, such as Arize Phoenix. Bifrost doesn’t document native Langfuse or Arize connectors, so treat OTLP plus OpenInference as the likely route there rather than a supported integration.
• vercel matches the Vercel AI SDK’s conventions. Pick it if the rest of your traces already come from the Vercel AI SDK.

Whichever you pick, send one test request and check the attribute names in your backend before building dashboards on them.

Prompt content in your traces

This is the setting most likely to surprise a security reviewer. Bifrost’s client-level disable_content_logging strips prompts and responses from its own log store, but it does not apply to observability connectors. OpenTelemetry, Datadog, BigQuery, Kafka and Pub/Sub each have their own disable_content_logging flag, and each defaults to false (client configuration). So a team that turns content logging off at the client and adds the OTel plugin is, by default, still exporting full prompts to its tracing backend.

For a metadata-only posture, set the flag on the client and on every connector you enable. Captured request headers are exported regardless of the content flags, so check which headers you capture too. If you need some content for debugging, an attribute-dropping processor in your collector is a second line of defense, but match it to the attribute names your chosen trace format actually emits.

Joining Bifrost spans to your application traces

End-to-end W3C trace-context propagation is documented for the native Datadog plugin, which uses dd-trace-go. For the generic OTel plugin, the docs don’t say whether an incoming traceparent header makes Bifrost’s span a child of your application’s span. Test this before you promise anyone a single waterfall from frontend to model. If it doesn’t join, the shared x-bf-dim-* dimension is your fallback for correlating the two.

The request log: the plane you already have

The logging plugin is the one plane you don’t have to set up. Points that make it useful in practice:

• GET /api/logs filters by provider, model, status, time range, latency, tokens and cost, and supports full-text content_search. A WebSocket stream gives you a live tail.
• Since v1.5.0, each row records the selected key, the retry count and an ordered attempt_trail showing every attempt, its key, the failure reason and whether it triggered key rotation. When a provider flaps, this is faster than any dashboard.
• The store is SQLite by default and PostgreSQL for production. Retention defaults to 365 days via log_retention_days, with a cleaner that runs every 24 hours.
• On v2.0.0, raw provider request and response capture is a toggle under Observability, then Logs Settings, in the dashboard. We turned it on to inspect exactly what Bifrost sent to each provider, and it is worth knowing it exists before you need it.

There is no rate-based sampling for logs. The nearest switch, drop_excess_requests, sheds load when queues fill; it is not a sampler. At high volume, plan storage for every request, or turn content logging off.

Gotchas, in one list

• The client content-logging switch doesn’t cover connectors. Set it per connector.
• Enabling auth makes /metrics require Basic auth; until auth is on, everything is open.
• Scraping through a load balancer scrambles per-pod counters. Scrape pods or use the Push Gateway.
• x-bf-dim-* values become Prometheus labels. Keep IDs out of them.
• The OTel plugin is off by default in Helm, even though it is free.
• Cost metrics are calculated from a pricing catalog that syncs from Maxim every 24 hours. In an air-gapped install you have to supply the pricing file yourself.
• Native alerting is Enterprise and covers spend and rate limits only. Latency and error alerts are yours to build.
• If you offload logs to S3 or GCS (Enterprise), the retention cleaner doesn’t delete those objects. Set a bucket lifecycle rule.

What’s OSS and what’s Enterprise

If the Enterprise question goes beyond observability, our breakdown of [USER TO FILL: link to the Bifrost Enterprise vs OSS post when live] covers identity, audit and clustering.

Where Bifrost monitoring stops

Bifrost tells you what happened. It doesn’t tell you whether the answer was any good. Quality signals such as evaluations, drift and hallucination checks live in Maxim’s separate platform, which the free Maxim plugin forwards traffic to, or in tooling you build from exported logs. The gateway also leaves dashboard design, alert rules, log storage sizing and per-team log policy to you. Content logging is on or off globally, per request or per connector; there is no rule like “never log the finance team’s prompts.”

The TrueFoundry alternative

The TrueFoundry AI Gateway covers the same three planes with less to assemble. A few differences that matter after reading the gotchas above:

• A dashboard you don’t have to build. The metrics dashboard covers cost, LLM and MCP calls, error breakdowns, guardrail outcomes, routing, caching and cost savings, and filters by model, user, virtual account and team (metrics dashboard).
• OTel export from the UI. Traces and metrics can each be exported to any OTLP backend over HTTP or gRPC, with custom headers and an option for delta temporality, configured in the gateway settings rather than in a config file (export OpenTelemetry data).
• Prometheus when you self-host the gateway. A self-hosted gateway exposes /metrics with model, tenant and user labels, and you choose which request metadata keys become labels, which keeps cardinality under your control (Prometheus and Grafana).
• Logging rules by who and what. Logging configs match on users, teams, virtual accounts, models or request metadata, decide whether bodies are logged with deny-wins semantics, and redact patterns from the stored copy (logging configuration). That is the “never log the finance team’s prompts” rule Bifrost doesn’t have.

If you would rather spend your time on what the metrics say than on wiring them up, see how the TrueFoundry AI Gateway handles LLM monitoring.

Related reading

• Observability in an AI gateway
• What is LLM observability?
• Best LLM observability tools
• AI agent observability: why did it do that?
• Bifrost pricing: OSS, Enterprise costs, and what teams should know

Conclusion

Bifrost gives you a good base for LLM monitoring for free: per-request logs, Prometheus metrics with governance-aware labels, and OpenTelemetry traces in your choice of convention. Making it production-ready is on you. Scrape per pod, keep IDs out of dimension labels, set the content flag on every connector, and build the alerts and dashboards it doesn’t ship. If that list looks like work you would rather not own, book a TrueFoundry demo and compare the two setups side by side.