Realtime / Live API and Voice Agents Through the Gateway

What the Realtime / Live API is

Voice agents used to simply be a pipeline of speech transcription to text (STT), which is passed on to LLM for generation, then converted back to speech (TTS). Three services, three hops, and cumulative latency at each step.

The Realtime API (OpenAI) and Live API (Google) collapse that into one. The model is connected through an established WebSocket and receives audio input and responds in kind, listening and talking all on its own. This produces noticeably better results in terms of lower latency, making the conversation more interactive, and this is why it is the standard choice for conversations using voice chatbots.

There are downsides associated with cost and control of that capability, and there is the practical fact that introducing an API means another key, another endpoint, another bill. These are both reasons to have this activity occur behind a gateway.

Realtime API vs the STT → LLM → TTS pipeline

Neither approach is strictly better — they optimize for different things.

The Realtime / Live API wins on latency and naturalness because there’s a single stream and the provider handles turn detection. The pipeline (the approach we covered in our Deepgram post) wins on cost and control: you pick the STT model, the LLM, and the TTS voice independently, tune each, and swap any one of them. A common pattern is to use Realtime for the premium, latency-sensitive experience and the pipeline for cost-sensitive or highly customized flows — and because the gateway supports both, that’s a routing decision, not a re-architecture.

What it costs

Realtime voice is priced in audio tokens, and the two leading providers are far apart.

At current rates, OpenAI’s Realtime API runs on the order of $0.30 per minute of active conversation, while Google’s Gemini Live is roughly $0.03 per minute — about a 10× gap [VERIFY current rates on OpenAI and Google pricing pages; figures are blended per-minute estimates]. Both bill per token, and crucially both re-bill the growing context on each turn, so a long call costs more per minute as it goes, not less. For a phone agent handling thousands of minutes a day, the provider choice and the call-length distribution dominate the bill — which is exactly the kind of thing you want metered per call at the gateway rather than discovered on an invoice.

Realtime through the gateway

The gateway proxies the WebSocket connection to the provider using each provider’s native SDK, so you get the real Realtime/Live protocol with a governance layer around it.

Your client opens one wss:// connection to the gateway; the gateway holds the provider key and relays the bidirectional audio stream. That gives you one endpoint and one key across providers, cost tracking on realtime sessions alongside your other LLM spend, and the ability to switch providers — OpenAI to Gemini, say, when the cost math changes — without re-pointing your client at a new vendor.

Four providers are supported today — OpenAI, Google Gemini, Google Vertex AI, and Azure AI Foundry — each via its native SDK (openai, google-genai, or azure-ai-voicelive) pointed at the gateway’s WebSocket URL.

Where teams get this wrong

They assume Realtime is always the answer. For cost-sensitive or highly custom flows, the STT → LLM → TTS pipeline is cheaper and more controllable. Match the approach to the use case.

They ignore per-turn re-billing. Realtime context is re-charged each turn, so long calls scale worse than you’d expect. Model cost on realistic call lengths.

They lock to one provider’s SDK in the client. Hard-wiring OpenAI’s endpoint into the app means switching to Gemini is a rewrite. Point the SDK at the gateway and switching is config.

They leave realtime spend untracked. Audio-token billing doesn’t show up in token dashboards built for text. Meter it at the gateway.

Want realtime voice cost and provider choice under control?

Book a demo

How to connect in TrueFoundry

Add your realtime model to a model account with the Realtime model type, then connect over WebSocket. The model ID in code must match the model’s display name on your account. For the native Live API path, connect to

{GATEWAY_BASE_URL}/live/{providerAccountName} using wss://; for OpenAI’s Realtime protocol you can also use /v1/realtime?model=....
import asyncio, json, websockets

URL = "{GATEWAY_BASE_URL}/v1/realtime?model=openai-main/realtime-preview"
headers = {"Authorization": "Bearer your-tfy-api-key"}

async def main():
    async with websockets.connect(URL, additional_headers=headers) as ws:
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {"modalities": ["audio", "text"], "model": "openai-main/realtime-preview"}
        }))
        await ws.send(json.dumps({"type": "response.create"}))
        async for msg in ws:
            event = json.loads(msg)
            if event.get("type") == "response.done":
                break

asyncio.run(main())

You keep OpenAI’s or Google’s full Realtime/Live event model — session.update, conversation.item.create, response.create, streaming deltas, and tool calling — because the gateway relays the native protocol. For telephony, the gateway also supports WebRTC and LiveKit.

A worked example: a phone agent that can switch providers

A company ships a phone-support voice agent on OpenAI Realtime for its natural turn-taking. Three months in, call volume is up and the Realtime bill is the single largest line in their AI spend. They want to test Gemini Live, which is roughly 10× cheaper per minute, on a slice of traffic.

Because the agent connects through the gateway, the test is a routing change: point a percentage of sessions at the Gemini account, keep the rest on OpenAI, and compare quality and cost side by side in the gateway’s metrics — same client code, same wss:// endpoint. If Gemini holds up on their calls, they shift more traffic; if not, they roll back instantly. Without the gateway, that experiment would mean integrating a second provider SDK into the client and shipping an app update to every caller.

Gotchas worth knowing

• Use wss:// and the right base path. Live API goes to /live/{account}; OpenAI Realtime also accepts /v1/realtime.
• Model ID = display name. The model in code must match your TrueFoundry model account’s display name.
• Per-turn re-billing. Long calls cost more per minute as context grows; budget accordingly.
• Realtime spend is audio tokens. Make sure it’s tracked, not assumed from text dashboards.
• Telephony needs transport. Use WebRTC or LiveKit for phone integrations.

Related reading

• Deepgram on the AI Gateway: STT, TTS, cost and governance
• Cartesia and the TrueFoundry AI Gateway: native passthrough for voice
• Why specialized observability is non-negotiable for production voice AI
• Multi-model routing on the AI Gateway
• LLM cost tracking: a practical solution
• What is LLM fallback?

Conclusion

The Realtime and Live APIs are the fastest path to a natural voice agent — one WebSocket, speech in and speech out, turn-taking handled for you. The catch is cost and lock-in: audio-token billing that grows with call length, and a per-provider SDK that’s easy to wire straight into your client.

Running realtime voice through the TrueFoundry AI Gateway keeps the native protocol and full feature set while adding the layer production needs: one endpoint and key across providers, cost tracked per session, and provider switching as a config change. Build the voice experience you want, and keep the economics and governance under control.

Run realtime voice through one governed endpoint

Book a demo