.png)
October 16, 2026
|
5 min read
Realtime / Live API and Voice Agents Through the Gateway
By Ashish Dubey
Published: October 16, 2026
.png)
Built for Speed: ~10ms Latency, Even Under Load
Blazingly fast way to build, track and deploy your models!
- Handles 350+ RPS on just 1 vCPU — no tuning needed
- Production-ready with full enterprise support
TrueFoundry AI Gateway delivers ~3–4 ms latency, handles 350+ RPS on 1 vCPU, scales horizontally with ease, and is production-ready, while LiteLLM suffers from high latency, struggles beyond moderate RPS, lacks built-in scaling, and is best for light or prototype workloads.
Built for Speed: ~10ms Latency, Even Under Load
Try now.
Start free
How Can You Prevent GenAI Costs From Spiraling at Scale?
Gartner Hype Cycle for Platform Engineering 2026
One Layer of Control for All AI
Discover More
No items found.
.png)
No items found.
Recent Blogs
.png)
.png)
.png)
.png)
.png)
Frequently asked questions
What is the Realtime / Live API?
A bidirectional, low-latency streaming API over WebSocket where a model receives and produces audio directly — speech-to-speech — instead of going through separate transcription and text-to-speech steps. OpenAI calls it the Realtime API; Google calls it the Live API.
How is it different from an STT → LLM → TTS pipeline?
The Realtime API is a single stream with lower latency and provider-managed turn-taking, but costs more per minute. The pipeline uses three separate services, which is cheaper and more tunable but adds latency. The gateway supports both.
How much does realtime voice cost?
It’s priced in audio tokens. OpenAI Realtime is roughly $0.30 per minute and Gemini Live roughly $0.03 per minute at current rates, and both re-bill context each turn. Verify current figures on the providers’ pricing pages.
Which providers can I use through the gateway?
OpenAI, Google Gemini, Google Vertex AI, and Azure AI Foundry, each via its native SDK pointed at the gateway’s WebSocket URL.
Can I switch realtime providers without changing my client?
Yes. Because the client connects to the gateway rather than a provider directly, switching from OpenAI Realtime to Gemini Live is a configuration change, and you can split traffic to compare them.
Can I run this in my own cloud?
Yes. TrueFoundry’s data plane runs in your VPC or on-prem, so DSPy’s prompts and optimization data stay in your environment.
