October 16, 2026
|
5 min read
LiteLLM Tutorial: Install, Configure, Run the Proxy
By Abhilash Sanap
Published: October 16, 2026
Built for Speed: ~10ms Latency, Even Under Load
Blazingly fast way to build, track and deploy your models!
- Handles 350+ RPS on just 1 vCPU — no tuning needed
- Production-ready with full enterprise support
How to install LiteLLM (pick a path)
This LiteLLM tutorial walks platform engineers through installing the proxy, writing a minimal config.yaml, setting the master key, sending a first OpenAI-SDK request, adding models, enabling Redis for multi-worker setups, and avoiding the misconfigs that show up in the first week of production. Commands follow the official LiteLLM quickstarts. Pin versions when you promote beyond a laptop—LiteLLM ships minors roughly weekly. New to the pattern itself? Start with what an LLM gateway is and why platform teams put one in front of model providers.
Path A: pip / uv (gateway on your machine)
Official gateway quickstart installs the proxy extras with uv:
TrueFoundry AI Gateway delivers ~3–4 ms latency, handles 350+ RPS on 1 vCPU, scales horizontally with ease, and is production-ready, while LiteLLM suffers from high latency, struggles beyond moderate RPS, lacks built-in scaling, and is best for light or prototype workloads.
Built for Speed: ~10ms Latency, Even Under Load
Try now.
Start free
How Can You Prevent GenAI Costs From Spiraling at Scale?
Gartner Hype Cycle for Platform Engineering 2026
One Layer of Control for All AI
Discover More
No items found.
.webp)
.png)
No items found.
Recent Blogs
.png)
.png)
.png)
.png)
.png)
.png)
.png)
Frequently asked questions
How do I install LiteLLM and run the proxy?
Install the proxy extras with uv tool install 'litellm[proxy]' (or pip install 'litellm[proxy]' in a virtualenv), write a config.yaml with a model_list, and start it with litellm --config config.yaml. The proxy listens on port 4000 by default. If you need the Admin UI, virtual keys, and spend tracking, use the Docker Compose quickstart, which adds Postgres.
How does an OpenAI SDK client talk to the LiteLLM proxy?
Set the client’s base_url to the proxy (for example http://localhost:4000 or your ingress) and pass a LiteLLM virtual key as api_key. The model field is the public model_name from config.yaml, and responses come back in OpenAI format. On a config-only run without a database, the master key is the Bearer credential, but apps should get virtual keys in any shared setup.
Why do I need Redis for a multi-replica LiteLLM deployment?
Without shared Redis, each worker or replica keeps its own rate-limit counters, router state, and cache. A 100 RPM limit across four workers then effectively allows about 400, revoked keys can linger on other workers, and cache hits stay local. Configure Redis in both router_settings and litellm_settings.cache_params—setting REDIS_HOST in the environment alone is not enough.
Does the TrueFoundry AI Gateway integrate with my existing observability stack?
Yes. The gateway is OpenTelemetry-compliant and plugs into Grafana, Datadog, Prometheus, or your preferred stack. Every request is traced from prompt to model execution, so metrics such as latency, cost, and errors land next to your existing dashboards.
