What Is LiteLLM? Architecture, Features, and Limits

What is LiteLLM?

LiteLLM is an open-source library and AI gateway that gives applications a single, OpenAI-compatible interface to call 100+ LLM providers—OpenAI, Anthropic, Azure OpenAI, Amazon Bedrock, Google Vertex AI, Cohere, local OpenAI-compatible servers, and many others. You can use it in two modes: as a Python SDK inside one application, or as a centralized proxy (AI Gateway) that your whole platform points at.

That distinction matters. The SDK is a developer dependency. The proxy is infrastructure: authentication, virtual keys, budgets, rate limits, routing, spend logs, an Admin UI, and integrations for observability. Official docs and the BerriAI/litellm GitHub README frame the product this way—SDK for direct library integration; AI Gateway for GenAI enablement and ML platform teams.

OpenAI-compatible API surface

The proxy accepts requests in OpenAI API shapes (for example /v1/chat/completions and related endpoints) and translates them to the underlying provider. Clients typically set the OpenAI SDK base_url to the proxy (commonly http://localhost:4000 or your ingress) and pass a LiteLLM master key or virtual key as the API key. The response comes back in OpenAI-style JSON so application code does not branch on every vendor SDK.

This is the main product promise: one client integration, many backends. It is not a claim that every provider feature maps 1:1. Provider-specific optional parameters, streaming edge cases, and tool schemas still need careful testing for your critical paths.

Architecture: gateway, control plane, and data stores

At a minimum production topology you run LiteLLM behind a load balancer, with Postgres for durable state and Redis once you have more than one worker or replica. Clients and admins hit the same gateway host; LLM traffic goes through auth, budget, and routing; management operations and the Admin UI depend on the database.

It is useful to think in two planes. The data plane is the hot path: authenticate the virtual key, enforce ACL and budgets, pick a deployment, translate the request, call the provider, return the response, and enqueue spend logging. The control plane is the Admin UI and management APIs: keys, teams, models, budgets, and day-2 config overlays stored in Postgres.

Componentized deployment (gateway, backend, UI)

LiteLLM can run as a monolithic image (one process serves LLM traffic, management APIs, and UI)—what the litellm-helm chart typically runs—or as microservices: a gateway for LLM traffic (port 4000), a backend for management APIs and UI backend (port 4001), and a UI (port 3000). Official production deployment docs describe both. The componentized chart and published Terraform modules deploy the microservices form so you can scale many gateway replicas against a smaller backend. Component images (litellm-gateway, -backend, -ui, -migrations) are documented as published from v1.89.0 onward—pin chart and image tags; do not assume :latest in production.

Supporting components called out in those docs: PostgreSQL for keys, teams, users, spend logs, and config; Redis for rate limiting, router state, and caching across instances; and a migrations job that applies schema changes while proxy pods can set DISABLE_SCHEMA_UPDATE=true.

What Postgres and Redis are for

Postgres is required for Admin UI model management, virtual keys, and spend tracking. Without a database, you can still run an OpenAI-compatible API from config.yaml, but budgets are not enforced the way teams expect: official quickstart docs warn that litellm_settings.max_budget does not act as a hard spend cap without a DB client, and virtual keys themselves need a connected database.

Redis (7.0+ is called out in production best practices) is recommended as soon as you run more than one proxy instance or more than one Uvicorn worker. Official “What Needs Redis” guidance: without shared Redis, each worker keeps its own counters and cache, so a 100 RPM limit across four workers effectively becomes ~400, revoked keys can linger on other workers until reload, and response cache hits stay local. Semantic caching backends that depend on Redis/Valkey/Qdrant need that infrastructure outright.

Virtual keys, budgets, and spend

Virtual keys are LiteLLM-issued credentials you hand to applications instead of raw provider keys. Each key can carry budgets, rate limits, and model allowlists; spend is tracked in Postgres. Teams and users sit above keys for multi-tenant governance. Key generation and editing are available via Admin UI and management APIs. Automated virtual-key rotation with grace periods is documented as an Enterprise feature; master-key rotation flows exist on the open-source build and are API-oriented.

Production docs also recommend batching spend writes (for example proxy_batch_write_at: 60) so high QPS does not hot-spot the database. At very high traffic (~1000+ RPS or 10+ instances), a Redis transaction buffer can queue spend updates before a single flusher writes to Postgres.

Admin UI

With the database-backed quickstart, the Admin UI is typically at /ui. Default login uses username admin and the LITELLM_MASTER_KEY as the password (the master key is also the root credential for management APIs). Treat that key like a root password. Enterprise adds SSO (Okta, Azure AD, Google Workspace, OIDC/SAML) for UI login; SSO is documented as free for up to five users, with a license beyond that.

Config.yaml versus database: drift risk

Models and settings can live in config.yaml (GitOps-friendly, restart to change file-owned models) and/or in the database when store_model_in_db is enabled (UI and API day-2 changes). Pods poll the DB to pick up runtime overlays; the default reload interval is on the order of 30 seconds and is tunable. Same model name in YAML and DB does not silently replace—you can end up with extra deployments. Decide a single source of truth for production model inventory and document which knobs are file-owned versus DB-owned.

Master key and salt key (ops ownership)

Set these early and keep them in a secret manager:

export LITELLM_MASTER_KEY="sk-<long-random-value>"
export LITELLM_SALT_KEY="sk-<permanent-salt-value>"

LITELLM_MASTER_KEY authenticates admin APIs and, by default, doubles as the Admin UI password. Production docs say it must start with sk-. LITELLM_SALT_KEY encrypts provider credentials stored in the database. If you change the salt key after models are stored, those credentials become unreadable until re-entered. If no salt key is set, the master key falls back as the encryption key—making master-key rotation a migration event. Set a permanent salt key early; keep both keys in a secret manager; never commit them.

What LiteLLM is not

• It does not serve GPUs or host model weights. Inference still runs at providers or at your own vLLM/Ollama/OpenAI-compatible endpoints behind the gateway.
• It is not a full replacement for an identity platform. Virtual keys and (Enterprise) SSO/JWT plug into your IdP; they are not a substitute for org-wide IAM design.
• It is not “set and forget.” You own Postgres, Redis, upgrades, migrations, key hygiene, provider outages, and config drift.
• It does not magically unify every vendor quirk. Expect translation gaps and test streaming, tools, and multimodal paths for models you rely on.

Enterprise versus OSS (high level)

OSS already includes the OpenAI-compatible gateway, virtual keys, spend tracking, budgets, fallbacks, and request/response logging. Enterprise adds controls for larger orgs: SSO + SCIM, OIDC/JWT auth for requests, org/team admin roles, secret-manager integrations, automated virtual-key rotations, IP allowlists, richer audit/management logs, per-team logging routing, multi-region under one license, and admin/worker split patterns. Some built-in moderation callback integrations are license-gated; custom guardrails and Presidio PII masking are called out as available in the OSS guardrail framework. Always re-check the current Enterprise comparison table before buying decisions.

Release cadence and supported versions

Official Enterprise docs state that LiteLLM ships a new minor line roughly every week, and supports the four most recent stable minor lines with ongoing patches. Older lines reach end of life and stop receiving updates. That support policy is documented as taking effect Monday, June 29, 2026. As of mid-June 2026 the supported lines listed were 1.86–1.89, rolling forward as new stables ship. Pin image tags in production; plan upgrades before your minor line ages out of the four-line window.

Practical limits platform engineers should budget for

• Ops ownership: secrets (master/salt), DB migrations, Redis HA, ingress TLS, and alerting are on you—not on the LLM provider.
• Multi-replica correctness depends on Redis for shared rate limits, router cooldowns, and cache; skipping it silently weakens controls.
• Config drift between YAML and DB is a real failure mode; pick an SoT.
• Salt key mishandling brick-walls stored provider credentials.
• Weekly minor cadence means patch discipline and regression testing matter.
• Without Postgres, “budget” settings and virtual keys do not give you the governance story marketing one-liners imply.

When LiteLLM is a good fit

LiteLLM fits platform teams that need one OpenAI-compatible front door across many providers, with key-level spend and access controls, without writing a custom gateway. It is less of a fit if you only ever call one provider from one app (the SDK or the provider SDK alone may be enough), or if you expected a managed GPU inference product. For TrueFoundry-style platform contexts, evaluate it as a control plane for LLM egress—honest about Postgres/Redis/ops cost—not as a model host. If you are weighing options against those ops costs, our roundup of LiteLLM alternatives compares how other gateways handle the same trade-offs.

Related reading

• What Is an LLM Gateway and How Does It Work? — the architectural primer behind the gateway pattern LiteLLM implements
• Top 5 LiteLLM Alternatives in 2026 — how other gateways, including TrueFoundry, compare for production teams
• LLM Load Balancing — routing, fallbacks, and spreading traffic across model deployments

Conclusion

LiteLLM is a capable open-source way to give every team one OpenAI-compatible front door to many LLM providers, with virtual keys, budgets, and spend tracking on top. The trade-off is ownership: Postgres, Redis, master and salt keys, weekly upgrades, and config drift are yours to run. Evaluate it as a control plane for LLM egress, with those costs budgeted up front.

If you want the same unified API with governance and observability built in, deployed in your own VPC, learn how TrueFoundry’s AI Gateway handles it →