LiteLLM Caching: Redis, Semantic Cache, What Breaks

What LiteLLM cache means

A LiteLLM cache stores LLM responses so a later identical (or, for semantic mode, similar) request can skip the provider call. That cuts latency and provider spend when traffic is repetitive—and creates correctness bugs when it is not. This article covers exact-match caches (in-memory, Redis/Valkey, S3/GCS, disk), semantic caches (Redis, Valkey, Qdrant), TTL and keys, multi-replica requirements, invalidation, and the failure modes platform engineers hit in production. Scope note: this is LiteLLM’s application-level response cache, not OpenAI/Anthropic provider-side prompt caching (a separate feature with its own configuration). Caching is one of several controls an LLM gateway applies on the request path, alongside auth, rate limits, and routing.

Supported cache backends at a glance

Official proxy caching docs list: redis (also Valkey/ElastiCache/Memorystore via Redis protocol), redis-semantic, valkey-semantic, qdrant-semantic, s3, gcs, local (in-memory), and disk. Redis is the right default past a single worker. Exact-match caches key on a hash of the whole request. Semantic caches embed the prompt and serve the closest match above a similarity threshold—suited to single-shot prompts and a poor fit for agentic traffic.

Cache hit versus miss (exact-match)

On a hit, responses can carry an x-litellm-cache-key header useful for debugging and for /cache/delete. With cache: True and no supported_call_types override, caching applies to /chat/completions, /completions, /embeddings, /audio/transcriptions, /rerank, /v1/responses, and /v1/messages (confirm on your version). On /v1/responses, exact-match keys on the request body—so previous_response_id versus an inlined conversation are different keys, and each multi-turn step is its own entry.

Enable caching on the proxy

Minimal enable (defaults toward Redis when credentials exist):

litellm_settings:
  cache: True

Then set REDIS_URL or REDIS_HOST / REDIS_PORT / REDIS_PASSWORD, and preferably explicit cache_params.type: redis. Debug with:

curl --location 'http://0.0.0.0:4000/cache/ping' \
  -H "Authorization: Bearer $LITELLM_API_KEY"

A healthy response reports cache_type, ping_response, and redis client metadata. Use this in readiness checks for multi-replica rollouts.

In-memory / local versus Redis

Laptop / single-process:

litellm_settings:
  cache: True
  cache_params:
    type: local

Disk: type: disk with optional disk_cache_dir. Neither is shared between workers or replicas—docs say use them for local development, not production.

Production Redis example with TTL:

litellm_settings:
  cache: True
  cache_params:
    type: redis
    ttl: 600
    host: os.environ/REDIS_HOST
    port: os.environ/REDIS_PORT
    password: os.environ/REDIS_PASSWORD

Four workers without Redis means roughly four independent caches—hit rate collapses by about the worker count. Also configure router_settings Redis for shared RPM/TPM, cooldowns, and routing state. “What Needs Redis” covers rate limits, budgets coordination, cache invalidation, and pod locks. REDIS_HOST alone without YAML pointers will not enable sharing.

TTL, namespaces, keys, and per-request controls

ttl under cache_params sets entry lifetime in seconds. Per-request cache object fields: ttl, s-maxage (only accept entries younger than N seconds), no-cache (skip read), no-store (skip write), namespace. Attach the same controls on virtual-key metadata so a class of traffic opts out without client changes—standard pattern for agent keys.

Narrow participation with cache_params.supported_call_types (empty list disables). By default only standard OpenAI-shaped parameters enter the cache key; set enable_caching_on_provider_specific_optional_params: True if Vertex-style optional fields must distinguish entries. Custom get_cache_key functions exist on the SDK Cache object for advanced cases—prefer documented controls on the proxy before custom code.

Semantic cache: what it needs

A semantic cache embeds the prompt (messages text, system included) and serves the closest cached response whose cosine similarity clears similarity_threshold. Backends: redis-semantic, valkey-semantic, qdrant-semantic. Configure an embedding model LiteLLM can call—usually a model_name from model_list. Hits may include x-litellm-semantic-similarity.

Example Redis semantic shape:

litellm_settings:
  cache: True
  cache_params:
    type: redis-semantic
    similarity_threshold: 0.8
    redis_semantic_cache_embedding_model: azure-embedding-model

Valkey semantic needs the valkey-search module (ElastiCache: node-based Valkey 8.2+, cluster-mode-disabled recommended; Serverless and multi-shard cluster-mode-enabled are called out as unsupported for this backend). Qdrant needs collection name, vector size matching the embedding dimensionality, and API credentials.

Embedding runs inline before the LLM. Default timeout is 5 seconds; past that the lookup is abandoned (similarity 0.0) and the request misses through. Tune semantic_cache_embedding_timeout or SEMANTIC_CACHE_EMBEDDING_TIMEOUT_SECONDS carefully—a higher deadline is also how long every request waits when embeddings hang.

semantic_cache_scope defaults to key (virtual key / team / org bucket). Set end_user to isolate by authenticated end-user id (x-litellm-customer-id, configured user_header_name, or request user). Otherwise similar prompts on a shared key can cross-serve responses between end users—including tool-call payloads.

Exact-match versus semantic: choosing deliberately

Exact-match Redis is the safe default for most platform gateways: predictable keys, no embedding dependency on the hot path, and natural correctness for multi-turn traffic because any message change misses. Semantic cache is an optimization for high-volume, paraphrased, single-shot workloads where you accept approximate reuse. Do not enable semantic mode “because it sounds smarter” on a shared gateway that also serves agents—split traffic with virtual-key metadata or separate gateways if product needs both.

When cache helps versus when it hurts

Helps:

• Idempotent, repetitive single-shot prompts (FAQs, repeated classification inputs, identical ETL prompts).
• Repeated embedding of the same corpus shards.
• Lower provider spend and latency when temperature and messages are stable.
• Exact-match Redis across many gateway replicas with shared hit rates.

Hurts or is risky:

• Non-idempotent tasks that must reflect live state, time, prices, or unique side effects.
• Streaming-sensitive UX—verify behavior on your version; do not assume cached replies match live stream semantics without tests.
• Tool-calling and agent loops—especially with semantic cache.
• High-temperature creative generation where identical prompts should not return identical prose.
• Provider passthrough routes (for example /anthropic/v1/messages) that docs say never go through the cache.

Semantic cache and multi-turn / agentic traffic (what breaks)

Official semantic-caching docs are direct: designed for single-shot prompts; on multi-turn or agentic traffic it will replay stale responses. Each new turn is previous text plus a small delta, so embeddings stay ~0.99 similar; at practical thresholds every turn matches the previous entry. Raising similarity_threshold does not reliably fix it. Assistant tool_calls are not part of the embedded text, which makes consecutive agent turns even harder to tell apart—symptoms include agents repeating the same tool call.

Mitigations from those docs: keep semantic caching for single-shot traffic; exclude agentic traffic with key metadata cache.no-cache true, or mode default_off, or per-request controls. If agents still need caching, use exact-match type: redis so any conversation change misses cleanly.

Invalidation and observability

TTL is the blunt instrument. Namespaces segment environments or tenants. no-store / no-cache suppress write/read. /cache/ping verifies the backend. On hits, x-litellm-cache-key can feed /cache/delete for targeted invalidation—confirm the delete API on your version before automating. Success callbacks can read kwargs["cache_hit"] for metrics pipelines. Track hit rate, embedding latency (semantic), Redis errors (no connection available → raise max_connections), and authenticity of answers after config changes.

Multi-replica Redis requirements

More than one worker process or replica ⇒ Redis (or compatible) for a coherent cache and for shared rate limits / router state. Semantic backends need their vector store healthy on every request path. Production docs also use Redis for the spend transaction buffer at very high QPS (~1000+ RPS or 10+ instances)—adjacent to caching but the same shared-state theme. Optional enable_redis_auth_cache mirrors virtual-key auth into Redis so replicas share auth entries; pair with general_settings.user_api_key_cache_ttl (default 60s) so revocations converge.

LiteLLM can point coordination at a different Redis than the response cache via general_settings.coordination_redis. If you split them, size and alert both. If you keep one cluster, still configure both router_settings and litellm_settings.cache_params—environment variables alone do not attach every subsystem. Cluster/Sentinel/IAM topologies are documented under Redis and Valkey pages. Per-command cache client timeout is cache_params.socket_timeout (default 5s), distinct from REDIS_SOCKET_TIMEOUT.

Operational checklist: what breaks

• Multi-worker without Redis → fragmented caches and multiplied RPM limits.
• Semantic cache on coding agents → stale tool-call loops.
• Shared virtual key + default semantic scope → cross-user similar-prompt leakage.
• Embedding outage → up to timeout seconds of added latency per request before miss.
• Over-long TTL on factual content → confidently wrong cached answers.
• Only wiring cache Redis while skipping router_settings → weak shared limits still.
• Caching non-idempotent prompts → silent incorrect reuse.
• Expecting passthrough provider routes to honor LiteLLM cache → they may bypass it.
• Assuming /cache/ping green means semantic embeddings are healthy → test embedding path separately.
• When Redis is unhealthy, local caches diverge and limits loosen per worker—alert on Redis errors alongside LLM latency.

Practical recommendations

Start with exact-match Redis, conservative TTL, and caching limited to known-safe call types. Opt agent virtual keys out via metadata. Only introduce semantic cache for clearly single-shot workloads with a tested similarity threshold, embedding SLOs inside the timeout budget, and end_user scope if keys are shared. Load-test hit rate with your real prompt distribution—synthetic identical jokes in demos overstate savings. Ground every change in /cache/ping, response headers, and spend dashboards rather than hope. Pair cache metrics with LLM cost tracking so savings are measured per team and model, not assumed.

Related reading

• What Is an LLM Gateway and How Does It Work? — the architectural primer behind the gateway pattern LiteLLM implements
• Top 5 LiteLLM Alternatives in 2026 — how other gateways, including TrueFoundry, compare for production teams
• LLM Load Balancing — routing, fallbacks, and spreading traffic across model deployments

Conclusion

LiteLLM caching can meaningfully cut latency and provider spend—when traffic is genuinely repetitive and the cache is shared through Redis. Start with exact-match Redis and a conservative TTL, scope semantic caching narrowly to single-shot workloads, and keep agents out.

For a gateway that brings caching together with routing, rate limits, and observability inside your own infrastructure, learn how TrueFoundry’s AI Gateway approaches it →