October 16, 2026
|
5 minutos de lectura
LiteLLM Caching: Redis, Semantic Cache, What Breaks
Por Abhilash Sanap
Published: October 16, 2026
Diseñado para la velocidad: ~ 10 ms de latencia, incluso bajo carga
¡Una forma increíblemente rápida de crear, rastrear e implementar sus modelos!
- Gestiona más de 350 RPS en solo 1 vCPU, sin necesidad de ajustes
- Listo para la producción con soporte empresarial completo
What LiteLLM cache means
A LiteLLM cache stores LLM responses so a later identical (or, for semantic mode, similar) request can skip the provider call. That cuts latency and provider spend when traffic is repetitive—and creates correctness bugs when it is not. This article covers exact-match caches (in-memory, Redis/Valkey, S3/GCS, disk), semantic caches (Redis, Valkey, Qdrant), TTL and keys, multi-replica requirements, invalidation, and the failure modes platform engineers hit in production. Scope note: this is LiteLLM’s application-level response cache, not OpenAI/Anthropic provider-side prompt caching (a separate feature with its own configuration). Caching is one of several controls an LLM gateway applies on the request path, alongside auth, rate limits, and routing.
Supported cache backends at a glance
Official proxy caching docs list: redis (also Valkey/ElastiCache/Memorystore via Redis protocol), redis-semantic, valkey-semantic, qdrant-semantic, s3, gcs, local (in-memory), and disk. Redis is the right default past a single worker. Exact-match caches key on a hash of the whole request. Semantic caches embed the prompt and serve the closest match above a similarity threshold—suited to single-shot prompts and a poor fit for agentic traffic.
Cache hit versus miss (exact-match)
Figure 1. Exact-match flow: hash the request, lookup, hit return or miss → provider → store.
On a hit, responses can carry an x-litellm-cache-key header useful for debugging and for /cache/delete. With cache: True and no supported_call_types override, caching applies to /chat/completions, /completions, /embeddings, /audio/transcriptions, /rerank, /v1/responses, and /v1/messages (confirm on your version). On /v1/responses, exact-match keys on the request body—so previous_response_id versus an inlined conversation are different keys, and each multi-turn step is its own entry.
TrueFoundry AI Gateway ofrece una latencia de entre 3 y 4 ms, gestiona más de 350 RPS en una vCPU, se escala horizontalmente con facilidad y está listo para la producción, mientras que LitellM presenta una latencia alta, tiene dificultades para superar un RPS moderado, carece de escalado integrado y es ideal para cargas de trabajo ligeras o de prototipos.
Diseñado para la velocidad: ~ 10 ms de latencia, incluso bajo carga
Try now.
Inscríbase
¿Cómo se puede evitar que los costos de GenAI se disparen a gran escala?
Gartner Hype Cycle for Platform Engineering 2026
One Layer of Control for All AI
Tabla de contenido
Controle, implemente y rastree la IA en su propia infraestructura
Summarize with
Descubra más
No se ha encontrado ningún artículo.
.webp)
.png)
No se ha encontrado ningún artículo.
Blogs recientes
.png)
.png)
.png)
.png)
.png)
.png)
.png)
Preguntas frecuentes
What is LiteLLM cache and how does it work?
LiteLLM cache is the response cache built into the LiteLLM proxy and SDK: it stores LLM responses so a later identical request—or a similar one in semantic mode—skips the provider call. You enable it with litellm_settings.cache: true and choose a backend in cache_params (local, disk, redis, s3, gcs, or a semantic backend). It cuts latency and provider spend on repetitive traffic, and it is separate from provider-side prompt caching.
When should I use Redis versus the in-memory LiteLLM cache?
Use the in-memory (local) or disk cache only for local development or a single process, because neither is shared between workers or replicas. Use Redis—or a Redis-protocol service such as Valkey, ElastiCache, or Memorystore—as soon as you run more than one worker or pod. Four workers with local caches behave like four separate caches, and hit rates drop roughly by the worker count.
What breaks when you use semantic cache on multi-turn or agent traffic?
Semantic cache is designed for single-shot prompts; on multi-turn traffic it replays stale responses. Each new turn is the previous conversation plus a small delta, so embeddings stay around 0.99 similar, and assistant tool_calls are not part of the embedded text—agents end up repeating the same tool call. Raising similarity_threshold does not reliably fix it; exclude agent keys with "cache": {"no-cache": true} metadata or use exact-match type: redis.
Does the TrueFoundry AI Gateway integrate with my existing observability stack?
Yes. The gateway is OpenTelemetry-compliant and plugs into Grafana, Datadog, Prometheus, or your preferred stack. Every request is traced from prompt to model execution, so metrics such as latency, cost, and errors land next to your existing dashboards.
