Semantic Caching for LLMs: Exact vs Similar Requests

Built for Speed: ~10ms Latency, Even Under Load

Blazingly fast way to build, track and deploy your models!

Handles 350+ RPS on just 1 vCPU — no tuning needed
Production-ready with full enterprise support

Get Started with Truefoundry Now Talk to the Expert

Caching sounds simple until language gets involved. Traditional caches ask whether two requests are identical. AI applications often care whether two requests mean the same thing. “How do I reset my password?” and “I forgot my password—what now?” are different strings but may deserve the same support answer.

That is the idea behind semantic caching: embed a query, compare it to previously cached queries, and return a prior response when similarity is high enough. The win can be dramatic because a cache hit removes the model call entirely. The danger is equally obvious: a near-match can be semantically close while still requiring a different answer.

Operating Principle and Key Takeaways

The operating principle: cache equivalence is a product decision, not an embedding score.

Key Takeaways

Prompt caching and response caching are different layers. Provider prompt caching reuses prompt-prefix computation while the model still generates a fresh completion; gateway response caching can skip the model call entirely on a hit.
Exact-match avoids semantic false hits, but it is not automatically correct. A byte-for-byte identical request can still return a stale or wrongly scoped cached answer if the underlying truth, authorization context, or cache boundary changed.
Similarity threshold is not enough. In TrueFoundry semantic caching, the last message is compared semantically while the rest of the request envelope is hashed and must match exactly. Application state that is not present in that request — such as downstream tenant, entitlement, evidence version, or policy revision — still needs an explicit cache boundary.
Isolation is layered. TrueFoundry automatically scopes cache entries to the calling user or virtual account; optional custom namespaces partition further inside that scope. If one virtual account fronts multiple downstream tenants or end users, namespace them explicitly.
TTL should follow freshness, not convenience. Policy answers and product status have very different expiration requirements, and version-based invalidation is often safer than waiting for time alone.
TrueFoundry provides exact-match and semantic gateway caching plus cache observability. The application still decides which request classes are safe to reuse and which state changes the correct answer.

1. Three Caches, Three Different Jobs

Cache Comparison Table

Cache	What is reused	Match	Model still runs?
Provider prompt cache	Prompt-prefix computation	Provider-specific prefix rules	Yes
Exact gateway cache	Complete model response	Identical complete request within the same cache scope	No on hit
Semantic gateway cache	Complete model response	Similar last message + exact match on the remaining request envelope + same cache scope	No on hit

These are three conceptual caching jobs, not necessarily three separate deployed caches. In TrueFoundry, semantic caching is a superset of exact-match caching: semantic mode can also return exact-match hits, so you do not need to run both gateway modes at once. Choose exact-match when only identical-request reuse is acceptable; choose semantic when you explicitly accept similarity-based reuse. Provider prompt caching can compose with either on gateway misses, where the model provider still receives the request.

TrueFoundry cache metrics dashboard — Official TrueFoundry Cache Metrics view: hit rate, cost savings, cache errors, and lookup latency are visible as first-class operational signals.

The semantic match

‍

TrueFoundry AI Gateway delivers ~3–4 ms latency, handles 350+ RPS on 1 vCPU, scales horizontally with ease, and is production-ready, while LiteLLM suffers from high latency, struggles beyond moderate RPS, lacks built-in scaling, and is best for light or prototype workloads.

Built for Speed: ~10ms Latency, Even Under Load

Schedule your Demo Now