Semantic Caching: When Similar Questions Should Share an Answer

Caching sounds simple until language gets involved. Traditional caches ask whether two requests are identical. AI applications often care whether two requests mean the same thing. “How do I reset my password?” and “I forgot my password—what now?” are different strings but may deserve the same support answer.

That is the idea behind semantic caching: embed a query, compare it to previously cached queries, and return a prior response when similarity is high enough. The win can be dramatic because a cache hit removes the model call entirely. The danger is equally obvious: a near-match can be semantically close while still requiring a different answer.

1. Three Caches, Three Different Jobs

These are three conceptual caching jobs, not necessarily three separate deployed caches. In TrueFoundry, semantic caching is a superset of exact-match caching: semantic mode can also return exact-match hits, so you do not need to run both gateway modes at once. Choose exact-match when only identical-request reuse is acceptable; choose semantic when you explicitly accept similarity-based reuse. Provider prompt caching can compose with either on gateway misses, where the model provider still receives the request.

The semantic match

new query
   ↓
embedding
   ↓
nearest cached queries
   ↓
similarity ≥ threshold?
   ├─ no → call model → cache response
   └─ yes → verify cache scope + exact state match → return cached response

The key step is the one people skip in diagrams: verify the complete equivalence boundary. In TrueFoundry, semantic similarity applies only to the last message; the rest of the request parameters must hash-match exactly, and cache entries are automatically scoped to the calling user or virtual account. But application state that never enters the request — for example a downstream tenant behind one shared virtual account, an entitlement lookup, an evidence revision, or an external account status — cannot influence the cache unless you encode it in the request, namespace, version boundary, or cache-eligibility policy.

Namespaces

Partition caches wherever the correct answer can differ across a boundary. In TrueFoundry there are two layers: user/virtual-account isolation is automatic, and an optional namespace partitions further inside that scope. The second layer matters when one virtual account serves multiple downstream tenants, end users, environments, prompt versions, or other contexts that must never share cached responses. A shared cache is appropriate only when the underlying answer is genuinely shared.

TTL and invalidation

A cached answer about “how to configure SSO” may stay correct for days. “Is service X currently degraded?” may become wrong in minutes. Use domain freshness to set TTLs, and invalidate aggressively when the underlying policy/content changes.

Observability

Measure hit rate, avoided cost, end-to-end latency, cache lookup overhead, false-hit reports, and cache errors. High hit rate is not automatically good; an over-broad threshold can manufacture an impressive metric while degrading answer correctness.

Design the equivalence class before the threshold

The central semantic-cache question is not “what cosine threshold should we use?” It is “which requests are allowed to share an answer?” That equivalence class may be narrower than semantic similarity. Two support questions can mean the same thing but require different answers because the users belong to different plans, regions, tenants, or product versions. The cache should encode those boundaries before similarity is even considered.

A practical design is to separate semantic identity from state identity. Semantic identity is the part allowed to vary in wording. State identity is everything that must be equivalent before reuse is safe. TrueFoundry implements a concrete version of this: it embeds the last message, while model, prior messages, temperature, and the rest of the request parameters are hashed and must match exactly. It also automatically isolates entries by user or virtual account. What it cannot infer is application state that is absent from the request; if locale, downstream tenant, entitlement, evidence revision, tool availability, or policy version changes the answer, represent that state in the request or namespace, version the cache, or mark the route ineligible for reuse.

Choose thresholds from error cost, not aesthetics

There is no universal “good” similarity score. TrueFoundry documents 0.9 as a starting point, with stricter thresholds for high-precision use cases, but that is an operating default rather than a correctness guarantee. A customer-support FAQ may tolerate broader semantic matching than a pricing, entitlement, or policy workflow. More importantly, no threshold can repair a missing state boundary: two questions can be embedding-close while differing on an entity, date, tenant, permission, or policy version that changes the answer. Tune the threshold against a labeled set of pairs — same-answer versus different-answer — only after those structural boundaries are encoded.

Track false positives separately from misses. A miss costs latency and tokens; a false positive can return the wrong answer without invoking the model at all. That asymmetry usually means production thresholds should bias toward precision first, then widen only when review data shows the extra hits are genuinely interchangeable.

Cache the evidence boundary, not just the wording

If an answer depends on changing documents or tools, tie cache validity to the evidence version. A knowledge-base deployment ID, policy revision, catalog version, or data freshness epoch can be represented in the request, an explicit namespace, or another invalidation/version boundary. This avoids a subtle failure mode where a perfectly matched question returns an answer generated from yesterday's evidence after the underlying source changed.

For agentic systems, be especially careful with cached intermediate decisions. Reusing “call tool X with arguments Y” is more consequential than reusing a prose FAQ answer. In most cases, semantic response caching belongs on stable informational steps, while actions continue through live policy and authorization.

Economics should include lookup overhead

Semantic caching adds embedding generation plus cache/vector lookup, whether that work is locally hosted or managed for you. That overhead is often small relative to an expensive generation, but it is not free. Measure the end-to-end result: cache lookup latency, embedding infrastructure or provider cost where applicable, hit rate, avoided generation cost, and correctness impact. TrueFoundry exposes average latency added by cache lookups in its Cache Metrics view; use measured overhead rather than assuming the cache is automatically faster for every route. A low-cost model with short responses may not justify semantic caching even when the hit rate looks attractive.

2. Where TrueFoundry Fits: Cache at the Gateway Boundary

TrueFoundry's AI Gateway caching supports exact-match and semantic response caching. Current docs distinguish that from provider prompt caching: gateway hits return the stored response without invoking the model, while provider prompt caching still generates a completion.

For semantic caching, TrueFoundry documents embedding-based similarity on the last message while hashing the rest of the request parameters; those non-semantic parameters must match exactly. Cache isolation is also two-level: entries are automatically scoped to the calling user or virtual account, and an optional custom namespace can partition further within that scope — for example when one virtual account represents multiple downstream tenants or environments. Response headers expose cache status, the original cached trace ID, and, for semantic hits, the similarity score. Those details matter because they make cache behavior inspectable rather than an invisible optimization.

The Metrics Dashboard documents a Cache Metrics view with total cache requests, cache-hit percentage, total cost saved, cache errors, and average latency added by cache lookups. That is the right operational feedback loop for deciding whether caching is economically useful. Correctness still needs an application-side signal — for example reviewed false hits or route-specific answer-quality evaluation — because hit rate and cost savings alone cannot tell you whether two requests should have shared an answer.

One implementation detail is deployment-specific. TrueFoundry SaaS manages the caching infrastructure and currently documents OpenAI text-embedding-3-small as the default semantic-cache embedding model. Self-hosted deployments can configure the embedding model, and the documented on-prem cache store is Redis or a Redis-compatible store such as Valkey. This matters for performance and data-boundary reviews: the semantic-cache path includes both the store and the embedding model, so evaluate where each runs in the deployment you actually operate.

3. Boundaries, Stated Plainly

Semantic caching is a poor fit for highly personalized answers, rapidly changing data, stochastic creative work, or consequential decisions where a near-match is unacceptable. Exact-match caching avoids semantic false hits, but it is only appropriate when the complete request is identical and the cached answer remains valid under the current freshness, authorization, evidence, and policy boundary.

Caching should be an explicit product contract: these request classes are reusable under these state boundaries for this long. Everything else is a miss by design.

References

• TrueFoundry: Caching — Exact and Semantic — cache types, last-message semantic matching, exact hashing of remaining parameters, automatic user/virtual-account isolation, namespaces, response headers, Redis/on-prem setup, and embedding-model configuration.
• TrueFoundry: Chat Completions — Prompt Caching — provider prompt-caching abstraction and provider-specific handling.
• TrueFoundry: Metrics Dashboard — Cache Metrics — requests, hit percentage, cost saved, errors, and lookup-latency overhead.
• OpenAI: Prompt Caching — cached prompt-prefix processing and cached-token accounting.
• Anthropic: Prompt Caching — automatic/explicit prompt-prefix caching and cache-control behavior.

Semantic similarity is not a correctness guarantee. Exact-match is not a freshness guarantee. TrueFoundry automatically isolates cache entries by user or virtual account; custom namespaces partition further inside that scope. Cache eligibility, application-state boundaries, invalidation/version rules, and false-hit review remain application decisions.

‍