What Is Prompt Caching and How Does It Work?

Every time you call an LLM, it processes the whole prompt from scratch, including the parts that never change. If your prompt carries a long system message, a style guide, or a document that is the same on every call, the model reprocesses all of it every time, and you pay for all of it every time.

Prompt caching removes that waste. The provider stores the processed form of the repeated prefix after the first call, and on later calls it reuses that work instead of redoing it. You pay a reduced rate for the cached tokens, and the model reaches the new part of the prompt faster.

This guide explains what prompt caching is, how it works, how it differs from semantic caching, and how to get the most from it.

What is prompt caching?

Prompt caching is a provider-side feature that stores the processed state of a repeated prompt prefix so it does not have to be computed again on the next call. When you send a prompt, the model reads it token by token and builds internal state as it goes. Prompt caching saves that state for the prefix and replays it on the next request that starts with the same prefix.

It is like a presenter who gives the same introduction before every talk. Instead of rewriting and rehearsing the intro each time, they keep it ready and jump straight to the part that changes. Prompt caching keeps the fixed part of your prompt ready so the model can jump to the new part.

The key word is prefix. Caching applies to the stable beginning of the prompt. The moment the content differs, caching stops and the model processes the rest normally.

How does prompt caching work?

A prompt is split into the part that repeats and the part that changes. Caching applies to the first.

• Split the prompt. The repeated prefix, such as the system prompt and context, is separated from the variable part, such as the user's new question.
• First call, cache miss. On the first call the provider processes the whole prompt and stores the processed prefix in its cache.
• Later calls, cache hit. If the next prompt starts with the same prefix, the provider loads the stored state instead of reprocessing those tokens.
• Process only the new part. The model processes only the changed tokens after the prefix, then generates the response.
• Pay less, start sooner. Cached tokens are billed at a lower rate, and skipping them lowers the time to first token.

Caching is prefix-based and order-sensitive. The cached part has to appear at the start and be identical. Put the stable content first and the variable content last to get the most out of it.

Prompt caching vs. semantic caching

• Prompt caching. Caches the processed prefix within one model call. It still calls the model, but reprocesses less of the prompt, which lowers cost and the time to first token. Matches only an identical prefix.
• Semantic caching. Caches whole responses keyed by meaning. On a hit it skips the model call entirely and returns a stored answer. Matches requests that mean the same thing, even if worded differently.

They solve different problems and work well together. Prompt caching makes the calls you do make cheaper and faster. Semantic caching reduces how many calls you make at all.

What are the benefits of prompt caching?

• Lower cost. Repeated prefix tokens are billed at a reduced rate, which matters most when the fixed part of the prompt is large.
• Faster first token. Skipping reprocessing of the prefix shortens the time to first token, so responses feel quicker.
• Bigger context, cheaper. Long system prompts, tool definitions, and documents become affordable to send on every call.
• No quality trade-off. Unlike a cache that returns a stored answer, the model still generates a fresh response, so output quality is unchanged.

What are the limits of prompt caching?

• Exact prefix only. Any change to the cached prefix, even early in it, breaks the cache and forces a full reprocess.
• Short lifetime. Cached prefixes expire after a short time to live, often a few minutes, so sporadic traffic may not benefit.
• Minimum size. Providers usually require a minimum number of tokens before a prefix can be cached.
• Provider-specific. Support, pricing, and rules vary by provider, and not every model offers it.

How to get the most from prompt caching

• Put the stable content first, the system prompt, instructions, and shared context, and the variable content last.
• Keep the cached prefix byte-for-byte identical across calls, since any change invalidates it.
• Batch related calls close together in time so they land within the cache's time to live.
• Cache large, reused blocks like tool definitions and long documents, where the saving is biggest.
• Check your provider's minimum token size and pricing so you cache things that actually qualify.
• Measure cost and time to first token before and after, to confirm the cache is being hit.

Conclusion

Prompt caching reuses the work a model already did on a repeated prompt prefix, so you do not pay full price to reprocess the same system prompt or document on every call. It lowers cost and the time to first token without changing the output, as long as the prefix stays identical and the calls are close in time.

TrueFoundry's AI Gateway sits in front of your providers, so you can combine prompt caching at the model with gateway-level caching, routing, and rate limits in one place. Structuring prompts with the stable part first is a small change that pays off on any workload with a large, repeated context.