What Is Prompt Caching and How Does It Work?

Many LLM applications send the same large block of text over and over: a long system prompt, a set of instructions, a document the user keeps asking about. Every time that text goes to the model, the model processes it again from the beginning, which costs time and money. Prompt caching removes that waste by saving the processed form of the repeated part and reusing it.

The result is faster, cheaper responses for any workload with a stable, repeated context. Instead of paying to re-read the same prefix on every request, the application pays once and then reuses it, which is why prompt caching has become a standard optimization for production LLM systems.

This guide explains what prompt caching is, how it works, when it helps, and what to watch out for.

What is prompt caching?

Prompt caching is an optimization that stores the model's internal processing of a prompt prefix so it can be reused on later requests that share that prefix. When a new request arrives whose beginning matches a cached prefix, the model skips reprocessing that part and starts generating from where the new, unique content begins.

Think of it like a lecturer who teaches the same introduction every day. Rather than preparing it from scratch each time, they reuse the same slides and jump straight to what is new. Prompt caching lets the model reuse the work it already did on the shared opening and spend effort only on what changed.

The key idea is reusing computation. The cached item is not just the text but the processed representation of it, so the saving is in the work avoided, not merely in storage.

How does prompt caching work?

The model processes a prefix once, stores the result, and reuses it when later prompts start the same way.

• First request. The model processes the full prompt, including the long shared prefix.
• Cache the prefix. The processed state for that prefix is stored, keyed to its exact content.
• Next request. A new prompt arrives that begins with the same prefix.
• Cache hit. The model reuses the stored state instead of reprocessing the prefix.
• Generate. The model processes only the new content and produces the response faster.

Because the cache is keyed to the exact prefix, even a small change at the start breaks the match. The most cacheable design puts the stable content, system prompt and fixed context, first, and the variable content last.

What does prompt caching save?

The benefit shows up in three ways. Latency drops because the model skips reprocessing the prefix, which especially shortens time to first token. Cost drops because providers typically charge less for cached prefix tokens than for fresh ones. Throughput improves because each request does less work, so the same hardware serves more traffic.

These gains scale with how large and how repeated the prefix is. A short prompt sent once gains little; a long context reused across thousands of requests gains a great deal, which is why caching matters most for high-volume applications with heavy shared context.

When does prompt caching help?

• Long system prompts. A big instruction block reused on every request caches well.
• Document Q&A. Repeated questions about the same document reuse its processed context.
• Few-shot prompts. Fixed examples at the start of each prompt are cacheable.
• Chat history. A stable conversation prefix can be reused as the chat grows.
• High request volume. The more the same prefix repeats, the more caching pays off.

What are the limitations?

• Exact match needed. Any change to the prefix, even small, causes a cache miss.
• Cache lifetime. Cached prefixes expire, so very infrequent reuse may not hit.
• Prompt ordering. Benefits require stable content first and variable content last.
• Not universal. One-off or highly variable prompts gain little from caching.

When should you use prompt caching?

Use it whenever your application repeatedly sends a large, stable prefix, which is common for assistants with long system prompts, document tools, and few-shot setups. The larger the shared context and the higher the request volume, the more caching improves both speed and cost.

For short, unique prompts sent occasionally, caching adds little. But structuring prompts so the stable part comes first, and enabling caching where the provider supports it, is usually an easy win for production workloads.

Best practices for prompt caching

• Put stable content, system prompt and fixed context, at the start of the prompt.
• Keep variable, per-request content at the end so the prefix stays constant.
• Reuse the exact same prefix text so the cache reliably hits.
• Measure latency and cost with and without caching to confirm the gain.
• Be aware of cache lifetimes so you know when reuse still benefits.
• Use a gateway to apply and monitor caching consistently across models.

Conclusion

Prompt caching reuses the processed form of a repeated prompt prefix so the model avoids recomputing it on every request. For applications that resend a large, stable context, it cuts latency and cost and raises throughput, with the gains growing as the prefix gets larger and more repeated. The main requirement is an exact prefix match, which rewards putting stable content first.

TrueFoundry's AI Gateway lets teams take advantage of prompt caching across models and monitor its effect on latency and cost in one place. As LLM applications send ever-larger shared contexts, caching is a direct way to make them faster and cheaper.