What Is Prompt Caching (and How Much Does It Save)?

Quick answer
Prompt caching stores the model’s processed representation of a prompt prefix so a later request with the same prefix does not pay full input cost for it again.
It works only when the prefix is byte-stable and comes first: system instructions, tool definitions, and reference documents are good candidates; the user’s question must come last.
A cache hit usually bills the reused input at a discount and a cache write may cost more than normal input, so caching pays off when reuse is frequent and the prefix is long.
Cache lifetime is short — minutes to an hour depending on provider and settings — so low-traffic prompts rarely benefit.
Caching does not reduce output tokens, which are usually the larger cost in generation-heavy workloads.

How does prompt caching work?

A language model processes input tokens into an internal state before generating. Prompt caching stores that state for a prefix of the prompt, keyed by the exact token sequence. When a later request begins with the same sequence, the provider can reuse the stored state instead of recomputing it, which reduces both latency and the billed input cost.

The provider does not understand your intent, so the match is literal. A single changed character in the system prompt, a reordered tool list, or a timestamp at the top of the prompt invalidates the prefix. That is why the stable material must be placed first and the variable material last.

How do you structure a prompt to benefit?

Order the prompt from most stable to least stable: static instructions, tool or function definitions, then long reference material that rarely changes, then the conversation history, then the current request. Everything before the current request is the candidate prefix. Anything that changes per request must come after the cached section.

Injecting the current date, a request ID, or user-specific context at the top is the most common mistake, because it makes every prompt unique and every cache miss. Move those values to the end, or keep them in a separate message that follows the cached block.

Stable prefix (cacheable)
  1. System instructions, versioned and unchanged between deploys
  2. Tool definitions in a fixed order
  3. Product documentation or schema used for grounding

Variable suffix (not cacheable)
  4. Retrieved passages for this request
  5. Conversation turns
  6. The user's current question, including any timestamp

How much does it actually save?

The saving depends on the share of input tokens that are reused and the provider’s pricing for cache writes and reads. If ninety percent of a long prompt is a stable document and you send it on every request, the reused portion dominates the bill and the discount has a large effect. If the prompt is short or rarely repeated, the write cost can exceed the saving.

Model it rather than assuming. Take a representative request, count input tokens in the stable prefix and in the variable suffix, multiply by the provider’s per-token prices for normal input, cache write, and cache read, then apply your expected hit rate. The result is usually decisive: either the prefix is large and repeated, or caching is not worth the complexity.

When does caching not help?

When traffic is too sparse to hit the cache window, when prompts are short, when the prefix changes on every deploy, or when output tokens dominate. Caching is an input-side optimisation; a feature that generates long answers pays mostly for output, and no cache changes that.

It also does not help when the model or provider changes frequently, because the cache is tied to a model version. During evaluation, when prompts and models change daily, the hit rate is near zero. Turn caching on for the stable production path, not for the experiment.

Where this bites vibecoders
A generated prompt often starts with a timestamp, the user’s name, and a freshly serialised JSON blob, then puts the long instructions at the end. That order defeats caching before it starts. Moving the instructions and tool definitions to the front and the per-request values to the back is a small refactor that can change the cost curve.

Where AI coding assistants get this wrong

Putting a timestamp or request ID at the top of the prompt.
Serialising tool definitions in a different order on each call.
Rebuilding the system prompt from a template that includes dynamic values.
Enabling caching for a low-traffic endpoint and paying write costs for no hits.
Assuming prompt caching reduces output token cost.
Measuring savings on input alone while ignoring the cache-write premium.

Checklist

Split the prompt into stable prefix and variable suffix.
Remove timestamps and IDs from the prefix.
Keep tool definitions ordered and stable.
Measure the prefix’s share of total input tokens.
Compare cache write, read, and normal input prices for your provider.
Track hit rate after enabling caching.

FAQ

Does prompt caching change the model’s answer?

No. The cached state represents the same prefix the model would have processed anyway, so output should be equivalent for the same input. If answers change after enabling caching, the usual cause is that the prompt itself changed, such as a reordered prefix or a different model version.

How long does a cache last?

It varies by provider and configuration, typically minutes to about an hour, and it can be evicted earlier under load. Treat the cache as an optimization, never as storage. Any design that requires a cache hit to be correct will fail unpredictably.

Is caching worth it for a chatbot?

Often yes, because the system prompt and tool definitions repeat on every turn, and conversation history grows. The saving is largest when the fixed prefix is long. It is smallest for short prompts with little fixed context, where the write premium may not be recovered.

Does caching affect privacy or data retention?

The provider stores a representation of your prompt prefix for the cache lifetime, which is a data-handling consideration separate from training use. Check the provider’s retention terms for cached content, especially if the prefix contains customer data, and avoid putting sensitive material in a long-lived shared prefix.