Most LLM applications resend a large chunk of identical content on every single request — a long system prompt, a tool-definition list, a reference document — with only a small part (the user’s actual message) changing each time. Prompt caching lets the API skip re-processing that unchanged part, which is where most of the cost and latency in a typical request actually comes from.
How it works: prefix matching
Prompt caching lets an LLM API reuse previously computed model state for a matching prefix of a request — commonly represented internally as key-value (KV) tensors that let the model process subsequent tokens without recomputing everything before them. The reusable prefix can include system/developer instructions, tool definitions, conversation history, and other supported context; the exact ordering and matching rules depend on the provider and API.
The practical lesson is the same regardless of provider: keep frequently-reused content stable. Avoid putting a changing timestamp or user-specific detail at the start of a shared prompt when it can go later instead; keep tool definitions and their ordering consistent; preserve conversation history where the API supports reusing it. A change to the prompt can prevent reuse from the point of divergence onward — but the matching portion before that divergence may still be served from cache. Reuse isn’t guaranteed even when the prefix matches: eligibility, cache lifetime, request routing, and provider-specific rules all factor in.
Cache breakpoints
Caching can be automatic, explicit, or both, depending on the provider and model. Some APIs automatically identify eligible prefixes with no developer action needed. Others let you mark an explicit cache breakpoint to control which portion of a request should be reusable — Anthropic’s cache_control field works this way, while OpenAI and Gemini implement their own automatic-caching and eligibility rules that don’t require (though in some cases also support) an explicit marker.
Providers generally enforce a minimum token threshold before a prefix becomes cacheable, and that threshold varies — not just between providers, but between models from the same provider. The number of supported breakpoints, the cache lifetime (TTL), and cache-write pricing all vary the same way. Supplying a caching control doesn’t guarantee a short prompt gets cached — below the threshold, it’s just processed normally.
Why it can save money and time
Prompt caching can reduce input-processing cost and improve latency when an application repeatedly sends the same long prefix — but the effect size is workload- and provider-dependent, not a fixed number:
- Latency — reusing cached model state can reduce the work spent processing repeated input before generation starts, which may improve time-to-first-token (TTFT). How much depends on the workload and the provider’s implementation.
- Cost — cached input tokens can be substantially cheaper than uncached ones, with discounts of 90% or more available on some provider/model combinations — but that discount applies to the eligible cached tokens specifically, not automatically to the whole request. Cache writes often cost more than ordinary input processing, and pricing can also vary by the cache lifetime chosen. Total savings depend on prefix length, cache-hit rate, cache-write cost, output-token usage, and the specific model’s pricing.
For an application repeatedly sending a large, stable system prompt or tool-definition list, caching can meaningfully cut cost — but the way to know by how much is measuring actual token usage and billing, not assuming a headline discount applies to the full bill.
Confirming it’s actually working
Check the API’s usage metadata rather than assuming caching is active — the field names differ by provider. OpenAI reports cached input tokens in usage.input_tokens_details.cached_tokens; Anthropic reports usage.cache_read_input_tokens and usage.cache_creation_input_tokens; Gemini documents usage.total_cached_tokens.
If repeated, supposedly-identical requests show no cached tokens, the usual suspects are: the prefix is below the provider’s minimum cacheable length, something in the “stable” prefix is actually varying (a live timestamp, non-deterministic JSON serialization, a tool list whose order isn’t fixed), the cache entry expired, or the request got routed somewhere that doesn’t hold that entry. A single request showing zero cache-read tokens doesn’t by itself prove a serialization bug — check it across several repeated requests before concluding that.
The practical takeaway
Prompt caching is a genuinely useful optimization for an application that repeatedly sends long, similar prompts — keep the reusable context stable, order it to maximize prefix reuse, learn the specific provider’s caching rules, and measure actual cache-hit tokens and billing rather than assuming a headline discount applies. It can be a high-impact lever, but weigh the measured savings against other options too — shortening prompts, trimming unnecessary context, or a cheaper model — rather than treating caching as automatically the top of the list.
See also: RAG vs Fine-Tuning: Picking the Right Approach for a Client Project for the companion decision — whether long reference content belongs in the prompt (where caching helps) or baked into the model itself.