If you're building an LLM-powered app that sends the same system prompt, tool definitions, or long context on every request, prompt caching is the single highest-leverage optimization you can make.
What gets cached
Most providers cache at the token level, starting from the beginning of the prompt. A typical structure looks like:
[system prompt] [tool definitions] [conversation history] [new user message]
If the first three sections are identical to a previous request, the provider can reuse the internal computation for those tokens instead of reprocessing them — you pay a much lower rate for the cached portion.
Why order matters
Caching is prefix-based. Put the parts of your prompt that change most often (user messages) at the end, and the stable parts (system instructions, tool schemas, long reference documents) at the beginning.
Stable, reused across requests → System prompt, tool defs, docs
Changes every request → User's latest messageIf you interleave stable and dynamic content, you break the cache prefix and lose the benefit entirely.
A concrete example
An agent that re-sends a 2,000-token tool definition block on every turn of a 20-turn conversation is paying full price for those 2,000 tokens twenty times over — 40,000 billed tokens for content that never changed. With caching, only the first turn pays full price; the remaining nineteen pay the cached rate.
When it's not worth it
- Single-shot requests with no repeated context get no benefit.
- Prompts that change substantially between requests (no stable prefix) won't cache well.
- Very short prompts may not clear the minimum token threshold some providers require.
Practical checklist
- Structure prompts with static content first, dynamic content last.
- Keep system prompts and tool definitions byte-for-byte identical across requests when possible.
- Monitor your provider's cache-hit metrics, not just token counts — a low hit rate usually means something in your "stable" section is silently changing.
Common mistakes
- Embedding a timestamp, request ID, or session ID inside the system prompt or tool definitions. It looks like a small detail, but it makes every single request's "stable" prefix unique, which silently defeats caching entirely while still looking correct.
- Serializing tool definitions from an object without a stable key order. If your JSON serializer doesn't guarantee consistent key ordering, byte-for-byte identical tool schemas can render as different strings between requests — breaking the cache prefix match without any visible error.
- Assuming caching is automatic. Most providers require the request to be structured correctly (stable content first) — caching isn't something that happens regardless of prompt shape, it's a property you have to design for.
- Optimizing token count instead of cache-hit rate. A shorter prompt that breaks caching can end up costing more than a longer one that caches well, especially across a long multi-turn conversation.
Caching across a multi-turn conversation
The benefit compounds specifically in conversational or agentic use cases, where each new turn re-sends the entire prior conversation as context:
Turn 1: [system] [tools] [msg 1] → full price
Turn 2: [system] [tools] [msg 1] [reply 1] [msg 2] → cached prefix + new suffix
Turn 3: [system] [tools] [msg 1] [reply 1] [msg 2] [reply 2] [msg 3] → cached prefix + new suffix
As long as the growing conversation history is only ever appended to — never edited or reordered — each turn's entire prior context remains a valid cached prefix, and only the newest message at the end is billed at full price. This is why prompt caching matters disproportionately more for longer conversations and agent loops than for simple single-shot completions — the savings compound with every additional turn built on the same unmodified history.
Caching and tool-heavy agents specifically
Agentic systems that call many tools tend to have the largest stable prefix of any common LLM use case — a full tool schema catalog can easily run several thousand tokens, repeated on every single step of an agent loop regardless of which specific tool actually gets called. This makes agent loops the single highest-leverage place to apply the ordering principle above: keep the full tool definition block first and completely stable across every step, and let only the growing conversation/action history vary at the end. Skipping this on an agentic system is one of the more expensive mistakes to make, since the stable block is both large and repeated on every step of what can be a long-running loop.
Cache invalidation isn't just about content — it's about time too
Even a perfectly structured, byte-identical prefix eventually falls out of cache after a period of inactivity (the specific window varies by provider, typically minutes rather than hours). This means cache benefits are strongest for genuinely active, back-to-back request patterns — a user actively chatting with an agent — and weakest for sporadic, spread-out usage, where the cache has already expired between requests regardless of how well the prompt is structured. Bursty traffic patterns (a batch job hitting the same system prompt across many rapid requests) are a particularly strong fit for exactly this reason.
Debugging a low cache-hit rate
- Log the exact byte length (or a hash) of your "stable" prefix per request — if it changes between calls that should be identical, you've found the leak.
- Check for non-deterministic ordering in anything programmatically generated: tool lists pulled from a database, feature flags, or dynamically assembled system prompts are common culprits.
- Verify your provider's minimum cacheable prompt length — very short system prompts may fall under the threshold and never qualify for caching regardless of how well-structured they are.
- Re-check after any prompt template change. A caching setup that worked can silently regress the moment someone edits the system prompt and reintroduces a dynamic value near the top.
Provider-specific cache TTLs and exact discount rates change over time and vary by model — treat this article as explaining the underlying mechanism, and check your specific provider's current documentation for their actual numbers before estimating cost savings.
Related reading
- Understanding RAG: Retrieval-Augmented Generation Explained — shares tags: ai, programming (same category).
- Async Python with asyncio: A Practical Introduction — shares tags: programming.
- Big O Notation Without the Math Panic — shares tags: programming.
- Clean Code Principles That Actually Hold Up in Practice — shares tags: programming.
- Core Web Vitals Explained: What Actually Affects Your Score — shares tags: programming.
