MnemoverseDocs

Appearance

Edward Izgorodin·Published Aug 16, 2026

The Stable Prefix Contract: Why Agent Prompt Caches Miss ​

TL;DR

  • A prompt cache key is derived from the exact prompt prefix. Anthropic, OpenAI, Gemini, vLLM, and SGLang build the key differently, but the invariant is the same: change one early token and everything after it stops matching.
  • A stable prefix is an engineering contract, not a hint. Providers state it in their own docs; none of the five systems covered here promises to canonicalize your prompt before hashing.
  • Six recurring breakers show up across the documented agent misses: timestamps, tool-list changes, nondeterministic JSON, memory or RAG injected into the cached region, invisible prefix mutations, and compaction.
  • The cost is measurable. One cross-provider evaluation reports 41–80% API cost cuts from caching; LangChain reports 49–80% in its own agent runs; both under their stated test conditions.

A prompt cache key is the identity used to look up cached prompt state, derived from the exact prompt prefix plus provider-specific routing or hash metadata. A stable prefix is the longest initial segment of a prompt that stays byte-for-byte identical across requests that should share a cache entry.

That is the whole contract. Providers expose it through different APIs. Serving engines store it in different data structures. The failure mode is shared: change the front of the prompt and the cached suffix stops being reusable.

This article covers the key contract and the six breakers that cause misses. For hit-rate economics, context layout, and the broader playbook, see KV-Cache Hit Rate: Why the Context Window Is Memory You Pay Rent On.

Stable prefix prompt cache contract: exact prefix identity ​

Anthropic states the rule directly: "Cache hits require 100% identical prompt segments, including all text and images up to and including the block marked with cache control." Its cache prefixes are created in the order tools, system, then messages, and "Changes at each level invalidate that level and all subsequent levels" (Anthropic prompt caching docs).

OpenAI uses the same premise: "Cache hits are only possible for exact prefix matches within a prompt." Its mitigation is prefix layout: put static instructions and examples at the beginning, and put variable user-specific information at the end. OpenAI also warns that changes to tool descriptions, parameter schemas, schema keys, or ordering can reduce reuse (OpenAI prompt caching guide).

Gemini's implicit caching applies the same discipline, though Google does not publish an implicit TTL or hit-increment granularity. The advice is to put "large and common contents at the beginning" of the prompt and to send requests with a similar prefix close together in time (Gemini caching docs).

OpenAI adds an operational caveat that matters in production: cache reuse is best-effort. A hit depends on an identical prefix, the cached content still being available, and the request reaching a machine that holds the matching entry (OpenAI prompt caching guide).

So a stable prefix is necessary. It is not always sufficient.

Prompt cache key mechanics: Anthropic, OpenAI, Gemini, vLLM, SGLang ​

Anthropic: cumulative prefix hash with breakpoints. On each request, the system computes a prefix hash at the cache_control breakpoint and checks for a matching entry. The hash is cumulative, so changing any block at or before the breakpoint produces a different hash. Up to four breakpoints are allowed per request; on a miss the system walks backward one block at a time, within a 20-block lookback window, searching for prior cache writes rather than merely stable content (Anthropic prompt caching docs). Minimum cacheable length is model-dependent, from 512 to 4,096 tokens (for example, 1,024 for Sonnet 4.5 and 4,096 for Haiku 4.5, per the docs as of August 2026); shorter prompts are silently processed without caching. The default TTL is 5 minutes, refreshed at no extra cost on each use; a 1-hour TTL is available. Cache writes bill at 1.25x base input (5-minute) or 2x (1-hour), and cache reads at 0.1x.

OpenAI: automatic caching with a two-level routing key. Caching is automatic for prompts of 1,024 tokens or more; that is a strict minimum on GPT-5.6 and later, while on earlier models the minimum ranges from 1,024 to 2,048 tokens and "Cache hits occur in increments of 128 tokens." Requests are routed to a machine by prompt_cache_key, with a hash of the initial prompt prefix as a secondary key, and OpenAI advises keeping traffic per key to roughly 15 requests per minute. The cacheable prefix can include messages, images, tool definitions, structured output schemas, and audio. For GPT-5.6 and later, the guide documents cached input at 0.1x the uncached rate, cache writes at 1.25x, a 30-minute exact TTL via prompt_cache_options.ttl, and explicit breakpoints (OpenAI prompt caching guide).

Gemini: implicit and explicit modes with different guarantees. Implicit caching is on by default for Gemini 2.5 and later (minimum 2,048 tokens on 2.5 Flash and Pro, 4,096 on the 3.x models listed), with hits reported in usage.total_cached_tokens and no published TTL (Gemini caching docs). Explicit caching via client.caches.create(), available through the generateContent API, is the only mode with a stated cost-saving guarantee: cached rates are about 10% of input rates on the listed 2.5 models, and storage is billed by TTL duration (Gemini explicit caching, Gemini pricing). The sharp edge is compatibility: explicit CachedContent cannot be combined with system_instruction, tools, or tool_config in the same request, which breaks LangChain's bind_tools() pattern outright (langchain-google issue 1528).

vLLM: hash-chained blocks. vLLM's Automatic Prefix Caching stores KV cache per fixed-size token block. In the v0 design the key was hash(prefix tokens + block tokens); in v1 each block's key is hash(parent block's hash, this block's tokens, extra keys), where extra keys include LoRA IDs, multimodal hashes, and cache salts. Since v0.11 the default hash is SHA-256 (vLLM prefix caching design).

text
vLLM hash chaining (v1):

[ Block 0 ] system prompt
  key = hash(tokens_0)
      |                         parent hash feeds the next key
[ Block 1 ] static instructions
  key = hash(key_0, tokens_1)
      |
[ Block 2 ] first dynamic turn   <- one changed token here
  key = hash(key_1, tokens_2')      changes this key and every key below it

vLLM caches only full blocks: in its worked example, when a prefix diverges mid-block, "only the first 2 blocks (8 tokens) hit the cache, because the 3rd block only matches 2 of 4 tokens." Because each key contains its parent's hash, one early changed token forward-invalidates every downstream block (vLLM prefix caching design). The stakes are physical memory: a single LLaMA-13B sequence can hold up to 1.7 GB of KV cache; block sharing cuts the memory use of parallel sampling and beam search by up to 55%, up to 2.2x throughput for those workloads per the blog, and overall the paper reports 2–4x throughput against FasterTransformer and Orca at the same latency (vLLM blog, PagedAttention paper).

SGLang: a radix tree over token sequences. RadixAttention maps token sequences (keys) to KV tensors (values) in a radix tree whose edges carry variable-length token runs; nodes split at divergence points (SGLang blog). Eviction is LRU over leaves with reference counting, and the scheduler serves the longest matched prefix first (SGLang paper).

text
SGLang radix tree:

            [ root ]
               |
      "system + tools prefix"
               |
           [ node A ]
           /        \
  "...turn 1a"     "...turn 1b"
       |                |
   [ node B ]       [ node C ]

In Chatbot Arena production the paper reports cache hit rates of 52.4% (LLaVA-Next-34B) and 74.1% (Vicuna-33B), and up to 6.4x throughput against the baseline systems it benchmarks (SGLang paper).

Different machinery. Same prefix identity.

Why prompt cache misses happen: the six breakers ​

BreakerWhy it missesMitigation
Timestamps in the prefixone changed token invalidates everything after itmove time out of the cached region, or truncate precision
Tool-list changes, MCP orderingtools sit first in the hash hierarchysort tools deterministically; defer-load schemas
Nondeterministic JSONkey order changes the byte sequenceserialize with sorted keys, fixed formatting
Memory/RAG in the cached regionrewrites the system-prompt bytesinject dynamic content at the turn level, order it stably
Invisible prefix mutationsprovider renders config into the promptpin web search, citations, thinking config, tool_choice
Compactionrewrites history from the compaction pointcompact rarely, at high thresholds

1. Timestamps in the cached prefix ​

Manus calls out a common mistake: putting a timestamp, especially one precise to the second, at the beginning of the system prompt. Even a single-token difference can invalidate the cache from that token onward (Manus engineering blog, vendor self-reported).

Anthropic's docs walk through the same failure with a timestamp block carrying the cache_control marker: the timestamp differs, the breakpoint hash differs, and the backward walk finds no earlier written entry (Anthropic prompt caching docs). The Claude Code team likewise lists detailed timestamps in the static system prompt among the cache breakers it has observed (Claude Code prompt caching lessons).

2. Tool-list changes, including MCP nondeterminism ​

Anthropic's Claude Code team calls changing the tool set mid-conversation one of the most common ways people break prompt caching (Claude Code prompt caching lessons). The mechanics explain why: tools sit first in Anthropic's hash hierarchy, so tool edits invalidate everything after them (Anthropic prompt caching docs).

MCP adds a practical source of nondeterminism: the tools/list section of the MCP specification (2025-06-18) defines no return-ordering guarantee, and newer drafts recommend, but do not require, a deterministic order (MCP spec: server tools). The openclaw project handled this by sorting MCP tools alphabetically before building requests (openclaw PR 58037):

text
Unsorted MCP output (cache miss):
  turn 1: [tool_read_db, tool_write_file]  -> prefix hash A
  turn 2: [tool_write_file, tool_read_db]  -> prefix hash B   (miss)

Sorted MCP output (cache hit):
  turn 1: [tool_read_db, tool_write_file]  -> prefix hash A
  turn 2: [tool_read_db, tool_write_file]  -> prefix hash A   (hit)

Behavior on tool changes is version-dependent and worth measuring rather than assuming: in one controlled experiment, dropping the last tool of about eight left GPT-5.2 with 2,560 cached tokens (76%), while GPT-5.5 kept zero in both tested variants, dropping the first tool or the last (practitioner measurement).

Claude Code's mitigation is deferred tool loading: name-only stubs with defer_loading: true stay in the stable prefix, and full schemas load only when the model selects a tool (Claude Code prompt caching lessons).

3. Nondeterministic JSON serialization ​

Manus warns that many languages and libraries do not guarantee stable key ordering when serializing JSON objects, which can silently break the cache (Manus engineering blog, vendor self-reported). OpenAI independently warns that changes to schema keys or ordering reduce cache reuse (OpenAI prompt caching guide).

Providers do not promise to canonicalize your JSON before hashing. Canonicalization is the client's job: serialize everything that enters the cached prefix with a fixed field order, fixed formatting, and no volatile values. In Python that is json.dumps(obj, sort_keys=True); most languages have an equivalent. This is not a provider API requirement, it is a client-side guardrail for the exact-match contract.

4. Memory or RAG injected into the cached region ​

A memory provider that rewrites the cached system prompt is a cache killer. In hermes-agent, the Honcho memory integration rewrote a system-prompt layer every N turns; at cadence 1 this produced a cache miss on 100% of turns, with a full prefill of about 20K tokens each time. The fix moved dynamic memory into the user-turn layer, keeping the cached system prompt byte-identical for the session (hermes-agent issue 13631).

RAG has a subtler version of the same problem: adjacent queries may retrieve overlapping evidence, but retrieval-score ordering scrambles it differently each time, so set overlap does not become reusable prefix overlap. Cache-aware evidence reordering recovered roughly 20–33% of median TTFT in the CacheWeaver evaluation (arXiv:2606.19667). LangChain names updating a memory and loading a new skill or tool as cache busters in Deep Agents (LangChain Deep Agents blog, vendor self-reported).

5. Invisible prefix mutations ​

Some breakers never appear in your application code. Anthropic documents that toggling web search or citations modifies the system prompt; that the thinking configuration (mode and budget_tokens) is rendered into the prompt, so changing it invalidates message blocks; that tool_choice changes invalidate the messages-level cache; and that adding or removing images changes the cacheable prefix (Anthropic prompt caching docs).

An agent's request body can look identical across turns while the effective prefix sent to the model changes under the hood. If the rendered prefix differs, the cache key differs.

6. Compaction resets the prefix ​

Compaction is necessary in long sessions, and it changes the prompt from the compaction point onward. A Codex CLI case study describes the static cacheable prefix as everything from system instructions through the first user message, and notes that each compaction changes the prefix and forces a miss; it recommends compaction thresholds of 150–200K tokens and gives a worked example arriving at about 57% cost reduction at an 85% hit rate (secondary source, illustrative calculation) (Codex CLI prompt caching writeup). LangChain lists compacting a conversation as a cache buster as well (LangChain Deep Agents blog).

Prompt caching invalidation has measurable cost ​

SourceCost reductionTTFT improvementEvidence type
"Don't Break the Cache" (arXiv:2601.06007), 500+ agent sessions across OpenAI, Anthropic, Google41–80%13–31%benchmark paper
LangChain Deep Agents (blog): claude-haiku-4-5 −77%, gpt-5.4-mini −80%, gemini-3.5-flash −49%49–80%not reportedvendor self-reported
Stable vs perturbed prefix, ~2,258 ms vs ~3,714 ms (Ankit Sinha)not reported~39% latencypractitioner-measured

The same benchmark paper adds a warning: naive full-context caching can paradoxically increase latency; the winning strategy places dynamic content at the end and keeps dynamic tool results out of the cached region.

Two operational signals show how seriously production teams take this. Manus calls KV-cache hit rate "the single most important metric for a production-stage AI agent," with an input-to-output token ratio around 100:1 in its own workload (vendor self-reported). Anthropic's Claude Code team runs alerts on its prompt cache hit rate and declares SEVs when it drops too low (Claude Code prompt caching lessons).

The engineering conclusion is short: do not treat prompt caching as a provider-side optimization you may or may not receive. Treat the stable prefix as part of your agent's interface.

Common questions ​

What is a stable prefix in prompt caching? ​

A stable prefix is the longest initial segment of a prompt that stays byte-for-byte identical across requests, so the provider or serving engine can reuse the cached KV state computed for it.

Why did my prompt cache miss? ​

Prompt cache misses usually happen because an early token changed, the cache entry expired or was evicted, or the request reached infrastructure that does not hold a matching entry. Provider caching is best-effort even when the prefix is identical.

What is a prompt cache key? ​

A prompt cache key is the identity used to look up cached prompt state. In practice it is derived from the exact prompt prefix: a cumulative hash at Anthropic, a routing key plus prefix hash at OpenAI, a chain of block hashes in vLLM, a radix-tree path in SGLang.

Do MCP tools break prompt cache? ​

MCP tools can break prompt cache when tool lists arrive in nondeterministic order or when tool schemas change, because tool definitions sit inside the cached prefix. The MCP spec does not guarantee listTools() ordering, so clients must sort tools deterministically.

What causes prompt caching invalidation? ​

Prompt caching invalidation is caused by changes in the cached prefix: timestamps, tool-list changes, JSON key-order instability, dynamic memory or RAG insertion, hidden configuration changes rendered into the prompt, and conversation compaction.

Sources ​

Provider docs ​

Engineering blogs ​

Papers ​

Issues, PRs, and measurements ​

Externalized memory helps here for one narrow reason: volatile facts that live outside the prompt do not churn the cached prefix. That is where a persistent-memory API supports prompt-cache discipline, without pretending to replace provider caches.

— Edward Izgorodin · Last updated 2026-08-16

— Mnemoverse is a persistent-memory API for AI agents. Free key: console.mnemoverse.com · Docs: Getting Started