Prompt caching

Every call re-processes the whole prompt. If the first part is identical across calls (system prompt, rules, few-shot examples, tool definitions), the provider can keep the processed prefix and reuse it. Cached tokens cost far less (around 10% of normal on Anthropic) and the first token arrives sooner.

The cache matches an identical prefix. Everything after the cached part can change freely. One changed character *before* the marker breaks the cache for everything after it.

• Anthropic: opt in with cache_control: {"type": "ephemeral"} on the last content block of the static part. It is an API field, not text in the prompt. Writes cost a little extra (about 1.25x), reads are cheap. Up to 4 breakpoints.
• OpenAI: automatic for prompts over about 1,024 tokens. You only keep the prefix stable.
• Limits: a minimum prompt length (about 1,024 tokens for most models) and a short lifetime (about 5 minutes, refreshed on each hit; longer options exist). It pays off only with repeated traffic.

system=[
  {"type": "text", "text": RULES_AND_EXAMPLES,      # static
   "cache_control": {"type": "ephemeral"}},         # cache everything up to here
  {"type": "text", "text": f"Today is {date}, user is {name}."},  # dynamic, after the marker
]

• A timestamp, user name or request id sits before the marker. Move it after.
• Tool definitions or examples are reordered between calls.
• The prompt is under the minimum length, or calls are too far apart for the cache to live.
• Verify in the response usage: cache-read tokens should be above zero.