Your prompt cache is the biggest line on the bill
Coding agents mark their long, stable prefix (system prompt, tools, history) for caching, so it is billed at roughly 0.1× on later turns. Lose that prefix and the next turn pays full price to rebuild it.
What a cache rebuild looks like
In request logs, a warm conversation reads hundreds of thousands of cached tokens per turn. A rebuild shows cached reads collapsing to a lower breakpoint, followed by a large cache write on the next turn.
What a gateway must never do
Anything that changes the prefix forces a rebuild. OpenLLM forwards Anthropic-to-Anthropic requests byte-for-byte, carries cache breakpoints across OpenAI-to-Anthropic translation, and never injects a system prompt of its own.
- No gateway-added system prompt or steering text
- cache_control breakpoints preserved on content, messages and tools
- Fallback stops on requests that can't succeed, instead of cold-missing another model's cache
See it in your own data
The usage dashboard reports cached reads and cache writes per request, so a rebuild is visible instead of hidden inside a total.
Questions
Something else? Contact us