# Reducing costs and improving cache reuse

Source: https://platform.lyrouter.com/ai_gateway/docs/cost-practice/en
Last updated: 2026-10-19

Agents and coding assistants resend the system prompt, tool definitions and the whole history on every turn, so input Tokens are often most of the cost, and the prompt cache hit rate largely decides what you pay for Tokens. The main ways to cut costs are to keep request prefixes stable and keep a session on the same channel, then to trim unnecessary output and context. After each change, verify the effect with the cache hit rate and cost in the logs and usage analytics.

## Find where the money goes

In **Usage analytics**, set "Perspective" to "Cost" and check the cost share by model and API key; the "Cache Hit Rate" card on the "Overview" and the "Cache Hit" column of the "By Model" table show how often the cache is hit. Models with concentrated cost and a low hit rate are the best targets for optimization. The four Token types of a specific request are in the [request logs](https://platform.lyrouter.com/ai_gateway/docs/logs/en) detail, and how costs are calculated is in [Estimating costs and cache pricing](https://platform.lyrouter.com/ai_gateway/docs/estimate/en).

(Screenshot: Switch the perspective in usage analytics to compare usage, cost and cache hit rate across models.)

With a cache hit rate of `h` and a ratio `r` of the model's cache read price to its input price, the input cost is about `(1 − h) + h × r` times what it would be with no hits at all. The smaller `r` is, the more a higher hit rate saves.

## Keep request prefixes stable

The upstream matches the cache by byte prefix: from the first byte that differs, everything after it is charged as a miss. Put the most stable content first. From front to back:

1. **Fixed role and rules**: the most stable, so they go first.
2. **Tool definitions**: unchanged within a session, in a fixed order.
3. **Examples and project context**: unchanged within a project.
4. **Environment info, date, etc.**: change often, so place them as late as possible.
5. **Conversation history**: append only, never rewrite.
6. **This turn's new message**.

- **No changing values in the prefix**: do not put request IDs, trace IDs, timestamps accurate to the second or random numbers at the start of the system prompt. If you must include them, put them in the last message, and keep dates to the day.
- **Do not change tools mid-session**: adding, removing or adjusting tools changes everything after the tool definitions. To restrict tools temporarily, do it in the prompt or in the calling logic; put tools added on demand at the end of the tool list.
- **Append history and replay it as is**: do not reassemble or reformat the history on every turn; send tool call arguments back exactly as the model returned them in the previous turn.
- **No need to worry about JSON key order**: before sending a request upstream, the gateway sorts the keys of JSON objects in the request body into a fixed order, including JSON inside tool call arguments (arguments containing 15 or more consecutive digits are left as they are, so that numbers are not rewritten). The order of messages, tools and content blocks is not changed, so the client has to keep it stable.

## Keep a session on the same channel

The cache lives on a specific provider channel, and switching channels means building it again. When the workspace's "Multi-channel routing strategy" is "Lowest Price" (the default for new workspaces) or "Load balanced", the gateway keeps calls from the same API key to the same model on the same channel:

- The binding lasts 30 minutes and is renewed by every call. It moves to another channel when the channel returns 429, 408, a 5xx or a connection failure, but not when the channel changed because the request itself was malformed.
- "Lowest Price" picks the channel with the lowest current estimated price only for new sessions or for calls after the binding has expired; changes in the price ranking do not move sessions in progress.
- "Lowest Latency" and "Highest Throughput" have no session affinity. Virtual models whose "Channel sort strategy" is "Price first" or "Load balanced" keep sessions in the same way.
- `lyrouter/auto` chooses a model from the content of each request, so one session may switch to different models; pin a single model ID for long sessions.

The caller only needs to keep each session or Agent on the same API key and the same model; no extra parameters are required. If the client sends an `X-Session-Id` request header, the gateway forwards it unchanged; use one fixed value per session instead of changing it on every request. The settings are covered in [Channels and routing](https://platform.lyrouter.com/ai_gateway/docs/routing/en).

## Using explicit caching (cache_control)

Implicit caching is done automatically by the upstream and needs no parameters. Explicit caching marks breakpoints in the request with `cache_control`, and only takes effect when the upstream that actually serves the call uses the Anthropic protocol:

| Endpoint called | Protocol of the serving upstream | How `cache_control` is handled |
| --- | --- | --- |
| `/v1/messages` | Anthropic | Markers on `system`, message text blocks and tool definitions are kept |
| `/v1/chat/completions` | Anthropic | Markers on message text parts and tool definitions are kept |
| Any endpoint | OpenAI-compatible | All markers are removed; only implicit caching applies |

- When you call `/v1/messages`, markers placed on image blocks or on `tool_result` blocks themselves are not forwarded; put breakpoints on text blocks.
- Several `system` text blocks are merged into one, and only the last marker is kept.
- When a marker contains `"ttl": "1h"`, the gateway sets the `anthropic-beta` request header to the value the 1-hour cache requires; any `anthropic-beta` value the client sent is overwritten.
- Anthropic allows at most 4 breakpoints per request, and a prefix shorter than the model's minimum cacheable length is not cached.

The example below marks a long, fixed system prompt as a cache breakpoint (replace `MODEL_ID` with a model served by an Anthropic-protocol upstream, such as the Claude series; the environment variables are set up in [Your first model call](https://platform.lyrouter.com/ai_gateway/docs/quickstart/en)):

```bash
curl "$LYROUTER_BASE_URL/v1/messages" \
  -H "x-api-key: $LYROUTER_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MODEL_ID",
    "max_tokens": 1024,
    "system": [{"type": "text", "text": "You are a contract review assistant. The review rules are as follows... (the same long text every time)", "cache_control": {"type": "ephemeral"}}],
    "messages": [{"role": "user", "content": "Check whether the liability for breach in clause 3 is balanced between the parties."}]
  }'
```

When Explicit cache write appears in the logs for the first call, the upstream has accepted the marker; for requests with the same prefix sent within the next few minutes, that part is recorded as Input (cache hit).

## Client notes

- **Claude Code**: if the hit rate is exactly 0, upgrade Claude Code first. Some versions write a `cch=` line that changes with every request at the very start of the system prompt, so the prefix differs from the first byte. Adjusting the available tools mid-session also makes everything after the tool definitions miss. Setup is covered in [Claude Code](https://platform.lyrouter.com/ai_gateway/docs/claude-code/en).
- **Your own applications**: make sure the SDK or framework does not insert timestamps or request IDs into every request, and does not reorder tools or messages on every turn.

## Cut unnecessary usage

- **Control output**: limit output length with `max_tokens`. For tasks that do not need deep thinking, lower the thinking effort or turn thinking off; the thinking process counts as Output.
- **Control context**: clear history unrelated to the task. Very long sessions may fall into a more expensive long-context tier, so summarize periodically and start a new session.
- **Choose models and sources**: compare the cost and results of different models and sources on the same task over the same time range; see [Choosing a model](https://platform.lyrouter.com/ai_gateway/docs/choose/en) and [Platform models or BYOK](https://platform.lyrouter.com/ai_gateway/docs/byok-choice/en). Budgets and quotas in Credits are covered in [Budgets, quotas and rate limits](https://platform.lyrouter.com/ai_gateway/docs/quota/en).

## Verify the effect

After a change, check in "Token usage" of the log detail whether Input (cache hit) grows over consecutive calls, then compare the cache hit rate and cost before and after the change in **Usage analytics** over periods of the same length. When you calculate the hit rate yourself, weight it by Tokens: Σ Input (cache hit) ÷ Σ (Input (cache miss) + Input (cache hit)); do not average the hit rates of individual calls. If the hit rate is still low, work through [Unexpected costs and cache misses](https://platform.lyrouter.com/ai_gateway/docs/cache-errors/en) item by item.

(Screenshot: Watch how cache-hit input changes across call turns in the log detail to verify the optimization.)