LLM API Cost Optimization: How to Choose Cache Reads and Reasoning Effort

1. Look beyond price per million tokens: two structural cost levers

When people talk about lowering LLM inference costs, their first instinct is usually to push down the price per million tokens. In real workloads, though, the structural factors that actually decide the size of your bill are usually two others: how cache reads are priced, and which reasoning effort level you select.

A single request’s total cost can be roughly broken down into four parts:

total cost ≈ uncached input tokens × input price
           + cache-hit tokens × cache read price
           + cache-write tokens × cache write price
           + output tokens × output price (including thinking tokens)

Worth emphasizing: when a provider announces a price cut, it is often not the base per-token rate that moves, but the cache read price being cut substantially. For short question-and-answer traffic that makes no difference, but for long-context, multi-turn agentic workloads the cache read portion is typically the bulk of the bill — so end-to-end costs drop noticeably.

2. Why cache reads matter so much

To understand the value of caching, start with how transformer inference works. During autoregressive generation, the model caches the Key/Value tensors of its attention layers for every token it has already processed — the KV cache. When a new request shares the same prefix as an earlier one (system prompt, tool definitions, few-shot examples, retrieved documents, conversation history), that prefix’s KV cache can in principle be reused instead of recomputed.

The “prompt caching” capability offered by serving platforms productizes this reuse mechanism: you mark a given prefix as cacheable, and subsequent cache hits are billed at a price far below the standard input rate.

Why does this matter so much for agent workloads? Because an agent runs as a loop: at every step it has to feed a large block of context (system prompt + tool schemas + accumulated history + retrieved documents) back to the model, while the only genuinely new content between steps is the small amount of newly generated or newly retrieved material. Cache hit rates are therefore naturally very high — the overwhelming majority of tokens are “old faces.” In that situation, cutting the cache read price pays off in multiples.

The following illustrative figures show the mechanism (pricing structures differ across providers, and these numbers are for illustration only):

Cost component	Illustrative price (per 1M tokens)	Note
Standard input	100	baseline
Output	400	typically several times the input rate
Cache write	125	slightly above standard input
Cache read	10	about one-tenth of the input rate

Assume an agent whose context is 50,000 tokens per step with a 90% cache hit rate. Only the remaining 10% is billed at the standard input price; everything else goes through the low cache read rate. The industry has widely observed that workloads with a high share of repeated context can see end-to-end costs fall by more than 30%, and agentic workloads can save close to half.

3. Reasoning effort: the trade-offs across five levels from low to max

The newer generation of reasoning models generally treats “thinking” as a tunable parameter rather than a simple on/off switch. The common approach is five levels of reasoning strength (for example low, medium, and high, up to max), and the option to turn reasoning off has been removed. The reason: even seemingly simple tasks become more reliable with a modest amount of thinking, and keeping behavior consistent — while preventing users from stepping into pitfalls with extreme configurations — is a deliberate product decision.

The key point is the cost mechanism: tokens produced during the reasoning (thinking) phase are billed as output. That means setting reasoning effort too high drives up both cost and latency, while setting it too low drops quality on hard problems. It is not a “higher is always better” knob, but a budget that has to be matched to the task.

Typical trade-offs at each tier:

• Low: short latency, low cost; suited to deterministic tasks such as extraction, classification, formatting, and summarization.
• Medium: balances cost and quality; suited to code refactoring, explanations, and moderately complex reasoning.
• High to max: many thinking tokens and long latency; suited to hard tasks such as complex debugging, architecture design, mathematical proofs, and multi-step planning.

4. A selection framework for different task types

When choosing, it is worth evaluating four dimensions at once: task difficulty and number of steps, latency SLA, cost budget, and the cost of being wrong. Scenarios where mistakes are expensive (production changes, financial calculations, and the like) should proactively raise the reasoning effort level.

A practical mapping example:

def choose_reasoning_effort(task_type: str) -> str:
    if task_type in {"extract", "classify", "format", "summarize"}:
        return "low"          # deterministic tasks, minimize cost
    if task_type in {"refactor", "explain", "moderate_code"}:
        return "medium"       # balance quality and cost
    if task_type in {"debug_complex", "architect", "prove", "plan_multistep"}:
        return "high"         # hard tasks, preserve quality
    return "max"              # highest difficulty or highest-risk scenarios

There is one more practical trick inside the agent loop: run routine steps at the low tier by default, and escalate to a higher tier for a retry only when a step fails, validation does not pass, or the model reports low confidence in its own answer. This “escalate on demand” strategy keeps the average cost down while preserving success rates.

5. Cache-friendly engineering practices

To capture the caching benefit, how you organize prompts matters more than the model itself:

1. Put the stable prefix first: place static content — system prompt, tool definitions, long reference documents — at the very front, and keep it unchanged across calls.
2. Put dynamic content last: timestamps, user input, and real-time retrieval results belong at the end.
3. Keep formatting identical: one extra space, one changed line break, or one rephrased word changes the prefix hash and invalidates the cache.
4. Manage cache lifetime: caches typically expire after a period of inactivity, so agent loops should reuse them before expiry and keep a reasonable call cadence.
5. Account for the cost of writing: the first write carries a premium, so only content that gets reused enough times is worth caching.

An illustrative message layout (the general idea of a “cacheable prefix”):

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},         # large and static -> cache first
    {"role": "user",   "content": RETRIEVED_DOCS},        # large and semi-static
    {"role": "user",   "content": f"Current question: {question}"}  # small and dynamic -> keep last
]

Most providers implement this by having you mark a prefix to declare it cacheable, and the core principle is the same everywhere: make the stable part as long as possible, and push the volatile part as far to the end as possible.

6. Cost estimation and optimization checklist

Before launch, it is worth doing a quantitative estimate: plug “cache hit rate × per-component unit price” into the formula above, compute the cost range for a typical request and for a worst-case request, and then use those figures to set caps on reasoning effort and on maximum output tokens so that individual requests cannot run away.

A self-check list for implementation:

• Is static content organized into a stable prefix, with dynamic content at the end?
• Has any timestamp, random ID, or other cache-breaking variable slipped into the prefix?
• Is reasoning effort tiered by task difficulty, rather than set to a high tier across the board?
• Does the agent loop implement an “escalate only on failure” retry strategy?
• Are cache hit rate, cost per task, and latency distribution being monitored?
• Is there a hard cap on maximum output / reasoning tokens?

7. Closing

Cutting cost without losing efficiency comes down to understanding where the money actually goes. The low price of cache reads rewards the engineering discipline of “reusing context well,” while the tiering of reasoning effort tests your judgment in “matching budget to task.” Cloud platforms, including Baidu AI Cloud, commonly expose tunable capabilities of this kind — caching and reasoning strength — in their LLM offerings, though the exact knobs, pricing, and availability differ by provider and region. Teams that get prompt structure, task tiering, and monitoring right can bring inference costs down to a reasonable range without sacrificing quality.