LLM Inference Cost Optimization: Prompt Caching and Reasoning Effort Tiers

Why Inference Cost Has Become a Central Concern

As generative models move into large-scale production, the discussion has shifted from whether they can be used to whether they can be used affordably. A typical application cost structure can be broken into four parts: normal input tokens, cache-hit input tokens, output tokens, and the computation consumed by the inference process itself. Input-side tokens often account for a large share of the bill, especially when an application repeatedly carries system prompts, knowledge-base snippets, tool definitions, and long conversation histories. Repeated input tokens can multiply costs.

Many teams focus only on switching to a cheaper model, but they overlook two higher-leverage actions. The first is routing repeated context through a cache instead of recomputing it at full price. The second is avoiding excessive reasoning on simple tasks. These two actions correspond to the two core mechanisms discussed in this article: cache reads and reasoning effort tiers.

Cache Reads: An Underestimated Cost Lever

When processing input, large language models compute and store intermediate representations for token segments, commonly referred to as the KV cache. If the same prefix—such as a system prompt, a long document, or a tool schema—appears across multiple requests, recomputing it from scratch every time is pure waste. Prompt caching solves this by saving the computed prefix state. When a later request matches that prefix, the cached state can be reused, and only newly added content needs to be computed.

From a billing perspective, a common price structure looks roughly like this:

• Normal input tokens: full price, for example on the order of $10 per million tokens.
• Output tokens: usually the most expensive, for example on the order of $50 per million tokens.
• Cache write: slightly higher than normal input because additional state must be stored.
• Cache read: significantly lower than normal input.

The cache-read tier deserves special attention. An increasing number of platforms price cache reads at about one quarter of the base input price. In other words, re-reading already processed context can cost only one quarter of the original input price. If a workload can reliably hit the cache, the input side alone can be reduced substantially.

A rough estimation logic looks like this:

Total input cost ≈ uncached input tokens × input price + cache-hit input tokens × cache-read price
If cache hit rate = h, normal input price = P, cache-read price = P/4, then:
Average input price = (1 - h) × P + h × P/4 = P × (1 - 0.75h)
h = 0    → average price = P       (no savings)
h = 0.6  → average price = 0.55P   (about 45% savings)
h = 0.9  → average price = 0.325P  (about 67% savings)

This also explains why the impact of cheaper cache reads on the total bill is often underestimated. The higher the hit rate, the closer the savings come to the full 75% discount associated with cache reads.

Agentic Workloads Are Naturally Suited to Cache Hits

Why is this mechanism especially relevant today? Because agentic workloads are almost tailor-made for cache hits.

Agentic applications usually follow a loop of reading, thinking, calling tools, and reading again. In each loop, the model repeatedly sees the same batch of context: fixed system instructions, tool or function definitions, task background descriptions, and a growing history whose prefix remains unchanged. In other words, each request grows in a pattern where the old prefix remains unchanged and new content is appended at the tail.

This leads to two results:

1. The cached prefix—system prompt, tool definitions, and stable background—can achieve nearly 100% hit rate throughout the session.
2. As the number of turns increases, the proportion of reused tokens rises, and the marginal cost drops.

In long-running tasks and multi-turn tool-calling scenarios, cache hit rates can easily exceed 80%. Combined with a cache-read price around one quarter of the normal input price, the effective cost of total input can fall by 40% or more. A useful rule of thumb is therefore: if a task is multi-turn, uses long context, and has a stable prefix, cache reuse should be treated as a default optimization, not an optional one.

There are several engineering details that must be handled carefully:

• Keep the prefix as stable as possible and place dynamic content, such as timestamps or random IDs, at the end of the prompt. Otherwise, caching can be broken.
• Caches usually have a time-to-live, or TTL. If they sit idle for too long, they may expire. Frequent, continuous calls are generally more cost-effective than scattered calls.
• A cache hit requires a byte-for-byte identical prefix. Any subtle change can invalidate the cache from that point onward.

Reasoning Effort Tiers: From Whether to Think to How Deeply to Think

The second cost lever is reasoning effort, also referred to as a reasoning budget. Modern reasoning models often support a period of internal thinking before producing an answer. Longer thinking usually improves accuracy on complex tasks, but it also increases latency and token consumption.

A key trend is that many platforms have removed the option to completely disable reasoning and instead offer multiple reasoning levels ranging from low to high, often around five tiers. The reasoning behind this is practical: completely disabling reasoning can significantly degrade quality. Rather than forcing users to choose between two extremes, it is better to provide a set of smoothly adjustable levels.

The rough positioning of different tiers is as follows:

• Lowest tier, such as minimal or low: little to no explicit reasoning. Suitable for tasks that can be answered at a glance, such as classification, extraction, formatting, and simple rewriting. It offers the lowest latency and cost.
• Low-to-medium tier: suitable for tasks requiring a small amount of multi-step judgment, such as conditional information organization and lightweight code completion.
• Medium tier, such as medium: the default tier. It covers most routine question answering and writing, balancing quality and cost.
• Medium-high or high tier, such as high: suitable for multi-step reasoning tasks such as math, complex debugging, and structured analysis.
• Highest tier, such as max: reserved for the hardest open-ended problems, such as long-chain planning, difficult code refactoring, and deep research. It comes with the highest latency and token consumption.

The core principle is that reasoning effort should match task difficulty, rather than being maxed out for every request. Applying the highest tier to all requests is one of the most common sources of wasted cost in inexperienced teams.

Practical Strategies for Choosing Five Reasoning Tiers

The following decision framework can be used directly as a default strategy for selecting reasoning tiers by task type:

Task Type	Recommended Tier	Reason
Classification, tagging, field extraction	Lowest tier	No reasoning needed; prioritize throughput and low cost
Formatting, translation, rewriting	Low tier	Rules are clear; light reasoning is enough
Routine question answering, summarization, document generation	Medium tier	Balance of quality and cost
Multi-step analysis, routine code generation	Medium-high or high tier	Requires some step-by-step reasoning
Mathematical proofs, complex debugging, long-task planning	Highest tier	High-value problems worth the extra cost

Three recommended practices for implementation are:

1. Use medium tier by default and escalate only when needed. Run requests at the medium tier first, and move to a higher tier only when a task fails or quality is not satisfactory. Avoid maxing out the tier for every request.
2. Use a proxy or routing layer. Distribute requests to different reasoning tiers based on task labels or a preliminary classification. Simple traffic should go to low tiers, while complex traffic should go to high tiers.
3. Combine reasoning tiers with caching. The reasoning tier determines how deeply the model thinks, while caching determines how cheap repeated context becomes. Using both together forms a complete cost optimization strategy.

The following is a conceptual illustration of routing and caching strategy:

def handle_request(task, context_prefix):
    # 1. Select reasoning effort based on task difficulty.
    if task.type in ("classify", "extract", "format"):
        effort = "low"
    elif task.type in ("qa", "summarize"):
        effort = "medium"
    elif task.type in ("debug", "math", "plan"):
        effort = "high"
    else:
        effort = "medium"
    # 2. Place the stable prefix first to hit the cache; place dynamic content at the end.
    prompt = build_prompt(
        static_part=context_prefix,   # System prompt, tool definitions, knowledge snippets (stable)
        dynamic_part=task.payload,    # Dynamic content such as timestamps or random IDs goes last
    )
    return model.generate(
        prompt=prompt,
        reasoning_effort=effort,      # Reasoning tier
        cache=True,                   # Enable prefix caching
    )

Engineering Considerations for Context Windows and Model Lifecycles

In addition to the two core levers, several engineering parameters can have a long-term impact on both cost and stability:

• Context window: Input windows for mainstream reasoning models have reached hundreds of thousands or even millions of tokens, while maximum output lengths are often around 100,000 tokens. A larger window can hold more context in a single request, but holding more does not mean it should be included. Overly long context increases cost and may dilute the model’s attention to key information, so it should be trimmed according to actual needs.
• Knowledge cutoff: Model training data has a cutoff date. For time-sensitive domains, external retrieval or tools are required; the model’s memory alone cannot be relied upon.
• Model retirement and migration: Vendors usually provide clear retirement timelines. For example, some versions may be committed to not being retired earlier than one year after release. Engineering teams should abstract model calls behind an adapter layer to avoid large-scale rework when a model version is retired.
• Multi-platform consistency: The same model is often provided through model platforms from multiple cloud providers. Interfaces and pricing may differ slightly. Cost estimates should be based on the actual platform being used.

Cost Optimization Checklist

Based on the analysis above, here is a checklist that can be used directly:

1. Identify high-cache-hit scenarios. If there is a stable long prefix plus multi-turn calls, enable caching and fix the system prompt and tool definitions at the front.
2. Keep the prefix byte-for-byte stable. Place dynamic content at the end to avoid breaking the cache. Pay attention to cache TTL and try to use frequent, continuous calls.
3. Match reasoning tier to task difficulty. Use medium tier by default, lower it for simple tasks, and raise it for difficult tasks. Use a routing layer instead of globally maxing out the tier.
4. Control context length. Include only necessary context, inject retrieval results only when needed, and watch out for doubled costs and attention dilution from overly long context.
5. Abstract the model access layer. Wrap billing, reasoning tier, cache switches, and related settings in a unified adapter layer to make horizontal comparison and fast switching easier.
6. Build cost observability. Separately monitor the proportions of normal input, cache-hit input, output, and reasoning tokens, and use data to drive continuous optimization.

In summary, cache reads and reasoning effort tiers compress cost from two dimensions: the input side and the reasoning side. For multi-turn, long-context agentic applications, cache reuse provides the most direct savings. For reasoning tiers, reasonable routing determines the cost ceiling per unit of quality. When both are handled correctly, most teams can achieve order-of-magnitude improvements in their bills while keeping quality largely unchanged.