LLM Inference Pricing: Cache Hits and Cost Structure Explained

    LLM inference costs depend on more than published per-token rates. This article explains how cache hit rates, input-to-output ratios, and output length shape effective pricing, using hypothetical rates and practical calculations. It also outlines cache-friendly design and cost monitoring strategies for enterprise agents and lower-volume developers.

    When comparing large language model inference prices, it is easy to focus only on the published rate per million tokens. That rate matters, but it is only the nominal price. Actual spending depends on the workload—especially cache hit rates, input-to-output ratios, output length distributions, and request reuse.

    As a result, the same inference service can have very different effective prices for different users. Rather than focusing on a particular brand or product, this article examines a general technical question: when uncached input, cached input, and output have separate prices, how should developers calculate and optimize their actual costs?

    The examples below use these hypothetical rates:

    Token category Price per million tokens
    Output CNY 100
    Uncached input CNY 20
    Cached input CNY 2

    These figures illustrate the calculation method; they do not represent any particular service’s actual pricing. The central conclusion does not depend on these exact values: when cached input is substantially cheaper than uncached input, and output is substantially more expensive than input, workload characteristics create significant cost differences.

    1. Three Pricing Tiers Reflect Different Resource Requirements

    Inference billing can typically be divided into three categories: uncached input, cached input, and output.

    Uncached input tokens require input processing and model forward computation. Cached input tokens reuse intermediate states from a matching prefix, reducing repeated computation. Output tokens typically require autoregressive, token-by-token generation, which is highly sequential, latency-sensitive, and keeps compute resources occupied for longer.

    This helps explain a common pricing pattern: output costs more than input, and uncached input costs more than cached input.

    In the hypothetical rate card, uncached input costs 10 times as much as cached input, while output costs 50 times as much. These ratios matter. Cache hits are not merely a small discount; they are a major lever for reducing effective input prices. At the same time, the share of output tokens can substantially change the blended cost.

    Enterprise workloads that repeatedly use the same system prompts, tool descriptions, codebase summaries, or knowledge prefixes are more likely to achieve high cache hit rates. Developers whose prompts change with every request may see lower hit rates, leaving their effective input price closer to the uncached rate.

    2. Effective Input Price: Cache Hit Rate as a Cost Lever

    The effective input price is a weighted average:

    1. Effective input price =
    2. cache hit rate × cached input price
    3. + (1 - cache hit rate) × uncached input price

    Using the hypothetical rates:

    • At a 90% cache hit rate, the effective input price is 0.9 × 2 + 0.1 × 20 = CNY 3.8 per million tokens.
    • At a 20% cache hit rate, it is 0.2 × 2 + 0.8 × 20 = CNY 16.4 per million tokens.

    That is a difference of more than fourfold before output costs are included.

    For this calculation, the hit rate should represent the share of input tokens billed at the cached rate, rather than simply the share of requests that contain a cache hit.

    High hit rates do not happen automatically. They depend on stable prefixes, sensible context organization, cache lifecycle management, tenant isolation, and how requests are assembled. Frequent system prompt changes, dynamically assembled tool descriptions, and inconsistent context ordering can all reduce reuse.

    Caching also has engineering costs. Storage, invalidation policies, isolation, and security checks require resources and management. Optimization therefore means evaluating whether cache savings outweigh storage and operational overhead—not just maximizing a hit-rate metric.

    3. Input-to-Output Ratio Determines the Blended Cost

    Long contexts increase the weight of input costs, while long responses increase the weight of output costs.

    The blended price per million tokens can be calculated as:

    1. Blended price =
    2. (input tokens × effective input price
    3. + output tokens × output price)
    4. / (input tokens + output tokens)

    Here, all prices use the same unit: cost per million tokens. The result is a blended per-million-token price, not the total request bill.

    Consider an enterprise agent workload with an input-to-output ratio of 20:1 and a 90% cache hit rate:

    1. Blended price = (20 × 3.8 + 1 × 100) / 21
    2. ≈ CNY 8.4 per million tokens

    Now consider a lower-volume developer whose prompts vary between requests, with a 20% cache hit rate and a 5:1 input-to-output ratio:

    1. Blended price = (5 × 16.4 + 1 × 100) / 6
    2. ≈ CNY 30.3 per million tokens

    The first workload’s blended unit cost is roughly one-quarter to one-third of the second’s. A headline price comparison alone would miss this difference: effective cost depends on workload composition and the ability to reuse cached input.

    This leads to an important selection principle: compare the cost of completing a business task, not just the listed token rates.

    A task with a long but highly reusable input can cost less than a task with a short input and a long output. Similarly, controlled, structured responses can be less expensive than lengthy free-form responses.

    4. Why Output Usually Costs More

    Output tokens are typically generated through sequential decoding. This process cannot be parallelized as extensively as input prefill, and output length directly affects how long a request occupies resources and how much latency users experience.

    Even with batching and continuous batching, output generation remains a major constraint on resource utilization and responsiveness. Higher output prices are therefore understandable.

    Practical ways to manage output costs include:

    • Controlling response length.
    • Using structured output where appropriate.
    • Setting maximum output-token limits.
    • Summarizing before generating a longer response.
    • Breaking long responses into steps generated only when needed.

    These measures address a different cost lever from caching. High cache reuse lowers input costs, but it does not remove the cost of generating long outputs.

    5. Optimization Strategies for Different Workloads

    Enterprise Agent Workloads

    Enterprise agent developers should prioritize cache-friendly design:

    • Keep system prompts and tool descriptions stable.
    • Use a consistent context order.
    • Isolate caches by tenant and scenario.
    • Monitor cache hit rates and reasons for invalidation.
    • Build cost dashboards that attribute spending to business tasks.

    The goal is to make repeated context reusable while preserving appropriate isolation and security controls.

    Lower-Volume Developer Workloads

    Developers with less frequent or more varied requests can focus on:

    • Standardizing prompts with reusable templates.
    • Reusing conversation context where appropriate.
    • Reducing repeated copy-and-paste context.
    • Grouping similar requests for batch processing.
    • Routing tasks to different models according to complexity.

    The following pseudocode illustrates the pricing calculations:

    1. function effective_input_price(hit_rate, hit_price, miss_price):
    2. return hit_rate * hit_price + (1 - hit_rate) * miss_price
    3. function blended_cost(input_tokens, output_tokens, hit_rate, hit_price, miss_price, output_price):
    4. input_price = effective_input_price(hit_rate, hit_price, miss_price)
    5. total = input_tokens + output_tokens
    6. return (input_tokens * input_price + output_tokens * output_price) / total

    With input prices expressed per million tokens, blended_cost returns the blended price per million tokens rather than the total amount billed.

    General-purpose cloud capabilities such as logging, monitoring and alerting, and container platforms can support collection of request volumes, input and output token counts, cache hit rates, and task-level costs. Cost dashboards can then break spending down by tenant, scenario, and model-routing choice.

    The objective is not simply to obtain the lowest price for an individual request. It is to make costs observable, attributable, and actionable. Published rates describe nominal pricing; workload design and engineering practices determine the effective price.