LLM Inference Cost Estimation and Cache Hit Rate: A Practical Guide to Token Billing

    This article explains why comparing a single price per token can lead to the wrong conclusion about LLM inference cost. It introduces two consistent cost models, walks through scenario calculations under high and low cache hit rates, and examines the factors that influence cache hit rate in production. It also covers engineering practices for improving cache hits, package quota evaluation, tier selection, hybrid on-demand and package strategies, reusable pseudocode, and a pre-selection checklist. Readers will learn how to model real cost from billing rules, traffic structure, and scheduling capability rather than from unit price alone.

    1. Real Cost Is Not the Same as Unit Price: From Billing Items to Cost Structure

    In LLM inference services, a common but misleading intuition is that comparing cache price, input price, or output price is enough to determine which option is cheaper. Real bills are usually driven by multiple factors, and a single dimension can easily lead to misjudgment.

    LLM services generally bill by tokens, and the tokens in a request can be divided into three categories:

    1. Cache-hit portion: If the request prefix, system prompt, context fragment, or content already cached by the service is matched, it is usually billed at a lower cache price.
    2. Miss-input portion: Input tokens that do not hit the cache must be recomputed and are usually billed at the regular input price.
    3. Output portion: The result generated by the model is usually billed at the output price.

    Therefore, real cost depends on at least four variables:

    • Cache hit rate: the share of cached input in total input or total tokens.
    • Input-to-output ratio: long-context tasks, code completion, document Q&A, and summarization have very different input and output structures.
    • Difference between cache price and regular input price: a cheap cache price does not necessarily mean a cheaper total cost. The key question is whether the hit rate can amplify the advantage of the lower price.
    • Packages, quotas, and peak windows: subscriptions, periodic quotas, and peak-hour limits can significantly change the effective unit price.

    In other words, evaluating whether an LLM service is cost-effective is not just a matter of asking whether its cache price is low. It requires asking what the overall cost is under the real traffic structure.

    2. Build a General Cost Model: Align the Definitions First

    Before estimating cost, you must align the definitions. There are two common approaches.

    Approach 1: Calculate by share of total tokens

    Let total tokens be T, cache-hit tokens be C, miss-input tokens be U, and output tokens be O. Then:

    C + U + O = T

    The cost is:

    Cost = C / 1,000,000 × cache_price + U / 1,000,000 × miss_input_price + O / 1,000,000 × output_price

    If expressed as shares, let:

    • c: cache-hit tokens as a share of total tokens;
    • u: miss-input tokens as a share of total tokens;
    • o: output tokens as a share of total tokens;

    and c + u + o = 1. Then the cost per million tokens is:

    Unit cost = c × cache_price + u × miss_input_price + o × output_price

    This approach is suitable for directly analyzing how much it costs per 100 million tokens.

    Approach 2: Calculate by input hit rate

    Let input tokens be I, output tokens be O, and total tokens be:

    T = I + O

    Let the input share be a and the output share be b, so:

    I = a × T
    O = b × T
    a + b = 1

    Let the cache hit rate be h, meaning the share of input that hits the cache. Then:

    Cache-hit tokens = a × h × T
    Miss-input tokens = a × (1 - h) × T

    The cost per million tokens is:

    Unit cost = a × h × cache_price + a × (1 - h) × miss_input_price + b × output_price

    This formula is suitable for analyzing how cost changes when the cache hit rate changes.

    It is important to note that some discussions confuse “cache-hit share of total tokens” with “cache-hit share of input.” This can lead to biased conclusions. In actual estimation, you should clearly state which definition is being used.

    3. Scenario Analysis: Conclusions Can Reverse Under High and Low Cache Hit Rates

    The following generic example illustrates the calculation method. The parameters are for demonstration only and do not refer to any specific service.

    Assume there are two billing plans.

    Plan A: lower cache price, but higher miss-input and output prices.

    • Cache price: CNY 0.05 per million tokens
    • Miss-input price: CNY 3.00 per million tokens
    • Output price: CNY 3.00 per million tokens

    Plan B: higher cache price, but lower miss-input and output prices.

    • Cache price: CNY 0.115 per million tokens
    • Miss-input price: CNY 0.50 per million tokens
    • Output price: CNY 1.50 per million tokens

    Scenario 1: High cache hit rate

    Assume that for every 100 million tokens:

    • Cache hit: 98,000,000 tokens
    • Miss input: 1,500,000 tokens
    • Output: 500,000 tokens

    Plan A cost:

    98 × 0.05 + 1.5 × 3.00 + 0.5 × 3.00
    = 4.90 + 4.50 + 1.50
    = CNY 10.90 per 100 million tokens

    Plan B cost:

    98 × 0.115 + 1.5 × 0.50 + 0.5 × 1.50
    = 11.27 + 0.75 + 0.75
    = CNY 12.77 per 100 million tokens

    In this case, Plan A is cheaper. The reason is direct: the cache-hit share is very high, so the advantage of the lower cache price is amplified.

    Scenario 2: Cache hit rate declines

    Assume that for every 100 million tokens:

    • Cache hit: 95,000,000 tokens
    • Miss input: 4,000,000 tokens
    • Output: 1,000,000 tokens

    Plan A cost:

    95 × 0.05 + 4 × 3.00 + 1 × 3.00
    = 4.75 + 12.00 + 3.00
    = CNY 19.75 per 100 million tokens

    Plan B cost:

    95 × 0.115 + 4 × 0.50 + 1 × 1.50
    = 10.925 + 2.00 + 1.50
    = CNY 14.425 per 100 million tokens

    Now Plan B is cheaper. The reason is that when the cache hit rate declines, the share of miss-input and output tokens rises. Plan B has lower prices for miss input and output, so its overall cost can become lower.

    This example illustrates an important principle:

    • When the cache hit rate is high enough, the plan with the lower cache price has an advantage.
    • When the cache hit rate declines, the plan with lower miss-input and output prices may be more cost-effective.
    • Real selection must be based on sensitivity analysis of your own traffic structure, not on a single price item.

    4. Why Is Cache Hit Rate Hard to Estimate?

    Many teams assume during selection that “the cache hit rate will be high,” but the actual hit rate depends on the business scenario and engineering implementation.

    Common influencing factors include:

    1. Whether the system prompt is stable
      If the system prompt, role setting, tool descriptions, and business rules remain fixed, cache hits are more likely. If time, random numbers, user IDs, session IDs, or other content are inserted dynamically into every request, the hit rate will drop significantly.

    2. Whether multi-turn conversation context is continuous
      In multi-turn conversations, if every turn carries the full history and the prefix is stable, the hit rate is higher. However, if historical content changes frequently, summaries are rewritten often, or sessions are truncated and reorganized, the hit rate will decline.

    3. Whether knowledge base fragments are stable
      In retrieval-augmented scenarios, if different document fragments are retrieved each time, the input prefix may change even for the same question, affecting cache hits.

    4. Whether context ordering is consistent
      The same content may fail to hit the cache if it is ordered differently. For example, changes in document fragment order, citation numbering, or tool list order can affect prefix consistency.

    5. Whether requests are similar in batches
      In batch tasks, code completion, and log analysis, if many requests share the same prefix, the cache hit rate is usually higher.

    Therefore, cache hit rate should not be estimated by guesswork alone. It is better to measure it based on real traffic sampling.

    5. Engineering Methods for Improving Cache Hit Rate

    If your business relies on caching to reduce cost, you can optimize from several directions:

    1. Stable prefix design
      Place invariant content at the front of the request, such as system instructions, fixed descriptions, common templates, and stable knowledge fragments. Put dynamic content later whenever possible.

    2. Reduce dynamic variable pollution of the prefix
      Avoid inserting timestamps, random strings, user identifiers, request IDs, and other high-frequency changing fields at the front of the request. These fields can be placed in less sensitive positions or adjusted according to service rules.

    3. Context compression and summary stabilization
      For multi-turn conversations, do not regenerate complex summaries every turn. If the summary is relatively stable, you can fix it as part of the prefix to improve hit probability.

    4. Session-affinity routing
      At the gateway or application layer, try to route requests from the same user, task, or session to the same processing path to reduce context rebuild cost.

    5. Shared templates for batch tasks
      For code generation, text extraction, log classification, and similar tasks, extract common templates so that many requests can reuse the same prefix.

    6. Warm-up before peak hours
      If your business has obvious peak periods, warm up high-frequency templates, common system prompts, or shared context before the peak arrives to improve cache hit probability.

    7. Establish monitoring metrics
      At a minimum, monitor the following:

      • Cache hit rate
      • Average input tokens
      • Average output tokens
      • Share of miss-input tokens
      • Share of requests during peak hours
      • Share of retry requests
      • Average cost per request
      • Cost per million valid business requests

    Without monitoring, there is no real cost optimization.

    6. Packages and Periodic Quotas: Do Not Look Only at the Monthly Fee

    In addition to pay-as-you-go billing, many LLM services offer subscription packages or periodic quota packages. A package may look cheap, but the real cost depends on the effectively usable quota.

    Package cost estimation should consider three factors:

    1. Whether the periodic quota is truly usable
      Packages usually provide token quotas per week, per month, or per billing period. The larger the quota, the lower the nominal unit price, but if the business cannot use it fully, the actual cost may not be low.

    2. Whether peak hours are restricted
      Some packages set peak-hour windows. For example, during a period from midday to afternoon on weekdays, requests may be limited, queued, slowed down, or excluded from discounted quota. The peak window needs to be converted into an actual loss.

    3. Whether the package supports stacking or overflow
      If calls are still needed after the package quota is exhausted, the overflow portion may be billed on demand. In this case, compare the total cost of “package baseline + overflow on-demand billing.”

    You can use the following formulas to estimate the package’s equivalent unit price:

    Monthly theoretical quota = weekly quota × 52 / 12

    Peak loss rate = peak hours / total hours in a week

    Effective quota = monthly theoretical quota × (1 - peak loss rate) × actual utilization

    Package equivalent unit price = monthly package fee / effective quota

    For example, if a package has a peak window of 4 hours per weekday, the weekly peak duration is 20 hours. A week has 168 hours, so the peak loss rate is approximately:

    20 / 168 ≈ 11.9%

    If business requests are evenly distributed throughout the day, the theoretical effective quota loss is about 11.9%. However, if the business peak happens to fall in that window, the actual loss may be higher.

    For another example, if a package claims a large quota but the team’s actual utilization is only 50%, then even with an obvious nominal discount, the true equivalent unit price may be higher than expected. Packages are suitable for stable, predictable, continuous workloads. For highly variable and hard-to-predict workloads, pay-as-you-go is often more flexible.

    7. Package Tier Selection: Balancing Entry, Advanced, and High-Performance Tiers

    Generic packages are often divided into multiple tiers, such as entry, advanced, and high-performance. Different tiers suit different workload characteristics.

    1. Entry tier
      Suitable for individual developers, lightweight experiments, low-frequency calls, or edge workloads. The advantage is a low monthly fee and low trial-and-error cost. The disadvantage is limited quota and concurrency capacity; once business volume rises, an upgrade may be needed quickly.

    2. Advanced tier
      Suitable for small and medium teams, regular business calls, and stable daily active usage. This tier usually balances cost and quota and is the first option most teams evaluate.

    3. High-performance tier
      Suitable for high concurrency, high stability requirements, batch tasks, or critical business paths. The advantage is stronger quota and capacity, and the unit cost may be lower. The disadvantage is that if business scale falls short of expectations, resources may sit idle.

    When choosing a tier, do not look only at which tier has the largest quota. Consider:

    • Real token consumption over the past 7 to 30 days
    • Whether peak-hour concurrency is concentrated
    • Whether tasks can be shifted off-peak
    • Whether the package quota is fully used
    • Whether overflow on-demand costs remain after the package expires
    • Whether the team can tolerate business interruption risk after quota exhaustion

    A practical rule: if the package quota is persistently underused, the tier is too high; if it is frequently exhausted early, the tier is too low or needs to be combined with pay-as-you-go billing.

    8. Hybrid Strategy for Pay-as-You-Go and Packages

    For production environments, choosing only pay-as-you-go or only packages is rarely optimal. A more common strategy is a hybrid model:

    1. Cover baseline load with packages
      If daily load is stable, use packages to handle baseline traffic and lower the unit cost.

    2. Handle peak load with pay-as-you-go
      If the business has obvious peaks, package quota may not cover them. Configure on-demand overflow to ensure availability.

    3. Run non-real-time tasks off-peak
      Log analysis, batch summarization, offline evaluation, and data cleaning can be moved to low-peak hours to reduce peak pressure.

    4. Prioritize real-time tasks
      Real-time workloads such as chat, customer service, and code completion need guaranteed response speed and should use stable quota or higher-grade resources first.

    The essence of this approach is to use packages to control average cost and pay-as-you-go to provide peak elasticity.

    9. Reusable Pseudocode for Cost Estimation

    Below is platform-independent pseudocode that engineering teams can use to build their own models.

    1. total_tokens = actual_total_consumption
    2. cache_hit_share = cache_hit_tokens / total_tokens
    3. miss_input_share = miss_input_tokens / total_tokens
    4. output_share = output_tokens / total_tokens
    5. assert cache_hit_share + miss_input_share + output_share == 1
    6. cost_per_million_tokens = cache_hit_share * cache_price
    7. cost_per_million_tokens += miss_input_share * miss_input_price
    8. cost_per_million_tokens += output_share * output_price

    If you use the input hit rate model instead, it can be written as:

    1. input_share = input_tokens / total_tokens
    2. output_share = output_tokens / total_tokens
    3. cache_hit_rate = cache_hit_tokens / input_tokens
    4. cost_per_million_tokens = input_share * cache_hit_rate * cache_price
    5. cost_per_million_tokens += input_share * (1 - cache_hit_rate) * miss_input_price
    6. cost_per_million_tokens += output_share * output_price

    For engineering implementation, it is recommended to package the formulas above into an internal cost estimation tool that supports different input parameters and outputs:

    • Cost per million tokens
    • Cost per 100 million tokens
    • Daily cost
    • Monthly cost
    • Cost curves under different hit rates
    • Equivalent unit prices under different package tiers

    10. Selection Checklist: Avoid Misjudgment from Looking Only at Unit Price

    Before formally selecting or adjusting an LLM service plan, follow this checklist:

    1. Sample real traffic
      Collect at least 7 days of data, covering weekdays, weekends, peak hours, and off-peak hours. Record total tokens, input tokens, output tokens, and cache-hit status.

    2. Calculate cost under both definitions
      Use both “share of total tokens” and “input hit rate” to avoid definition confusion.

    3. Perform sensitivity analysis
      Focus on how cost changes when the cache hit rate moves from 90%, 95%, 98%, to 99%. Cache hit rate is one of the most sensitive variables.

    4. Evaluate peak windows
      Clarify whether peak hours affect request success rate, response latency, quota consumption, or price discounts. If the peak window overlaps with business peaks, include the extra loss.

    5. Calculate package equivalent unit price
      Do not look only at the monthly fee. Include effective quota, utilization, peak loss, and overflow cost.

    6. Assess engineering optimization space
      Can prefixes be stabilized? Can dynamic content be reduced? Can batch tasks be shifted off-peak? Can context be compressed? All of these affect the final bill.

    7. Reserve capacity and set alerts
      Production environments cannot rely only on average cost. Consider extra consumption caused by burst traffic, retries, timeouts, and failover.

    11. Conclusion: Real Cost Comes from Billing Rules × Traffic Structure × Scheduling Capability

    Cost optimization for LLM services is not simply a search for the lowest unit price. It requires a cost model that can be calculated, monitored, and iterated. When the cache hit rate is high, a plan with a lower cache price may be better. When the cache hit rate declines, a plan with lower miss-input and output prices may be more cost-effective. Whether a package is worth buying also depends on effective quota, peak windows, and actual utilization.

    For developers and enterprises, the most valuable actions include:

    • Standardize token statistics definitions
    • Build cache hit rate monitoring
    • Use real traffic for cost estimation
    • Combine packages with pay-as-you-go billing
    • Schedule non-real-time tasks off-peak
    • Continuously optimize prefix stability and context structure

    Ultimately, LLM cost control is no longer just about choosing a cheap service. It becomes an engineering capability: whether you can accurately predict traffic, improve cache hits, reasonably arrange packages and on-demand billing, and maintain stability while controlling cost during peak hours. That is the long-term capability technical teams need to build.