How to Calculate the True Cost of Large Language Models: Cache Hit Rates and Subscription Value

    The real cost of an LLM is determined by more than the advertised price per million tokens. This guide explains how input tokens, cached tokens, output tokens, cache hit rates, peak-time pricing, and monthly subscription limits affect total spending. It also provides reusable formulas, budgeting methods, and engineering practices for comparing pay-as-you-go pricing with subscription plans.

    As large language model applications move from experimentation into production, cost often becomes an engineering constraint before model quality does. When comparing models, many teams focus only on the price per million tokens. In practice, at least four factors have a major impact on the bill: the proportion of input tokens, the proportion of output tokens, the cache hit rate, and price differences across time periods.

    This is particularly important for code completion, knowledge-base question answering, customer service, and AI Agent workflows. Many requests repeatedly include the same system prompt, tool definitions, conversation history, or fixed documents. If this content can be cached, the actual cost may be significantly lower than an estimate based entirely on standard input pricing. On the other hand, if the cache hit rate is unstable, a plan that initially appears inexpensive can quickly lose its advantage.

    The following framework can be applied to different models, cloud providers, and both pay-as-you-go and monthly subscription plans.

    1. Separate the Three Types of Tokens

    A model request can usually be divided into three billing categories:

    1. Standard input tokens
      Newly submitted prompts, user questions, context, or tool descriptions that do not hit the cache.

    2. Cached input tokens
      Repeated input content that appeared in previous requests and was recognized by the service as reusable. Cached tokens are typically billed at a lower rate, although the exact rules depend on the provider’s pricing documentation.

    3. Output tokens
      Tokens generated by the model, including answers, code, structured data, and tool-call parameters. Output pricing is often higher than input pricing, so response length can have a substantial effect on total cost.

    Let:

    • T = total token volume
    • r = cached-token ratio
    • i = standard-input ratio
    • o = output ratio

    Then:

    1. r + i + o = 1

    If pricing is measured per million tokens, the average cost can be expressed as:

    1. Average cost =
    2. T × r × cached-input price
    3. + T × i × standard-input price
    4. + T × o × output price

    T may represent millions or hundreds of millions of tokens. As long as all units are consistent, the calculation will remain accurate.

    2. Why You Should Not Compare Cache Prices Alone

    When the cache hit rate is high, the cached-input price can have a significant effect on total cost. However, it is not the only variable.

    For example, one plan may offer a low cached-input price but charge considerably more for output tokens. In scenarios involving long answers, code generation, or complex reasoning, the output cost may offset the savings from caching. Another plan may have a slightly higher cache price but produce more compact responses, resulting in a lower final bill.

    The correct comparison is not:

    1. The plan with the lower cache price is always cheaper.

    Instead, compare:

    1. The total cost under the same token structure.

    At a minimum, the following conditions should be aligned:

    • Total token volume per request or billing period;
    • Cache hit rate;
    • Input-to-output ratio;
    • Whether tool calls and reasoning tokens are included;
    • Whether context truncation or automatic retries occur;
    • Whether peak and off-peak periods have different prices or limits.

    If these conditions differ, comparing pricing tables alone can easily produce misleading conclusions.

    3. Use Multiple Cache Hit Rates to Analyze Cost Changes

    To illustrate the impact of cache hit rates, consider two general scenarios. Assume that the input and output proportions remain constant while the cache hit rate changes.

    Scenario 1: High Cache Hit Rate

    Assume:

    • Cached input: 98%;
    • Standard input: 1.5%;
    • Output: 0.5%.

    This structure is common when applications have long, stable system prompts, frequently reused conversation history, or fixed Agent workflows. Examples include ongoing conversations, code editing, and workflow-based Agents.

    In this scenario, most input tokens are billed at the cached-input price, making that price a major variable. However, output tokens still require attention. Although they account for only a small share of total tokens, their contribution to the bill may be substantial because output pricing is usually much higher than cached-input pricing.

    If two plans have a large difference in output pricing, even a twofold difference in cache pricing may not produce a twofold difference in the final bill.

    Scenario 2: Moderate Cache Hit Rate

    Assume that the cache hit rate falls to 95%:

    • Cached input: 95%;
    • Standard input: 4.5%;
    • Output: 0.5%.

    At first glance, the change appears small: only 3.5 percentage points of input tokens have moved from cached to standard pricing. At the scale of hundreds of millions or billions of tokens, however, this difference becomes significant.

    For plans with relatively high prices for both cached and standard input, the total cost will increase more sharply.

    This demonstrates that cache hit rate is not an on-or-off feature. It is a continuous variable. Engineering teams should not build budgets only around idealized values such as 98% or 99%. At a minimum, estimate costs across the following ranges:

    • 90% cache hit rate;
    • 95% cache hit rate;
    • 98% cache hit rate;
    • 99% or higher as an upper-bound scenario.

    If a plan is cost-effective only at a 98% cache hit rate, its economics depend heavily on the caching mechanism. If it remains competitive between 90% and 95%, the budget is likely to be more stable.

    4. A Reusable Cost Calculation Example

    Assume a project processes 1 billion tokens per month with the following distribution:

    • Cached input: 96%;
    • Standard input: 3%;
    • Output: 1%.

    Let the three prices be:

    • Cached input: Pc;
    • Standard input: Pi;
    • Output: Po.

    The monthly cost is:

    1. Monthly cost =
    2. 1 billion × 96% × Pc
    3. + 1 billion × 3% × Pi
    4. + 1 billion × 1% × Po

    If prices are expressed in currency units per million tokens, 1 billion tokens must first be converted into 1,000 million-token units:

    1. Monthly cost =
    2. 1000 × 96% × Pc
    3. + 1000 × 3% × Pi
    4. + 1000 × 1% × Po

    A simple pseudocode implementation can reduce manual calculation errors:

    1. total_tokens = 1_000_000_000
    2. cache_ratio = 0.96
    3. input_ratio = 0.03
    4. output_ratio = 0.01
    5. cost = (
    6. total_tokens / 1_000_000
    7. * cache_ratio * cache_price
    8. + total_tokens / 1_000_000
    9. * input_ratio * input_price
    10. + total_tokens / 1_000_000
    11. * output_ratio * output_price
    12. )

    In production, prices, token ratios, and cache hit rates should be configurable. It is also better to generate a cost curve across different hit rates than to calculate only one static figure.

    5. A Monthly Subscription Does Not Mean Unlimited Tokens

    In addition to usage-based pricing, some platforms offer monthly programming or development subscriptions. These plans may vary by user tier, request frequency, concurrency limits, time period, and available quota.

    The most important step when evaluating a subscription is to convert its quota into an equivalent API value.

    Step 1: Confirm the Usable Quota

    The weekly or monthly quota advertised by a plan may be subject to limitations such as:

    • Different quotas during peak and off-peak periods;
    • Different consumption multipliers for models or task types;
    • Concurrency limits;
    • Additional charges after the quota is exceeded;
    • Quotas based on calendar weeks, calendar months, or rolling periods;
    • Different rules for individual and team accounts.

    Therefore, the maximum token count shown in promotional material should not automatically be treated as a stable, fully usable quota.

    Step 2: Convert the Quota into Standard API Cost

    If a plan allows X hundred million tokens per week, its equivalent value at the corresponding API prices is:

    1. Equivalent API value =
    2. cached tokens × cached-input price
    3. + standard input tokens × standard-input price
    4. + output tokens × output price

    The calculation must use the actual token structure. The entire quota should not be valued using the lowest cached-input price.

    Step 3: Calculate the Subscription Discount

    1. Subscription discount ratio =
    2. monthly subscription fee ÷ equivalent monthly API value

    For example, if a subscription costs 120 currency units per month and its equivalent API value is 200 currency units based on the actual usage structure, the subscription discount ratio is 60%. In other words, the subscription costs approximately 60% of the equivalent usage-based amount.

    This discount is meaningful only if the user can make substantial use of the quota. If the user consumes only 20% of the monthly allowance, the effective discount becomes much smaller and may even disappear compared with pay-as-you-go pricing.

    6. Peak Periods Can Change the Real Value of a Subscription

    Many subscriptions do not provide the same quota throughout the day. Working hours may be divided into peak and off-peak periods. Peak periods are often fixed business-hour windows, while off-peak periods may include nights, weekends, or holidays.

    For individual developers who mainly work at night or on weekends, the off-peak quota may be sufficient. For enterprise teams, however, most activity may take place during peak hours. In that case, the plan’s advertised total quota may not translate directly into usable production capacity.

    When evaluating a subscription, record the team’s actual usage distribution:

    1. Peak-period usage ratio = peak-period tokens ÷ total tokens
    2. Off-peak usage ratio = off-peak tokens ÷ total tokens

    If peak-period usage accounts for a large share of total activity, focus on peak-period quotas and concurrency rules rather than looking only at the total weekly allowance.

    7. Enterprise Budgets Should Use Ranges, Not a Single Number

    LLM usage can fluctuate significantly with business demand. A newly launched application may process only a few million tokens per day, while user growth can quickly push usage to tens of billions of tokens.

    A more reliable approach is to create three budget scenarios:

    • Conservative scenario: lower cache hit rate, higher output ratio, and lower usage volume;
    • Baseline scenario: estimates based on historical monitoring data;
    • Optimistic scenario: high cache reuse, high cache hit rate, and stable output length.

    For example:

    1. Monthly budget range =
    2. cost at low usage volume ~ cost at high usage volume

    Cost alerts should also be configured for events such as:

    • Daily token consumption exceeding the historical average;
    • Cache hit rate falling below the target;
    • Average output length increasing unexpectedly;
    • Retry or failure rates rising;
    • Peak-period usage growing rapidly.

    These indicators are more useful than simply checking the bill at the end of the month because they can reveal cost problems before they become difficult to control.

    8. Engineering Practices for Reducing Real-World Costs

    1. Place Stable Content First

    Put stable system rules, tool descriptions, and shared background information at the beginning of the request to improve cache reuse. Dynamic content should be placed later whenever possible. Small changes near the beginning of a context can otherwise prevent the larger section from being reused.

    2. Control Irrelevant Conversation History

    Not every historical message needs to be included in every request. Summarization, key-information extraction, and layered memory can reduce repeated input while also lowering context noise.

    3. Limit Output Length

    Set a reasonable maximum output-token limit and use structured formats to reduce unnecessary explanations. For code-generation tasks, ask the model to return only the changed sections instead of reproducing the entire file.

    4. Route Tasks by Complexity

    Simple classification, field extraction, and format conversion do not always require the most expensive model. Route tasks according to their complexity: use more capable models for difficult work and lower-cost models for repetitive operations.

    5. Record Actual Token Billing

    Do not track request counts alone. Record input tokens, cached tokens, output tokens, retry counts, and actual spending. Only complete observability can provide the data needed to correct and improve a budget model.

    Conclusion

    The goal of LLM cost analysis is not to find a single universally lowest price. It is to build a cost model that reflects the application’s actual usage pattern.

    When cache hit rates are high, cached-input pricing is highly important. As the hit rate falls, standard-input and output pricing become increasingly influential. Monthly subscriptions may appear to offer substantial discounts, but they are economically valuable only when their quotas, time windows, and concurrency limits match the team’s real working patterns.

    A reliable decision process is:

    1. Measure the actual token structure;
    2. Estimate costs across multiple cache hit rates;
    3. Compare pay-as-you-go pricing and monthly subscriptions within the same cost model.

    This produces conclusions that are explainable, reproducible, and useful for budget planning.