Large Model API Cost Analysis: Caching, Usage Plans, and Real-World Pricing

    A practical framework for estimating the true cost of large language model APIs. This article explains how cache hit rates, input and output token ratios, peak/off-peak scheduling, subscription quotas, retries, and utilization affect effective costs. It also provides formulas, sensitivity analysis methods, break-even calculations, and implementation guidance for request-level cost monitoring.

    Large Model API Cost Analysis: Caching, Usage Plans, and Real-World Pricing

    1. The Core Issue: Price Lists Show Unit Rates, Not Real Costs

    When estimating the cost of large language model services, it is easy to compare only a few price labels: the cost per million cached input tokens, regular input tokens, and output tokens, or the monthly price of different subscription plans.

    This approach only answers which option is cheaper under a specific idealized request pattern. It does not answer how much a real business will ultimately spend.

    Actual billing depends on several factors:

    1. Cache hit rate: Cached input tokens are usually billed at a lower rate, while uncached input tokens use the standard input price.
    2. Input-to-output ratio: Output tokens are often more expensive than input tokens. Even a relatively small output share can significantly increase total cost.
    3. Cache reusability: Stable prefixes, changing context, cache expiration, and cache reconstruction all affect the actual hit rate.
    4. Peak and off-peak scheduling: When pricing or quotas vary by time of day, the ability to move batch workloads to off-peak periods changes the effective unit cost.
    5. Subscription utilization: Monthly fees, weekly quotas, peak-period limits, overage rules, and expiration policies determine whether a nominal discount can actually be realized.
    6. Additional engineering costs: Retries, failed requests, oversized contexts, cache writes, observability, and gateway infrastructure also contribute to total cost of ownership.

    The correct approach is to establish a consistent accounting model: separate usage by token category, calculate weighted costs based on actual proportions, and then perform sensitivity and break-even analysis.

    2. A Unified Cost Model

    For a given measurement period, define:

    • H: Cached input tokens
    • U: Uncached input tokens
    • O: Output tokens
    • Pc: Price per million cached input tokens
    • Pi: Price per million regular input tokens
    • Po: Price per million output tokens
    • F: Fixed costs, such as a monthly subscription or platform fee
    • E: Additional costs, such as cache writes, storage, retry overhead, observability, and gateway infrastructure

    The cost for the measurement period can be expressed as:

    1. Cost = H / 1,000,000 × Pc
    2. + U / 1,000,000 × Pi
    3. + O / 1,000,000 × Po
    4. + F
    5. + E

    To compare different services, models, or plans, it is useful to normalize the result as the effective cost per million total tokens:

    1. TotalTokens = H + U + O
    2. EffectiveCostPerM = Cost / (TotalTokens / 1,000,000)

    The accounting definition must remain consistent. In practice, “cache hit rate” may refer to all tokens, input tokens only, or a denominator that also includes output tokens. These definitions are not interchangeable.

    A billing-verifiable engineering definition is:

    1. CacheHitRate = H / (H + U)
    2. OutputRatio = O / TotalTokens

    Only consistent definitions make comparisons, budgets, and optimization decisions meaningful.

    3. How Cache Hit Rate Changes the Result

    The effect of cache hit rate is not linear across pricing plans. In general:

    • When the hit rate is very high, differences in cached-token prices become more important.
    • As the hit rate declines, regular input and output prices quickly become more influential.
    • A plan with a very low cache price but higher regular input and output prices may be advantageous only at extremely high hit rates.
    • A plan with a slightly higher cache price but cheaper regular input and output may be more resilient to hit-rate fluctuations.

    The following is a purely mathematical example. All prices are hypothetical and do not represent any actual service quotation.

    Assume a monthly window with 100 million input tokens and 500,000 output tokens. Compare two hypothetical plans:

    Item Plan A Plan B
    Cached input price per million tokens ¥0.12 ¥0.05
    Regular input price per million tokens ¥1.00 ¥2.00
    Output price per million tokens ¥4.00 ¥8.00

    The estimated costs at different cache hit rates are:

    Cache hit rate Plan A cost Plan B cost Lower-cost option
    98% ¥15.76 ¥12.90 Plan B
    95% ¥18.40 ¥18.75 Plan A
    90% ¥22.80 ¥28.50 Plan A

    In this example, the break-even cache hit rate is approximately 95.33%. Above that point, Plan B, with the lower cached-input price, is more attractive. Below it, Plan A is cheaper because its regular input and output prices are lower.

    This demonstrates why a lower cached-token price cannot be used as the sole selection criterion. The result depends on the combination of cache hit rate and the complete pricing structure.

    Output-token share should also be modeled separately. If output increases from 0.5% to 2% of total usage, output costs may rise several times even when input usage remains unchanged. For code generation, long-form generation, and structured-data generation, output price, maximum output length, and retry rate may matter more than cache pricing.

    4. Subscription Plans: Converting Monthly Fees into Realizable Unit Costs

    Subscription plans are attractive because they promise discounts. However, those discounts are meaningful only when quotas can be used sufficiently and the workload can operate consistently within the plan’s rules.

    Do not evaluate a subscription solely by its monthly fee or by dividing that fee by the nominal quota. Instead, calculate the realizable effective unit cost.

    Let:

    • P: Subscription fee
    • Q: Total quota included in one billing period
    • A: Quota that can actually be used

    The effective cost per token is:

    1. EffectiveCostPerToken = P / A

    If the equivalent pay-as-you-go cost for the workload is E_api, the subscription break-even usage is:

    1. BreakEvenUsage = P / E_api

    The plan-to-usage cost ratio can be represented as:

    1. PlanCostRatio = P / E_api

    When this ratio is below 1 and the workload can consistently reach the break-even usage level, the subscription may provide a cost advantage.

    The key variable is how A is calculated:

    1. Does the quota refresh weekly? Weekly quotas should not automatically be interpreted as a monthly quota divided by the monthly price. If unused weekly capacity expires, actual usable capacity must be calculated at a weekly or more granular level.
    2. Are there peak-period restrictions? If the plan provides more capacity during certain periods and limits usage during peak hours, measure the workload’s peak and off-peak distribution.
    3. Are there overage, throttling, or fallback rules? Overage pricing, concurrency limits, and queueing delays all affect engineering cost.
    4. Can unused quota roll over? If it cannot, low utilization can substantially increase the actual unit cost.
    5. Are output tokens converted separately? Some plans use a unified token quota or weighted-token accounting. These rules must be aligned with the pay-as-you-go billing model.

    For example, suppose a plan costs P, but only 60% of the workload occurs during periods when the quota can be used flexibly. The remaining 40% occurs during peak periods and is throttled or requires additional capacity. The realizable quota should not be calculated using the full nominal value Q; it must be adjusted according to the workload’s scheduling pattern.

    Using the nominal quota without this adjustment will underestimate the real cost.

    5. Including Peak and Off-Peak Scheduling

    When pricing or capacity varies by time of day, divide the workload into two categories:

    • L_off: Tasks that can be moved to off-peak periods, such as offline batch processing, bulk summarization, and non-real-time evaluation
    • L_peak: Tasks that must be completed immediately, such as interactive question answering, online editing assistance, and real-time customer support

    Schedule flexible workloads during off-peak periods whenever possible to reduce the weighted unit cost. Real-time workloads should be budgeted using peak-period costs rather than assuming that all requests can receive off-peak pricing.

    If a subscription limits peak-period capacity, calculate the cost of capacity shortfalls separately:

    1. TotalCost = PlanFee
    2. + Cost(L_off within quota)
    3. + Cost(L_peak within quota)
    4. + Cost(overflow)

    A practical strategy is to:

    1. Use pay-as-you-go capacity for peaks and sudden traffic spikes.
    2. Use subscriptions for stable, predictable workloads that can reliably consume the included quota.
    3. Schedule offline jobs during off-peak periods whenever possible.
    4. Monitor peak-period quota utilization and the percentage of overflow requests.

    6. Implementation: Track Token Structure Per Request

    Enterprise cost management requires billing visibility at the request and business-unit level, not just a monthly total.

    The model gateway or application layer should record fields such as:

    • Timestamp, application, tenant, and business scenario
    • Model or service specification
    • Cached input token count
    • Uncached input token count
    • Output token count
    • Whether the request was retried, the failure reason, and retry count
    • Latency and concurrency
    • Billing plan and peak/off-peak period label

    The following vendor-neutral pseudocode illustrates the calculation:

    1. Usage record:
    2. cached_input
    3. uncached_input
    4. output
    5. retry_count
    6. Price record:
    7. cache_per_m
    8. input_per_m
    9. output_per_m
    10. function token_cost(usage, price):
    11. base = usage.cached_input / 1_000_000 × price.cache_per_m
    12. + usage.uncached_input / 1_000_000 × price.input_per_m
    13. + usage.output / 1_000_000 × price.output_per_m
    14. return base × (1 + usage.retry_count)
    15. function hit_rate(usage_list):
    16. cached = sum(item.cached_input)
    17. total_input = sum(item.cached_input + item.uncached_input)
    18. return cached / total_input

    The break-even cache hit rate can also be expressed as a general function.

    Assume the cached-input, regular-input, and output prices for Plans A and B are pca, pia, poa and pcb, pib, pob, respectively. Let I be the input-token volume in a fixed measurement window, O the output-token volume, and h the cache hit rate:

    1. CostA = h × I × pca + (1 - h) × I × pia + O × poa
    2. CostB = h × I × pcb + (1 - h) × I × pib + O × pob

    Setting the two costs equal produces the crossover point:

    1. h = [I × (pia - pib) + O × (poa - pob)]
    2. / [I × (pca - pcb - pia + pib)]

    In production, this logic should be integrated into a cost-analysis script and recalculated weekly using real traffic, rather than relying on a static price table for a one-time decision.

    For observability, organizations can combine log services, monitoring alerts, and billing data in a general cloud environment, including environments such as Baidu AI Cloud. The goal is not merely to display the total bill, but to answer three operational questions:

    • Has the cache hit rate declined?
    • Has the output ratio increased?
    • Has peak-period overflow grown?

    7. Practical Optimization Checklist

    1. Create stable cacheable prefixes: Put system instructions, tool descriptions, output formats, and fixed knowledge first. Place timestamps, random values, and dynamic user content later to reduce unnecessary cache invalidation.
    2. Separate static and dynamic context: For large codebases and document collections, segment and trim context so that each request does not include irrelevant content.
    3. Control output tokens: Use structured output, maximum-length limits, concise summaries, and staged generation to reduce unnecessary output.
    4. Budget by business line: Do not use one global cache hit rate for every application. Real-time question answering, code completion, and batch summarization have very different usage structures.
    5. Build sensitivity ranges: At minimum, test high, medium, and low cache hit rates together with high, medium, and low output ratios.
    6. Combine subscriptions with pay-as-you-go: Use subscriptions for stable workloads and pay-as-you-go capacity for spikes. This avoids overbuying for peak demand and reduces waste from underutilized plans.
    7. Review continuously: Prices, plan rules, and traffic patterns change. Revisit the cost model monthly instead of treating the initial selection as permanent.

    8. Conclusion

    The real cost of a large language model API is not a single price. It is the combined result of usage structure, billing rules, and scheduling capability.

    When the cache hit rate is high, differences in cached-input prices are amplified. As the hit rate falls, regular input and output prices may become the primary cost drivers, potentially reversing the ranking of competing plans. Subscription plans must be evaluated using monthly fees, quotas, peak-period restrictions, expiration rules, and actual utilization rather than nominal unit prices.

    The most reliable approach is to standardize token definitions, record usage at the request level, calculate weighted costs and break-even points, perform sensitivity analysis using real traffic, and evaluate schedulable and real-time workloads separately.

    Only by converting a price list into a workload-specific billing model can teams make cost decisions that are both economical and sustainable.