LLM Inference Pricing: Model Scale, Cost Structure, and Unit Economics

    This article explains how model scale affects LLM inference pricing by breaking down compute, memory, concurrency, prefill/decode asymmetry, and prefix caching. It also shows how to move from token-level unit prices to task-level cost, including a request-cost calculation example, break-even analysis, model selection guidance, and a deployment checklist.

    LLM Inference Pricing: Model Scale, Cost Structure, and Unit Economics

    1. Why Pricing Discussions Cannot Avoid Model Scale

    Changes in model parameter scale affect training cost, deployment cost, and inference cost at the same time. Users most directly observe the unit price, but the unit price is only an external result of the provider’s cost structure and pricing strategy. To understand why unit prices change, it is necessary to return to the inference process itself and analyze which resources each token consumes.

    A common question in pricing discussions is whether a larger model should have a higher unit price. The answer depends on three things: how compute per token changes, how memory usage and deployment scale change, and how provider utilization and pricing strategy change. These three factors need to be analyzed separately.

    2. Three Sources of Inference Cost

    2.1 Compute

    When a dense model processes a token, the forward-pass compute is approximately proportional to the number of parameters. A common industry rule of thumb is that each parameter requires roughly two floating-point operations per token processed. If the parameter count doubles, the compute for a single token roughly doubles.

    This rule applies only to dense architectures. A model using a mixture-of-experts (MoE) architecture may have a very large total parameter count, but only a subset of experts is activated for each token. In this case, compute is determined by active parameters, while memory usage is determined by total parameters. Therefore, a larger total parameter count does not necessarily increase compute per token proportionally, but its impact on deployment scale is real.

    2.2 Memory Usage

    Model weights need to reside in memory. Weight memory can be roughly estimated as:

    1. weight memory ≈ parameter count × bytes per parameter

    For example, a 100-billion-parameter model using 16-bit floating-point precision (2 bytes per parameter) requires about 200 GB of memory for weights alone. As parameter scale continues to grow, the number of required accelerator cards increases, and communication overhead between multiple cards also rises. This cost is amortized across each request and directly affects the unit price.

    In addition, inference requires storing a key-value cache (KV Cache) for each active request. The longer the context and the higher the concurrency, the more KV cache is used. Cost pressure in long-context scenarios is often much greater than in short-text scenarios.

    2.3 Concurrency and Utilization

    With the same hardware under high concurrency, the amortized cost per token is lower. The reason is that batching allows compute units to work more fully. If the actual service load is low and hardware sits idle, the cost allocated to each request increases.

    Therefore, a higher unit price for a new model may reflect higher hardware costs, or it may reflect changes in traffic scale, capacity reservation, or service level. The unit price alone cannot determine the specific cause.

    3. Why Input and Output Are Priced Differently

    LLM inference is usually divided into two stages.

    Prefill stage: The model reads all input tokens at once, and computation for each token can be completed in parallel. This stage is mainly limited by compute capacity and is relatively efficient.

    Decode stage: The model generates output tokens one by one. Each generated token depends on the previous token, and each step must read all weights and the KV cache. This stage is mainly limited by memory bandwidth. The compute per step is small, but memory access is frequent, making it less efficient.

    Therefore, for the same number of tokens, output usually consumes more resources than input. Most services price output higher than input, which reflects this difference. When selecting models or estimating costs, input and output token counts should be counted separately and should not be simply combined.

    4. Why Cache Hits Are Cheaper

    Prefix caching is a common technique for reducing the cost of repeated input. When multiple requests share the same prefix, such as the same system instructions, tool definitions, or reference documents, the server can reuse the KV cache already computed for that prefix and skip repeated prefill computation.

    Tokens that hit the cache do not need to be recomputed, so their unit price is much lower than that of uncached input tokens. However, the cache still occupies memory or other storage resources, so the hit price is usually higher than zero rather than free.

    To improve cache hit rates, prompt structure should follow the principle of “fixed content first, variable content later”:

    1. [system instructions (fixed)][tool definitions (fixed)][reference documents (fixed)][user question (changes each time)]

    If timestamps, random IDs, or usernames are placed at the very beginning, the prefix cannot be reused and the cache is almost ineffective. Such details are often overlooked in real projects, but they directly change the bill.

    5. Possible Explanations for Price Increases

    Without official explanations, a unit price increase may be caused by several categories of factors. The following are only common directions for analysis and should not be treated as definitive conclusions.

    1. Hardware and capacity costs: A new-generation model may require more accelerator cards or higher-spec hardware, and higher fixed costs can be passed through to unit prices.
    2. Service level and resource reservation: To guarantee latency and availability, the provider may reserve more resources, and this cost is included in the price.
    3. Pricing strategy: Early low prices may be intended to acquire users and accumulate data, while later price adjustments may be intended to recover costs.
    4. Product positioning: A higher price may correspond to high-end capabilities for complex tasks, with a target user group different from before.

    Note that parameter scale and unit price are not linearly related. After model capability improves, the number of tokens and retries required to complete the same task may decrease. This means that even if the unit price rises, the cost per task may not rise.

    6. From Unit Price to Cost per Task

    To judge whether a model is cost-effective, look at cost per task rather than unit price. Cost per task can be estimated with the following formula:

    1. cost per task = cost per request × average number of requests ÷ task success rate

    The higher the task success rate, the fewer retries and manual corrections are needed, and the lower the cost per task.

    The cost per request depends on three parts: input, cache hits, and output. The following example code calculates the cost of a single request:

    1. from dataclasses import dataclass
    2. @dataclass
    3. class Pricing:
    4. input_per_m: float # price per million uncached input tokens (currency units)
    5. cache_hit_per_m: float # price per million cache-hit tokens (currency units)
    6. output_per_m: float # price per million output tokens (currency units)
    7. def request_cost(input_tokens: int, cached_tokens: int,
    8. output_tokens: int, p: Pricing) -> float:
    9. """Calculate the cost of a single request in currency units."""
    10. if cached_tokens > input_tokens:
    11. raise ValueError("cached tokens cannot exceed input tokens")
    12. uncached = input_tokens - cached_tokens
    13. total = (uncached * p.input_per_m
    14. + cached_tokens * p.cache_hit_per_m
    15. + output_tokens * p.output_per_m)
    16. return total / 1_000_000
    17. if __name__ == "__main__":
    18. # The following prices are assumptions for demonstration only,
    19. # not the real prices of any service.
    20. demo_price = Pricing(input_per_m=10, cache_hit_per_m=1, output_per_m=40)
    21. with_cache = request_cost(20_000, 15_000, 1_000, demo_price)
    22. without_cache = request_cost(20_000, 0, 1_000, demo_price)
    23. print(f"With cache hit: {with_cache:.3f}") # 0.105
    24. print(f"Without cache hit: {without_cache:.3f}") # 0.240

    In the demonstration above, after 15,000 input tokens hit the cache, the cost per request drops by about 56%. This shows that cache design can have a greater impact on cost than model selection itself.

    Break-Even Analysis

    Assume the unit price of the new model is k times that of the old model. To keep cost per task unchanged, the new model needs to satisfy the following condition:

    1. new model's (tokens per request × average number of requests ÷ task success rate)
    2. ≤ old model's corresponding value ÷ k

    In other words, the new model needs to achieve a corresponding improvement in token consumption, retry count, or task success rate. Teams should measure this result using their own real task samples rather than relying on general benchmarks.

    7. Recommendations for Model Selection and Cost Control

    1. Evaluate models by task success rate: When comparing different models, measure completion rate, correction count, and average latency on your own business tasks, then calculate cost per task.
    2. Design cache-friendly prompt structures: Put fixed content first and variable content later, and avoid inserting dynamic fields such as timestamps into the prefix.
    3. Use tiered routing: Assign simple, structured tasks to smaller models or rule systems, and call large models only for complex tasks.
    4. Control output length: Set reasonable output limits and require the model to return results in a fixed format to reduce redundant text.
    5. Record usage metrics: Record input, cache-hit, and output token counts and call counts by request type, and review the bill structure regularly.
    6. Validate with small traffic: Before switching to a model with a higher unit price, run a comparison experiment with a portion of real traffic, compare success rates and total costs, and then decide whether to switch all traffic.

    8. Deployment Checklist

    • Have tokens been counted separately for input, cache hits, and output?
    • Has cost per task been calculated, not only cost per request?
    • Is the prompt prefix stable, and is the cache hit rate monitored?
    • Has tiered routing been implemented for different tasks?
    • Before switching models, has a comparison test been completed with real samples?