LLM API Cost Evaluation: From Price Lists to Real Bills

    Evaluating the cost of Large Language Model (LLM) APIs goes far beyond checking the input price per million tokens. This article explains why price lists alone are insufficient and introduces a unified billing model to simulate real-world bills. It covers common pricing structures, a pseudocode-based cost simulator, a structured selection process, and engineering optimizations to help developers accurately assess and control LLM API expenses.

    1. Why “Price Lists” Cannot Directly Answer Cost Questions

    LLM API pricing pages typically display input, output, and cache hit rates per million tokens. However, real-world bills are determined by multiple factors: whether inputs hit the cache, output length distribution, cache storage duration, whether requests occur during peak or off-peak hours, whether batch processing is used, whether batch and cache discounts can stack, whether retries and failed requests are billed, and whether long contexts trigger additional storage costs. Relying solely on input prices for model selection often leads to incorrect conclusions.

    A more reasonable approach is to abstract the pricing page into a unified billing model and simulate the bill using your own workload log distribution. The price list serves as the “parameters,” while your business traffic is the “input.” Comparing costs without a traffic profile is merely a theoretical exercise.

    2. Building a Unified Billing Model: Standardize Fields First, Then Compare Prices

    Regardless of which LLM API service you use, it is recommended to standardize the following billing fields first:

    Billing Dimension Fields to Collect Impact on Cost
    Input (Cache Miss) Input tokens (cache miss) Usually the primary input cost
    Input (Cache Hit) Input tokens (cache hit) Unit price may be significantly lower than a miss
    Output Output tokens Output unit price is typically higher than input; requires strict control
    Cache Storage Cache capacity × storage hours Long contexts and long-TTL caches may incur continuous fees
    Cache Write Written tokens and TTL tier Some pricing is tiered by 5 mins, 1 hour, etc.
    Peak/Off-peak Peak/off-peak request ratio, completion time Peak/off-peak price difference can reach 2x
    Batch Processing Deferrable task ratio, batch discount Usually a standard price discount, but stacking rules must be verified
    Discount Rules Whether batch and cache discounts are mutually exclusive If non-stackable, calculate separately and choose the lower cost

    A computable basic formula can be written as:

    1. Base Cost =
    2. Input miss tokens/1e6 × miss unit price
    3. + Input hit tokens/1e6 × hit unit price
    4. + Output tokens/1e6 × output unit price
    5. + Cache storage MB·hours × storage unit price
    6. + Cache write tokens/1e6 × write unit price
    7. Final Cost = Base Cost × Time Multiplier × Applicable Discount

    Note that “Applicable Discount” cannot be simply multiplied. Some public billing rules explicitly state that batch processing discounts and context cache discounts cannot apply simultaneously. In such cases, calculate the cost for both the “batch processing” and “cache” paths separately and choose the lower one, rather than multiplying the two discounts.

    3. Abstracting Four Common Pricing Structures

    While public pricing pages differ in details, they can generally be categorized into four structures. Understanding the structure is more important than memorizing specific numbers.

    First, time-based peak/off-peak pricing. Different unit prices apply to peak and off-peak hours, with potential price differences of up to 2x. The key point is that billing tiers are often determined by the request completion or settlement time. For long tasks crossing time slots, the scheduling layer should aim to push the completion time into the off-peak tier. For deferrable tasks, peak/off-peak scheduling directly reduces costs.

    Second, combined cache hit and cache storage pricing. Cache hit prices are usually much lower than miss prices, but caching is not free: cache storage is billed by capacity and duration, and writes may also be tiered by TTL. For short requests or scenarios with low repeated prefixes, cache benefits may be offset by storage fees. You should calculate net benefits based on “repeated prefix ratio × hit rate × TTL.”

    Third, batch processing discounts. Batch processing is suitable for deferrable tasks like offline evaluation, data cleaning, and batch generation, usually offering a percentage discount on the standard price. However, you must verify the discount scope: does it apply only to inputs, include outputs, include cache storage, or is it mutually exclusive with cache discounts?

    Fourth, low-latency or high-speed channel premium. For real-time interactions like online customer service, low latency holds business value but may carry a significantly higher unit price than standard channels. When selecting a model, convert SLA requirements into an acceptable upper limit for unit cost rather than blindly pursuing the lowest price.

    4. Turning Price Lists into a Computable Model with Pseudocode

    During the selection phase, it is recommended to write a lightweight cost simulator that takes pricing parameters and traffic logs as input and outputs the bill range under different strategies. The pseudocode is as follows:

    1. def estimate_cost(usage, price, peak=False, batch=False, cache_discount=False):
    2. mult = price["peak_mult"] if peak else 1.0
    3. base = (
    4. usage["in_miss"] / 1e6 * price["in_miss"] * mult
    5. + usage["in_hit"] / 1e6 * price["in_hit"] * mult
    6. + usage["out"] / 1e6 * price["out"] * mult
    7. + usage["cache_mb_hours"] * price["cache_storage"]
    8. + usage["cache_write"] / 1e6 * price["cache_write"]
    9. )
    10. candidates = [base]
    11. if batch:
    12. candidates.append(base * price["batch_mult"])
    13. if cache_discount:
    14. candidates.append(base * price["cache_discount_mult"])
    15. if price["stackable"]:
    16. candidates.append(base * price["batch_mult"] * price["cache_discount_mult"])
    17. return min(candidates)

    When simulating, do not just use the average input length. Use P50, P90, and P99 length distributions to calculate costs for normal requests, cache hit requests, long context requests, and retry requests. Output length requires separate modeling because the output unit price is typically higher than input, and output length is influenced by prompts, stop conditions, and max output limits.

    5. Selection Process: From Workload Profiling to Continuous Governance

    It is recommended to execute in five steps:

    1. Workload Profiling. Collect request volume, input length distribution, output length distribution, repeated prefix ratio, cacheable ratio, deferrable task ratio, peak/off-peak request ratio, and failure retry rate.
    2. Price Field Mapping. Map each service’s pricing page uniformly to the above billing fields, flagging missing or uncertain items to avoid masking differences with “default values.”
    3. Multi-Strategy Simulation. Simulate strategies like “standard channel only,” “peak/off-peak scheduling,” “increased cache hits,” “batch processing,” and “high-speed + standard channel mix,” outputting cost ranges and SLA risks.
    4. Low-Traffic Stress Testing. Use real, desensitized log replays to validate cache hit rates, completion time distribution, retry rates, and billing deviations.
    5. Online Monitoring. Split costs by project, tenant, model, and task type; set budget alerts and anomaly detection; regularly review pricing changes.

    In Baidu AI Cloud’s LLM service selection and operations scenarios, you can also integrate billing details, call logs, and budget alerts to form a cost dashboard categorized by project, tenant, and task type; specific fields depend on actual billing and monitoring capabilities.

    6. Engineering Optimization: Shifting Cost Control to the Calling Layer

    Cost optimization should not only occur at the procurement stage but should be integrated into the call chain:

    • Increase cache hit rate: Stabilize prompt prefixes, reduce dynamic content at the beginning, and split cache keys by business domain; simultaneously evaluate if cache storage fees are lower than the input fees saved.
    • Control output length: Set reasonable max output limits, use stop conditions, compress structured outputs, and adopt segmented strategies for long-text generation.
    • Peak/off-peak scheduling: Schedule offline tasks to complete during off-peak tiers, avoiding long tasks spilling into peak hours.
    • Optimize batch processing vs. caching: When discounts are non-stackable, choose the better path based on task deferrability and repeated prefix ratio.
    • Request merging and degradation: Merge small requests, reduce invalid retries, and downgrade non-critical tasks to low-cost channels.
    • Cache lifecycle management: Short-TTL caches suit high-frequency repeated prefixes, while long-TTL caches require calculating the balance point between storage costs and hit benefits.

    7. Common Pitfalls and Conclusion

    Common pitfalls include: looking only at input price while ignoring output price; looking only at cache hit price while ignoring cache storage and write fees; directly multiplying batch and cache discounts; ignoring that time-based billing is judged by completion time; using averages instead of long-tail distributions; and ignoring retry and failed requests. The final conclusion is: LLM API cost evaluation should establish a unified billing model, use real traffic distributions for simulation, incorporate caching, batch processing, peak/off-peak scheduling, and output control into engineering optimization, and continuously calibrate through monitoring and governance. Price lists are merely the starting point; the real bill is the ultimate basis for selection and optimization.