LLM API Price Hikes Explained: A Breakdown of Compute and Caching Costs

    This article examines why an open-source large-model vendor might raise API prices even while offering free, open weights. It breaks down the compute economics behind the decision: how online inference and offline training compete for the same accelerators, how cache-hit billing dilutes per-request revenue, and how pricing acts as a lever for load shaping. Using illustrative, non-official figures, it estimates API revenue at a scale of one trillion tokens per day, compares post-increase prices against other platforms, and weighs pay-as-you-go against subscription models. It closes with a practical cost-optimization checklist, a before-and-after billing example, and answers to common questions about caching, image-token billing, retries, and rate limits.

    1. The short answer: why raise prices on a free, open-source model?

    Free open source and higher API prices are not necessarily contradictory. This article uses a particular vendor (referred to as “the vendor”) as an observation subject; pricing and product details are governed by the vendor’s official documentation. Three explanations are most plausible:

    1. Online inference consumes large amounts of compute and competes with next-generation model training (Section 2).
    2. Cheap cache hits settle a large share of tokens at discounted rates, which lowers effective revenue (Sections 3 and 4).
    3. A price increase is a form of load shaping: it uses price signals to filter for high-value calls and frees up resources to feed back into training (Section 2).

    If you want to see how a price increase affects your own bill, Section 8 includes a billing example and a switching checklist.

    2. Why do inference and training compete for compute?

    A large-model vendor’s compute pool typically serves two kinds of workloads:

    • Online inference (serving): responds to users in real time, requiring low latency, high throughput, and high availability; the load fluctuates with call volume.
    • Offline training and post-training: includes pre-training, instruction fine-tuning, reinforcement learning from human feedback, and more; it is compute-heavy, long-running, and occupies accelerators for extended periods once started.

    The problem is that these two workloads may share the same expensive hardware. Whether training and inference clusters are shared, and to what degree, varies by vendor architecture; any estimate without public information should be treated as inference. When a free or low-price strategy pushes up inference call volume, the inference side consumes large amounts of card-hours and squeezes training resources — yet training the next generation of models is how a vendor stays competitive.

    1. # Simplified compute allocation
    2. Total available card-hours = online serving + offline training + redundancy/fault tolerance
    3. # When call volume spikes and the unit price is too low:
    4. # online serving usage ↑↑ → card-hours available for training ↓↓ → next-generation iteration may stall

    So “freeing up compute” usually follows one of two paths: scale up through procurement, or reduce the compute consumed per request (for example, by raising the cache hit rate, batching, quantization, or distillation). The former is constrained by high-end AI accelerator supply, delivery lead times, and channel premiums; the latter is mostly engineering optimization. One distinction matters: a price increase filters total call volume on the demand side and does not directly reduce per-request compute — it can only influence resource allocation indirectly.

    3. How many accelerators can API revenue buy?

    Many people assume “selling an API must be very profitable,” but the numbers are often less encouraging than expected once you lay them out.

    To discuss scale, this article assumes a daily volume of one trillion tokens. This assumption is illustrative only and does not represent any vendor’s real data; the unit prices below are likewise used only for order-of-magnitude estimates and do not reflect any vendor’s current pricing.

    1. # Revenue-side order-of-magnitude estimate (illustrative, not an official audit)
    2. Daily call volume = 1T tokens/day = 1 trillion tokens/day = 10^12 tokens/day = 10^6 million tokens/day
    3. Annual call volume = 1T/day × 365 = 365T/year = 3.65×10^14 tokens/year = 3.65×10^8 million tokens/year
    4. Assume all input settles at the low-price cache tier:
    5. Input unit price = CNY 0.05 per million tokens
    6. Annual revenue ≈ 3.65×10^8 × 0.05 ≈ CNY 18.25 million
    7. Output tokens are 10% of input tokens, i.e., output tokens = input tokens × 10%
    8. Output unit price = CNY 0.5 per million tokens:
    9. Output revenue ≈ 3.65×10^8 × 10% × 0.5 ≈ CNY 18.25 million
    10. Total revenue ≈ CNY 36.5 million

    Note that CNY 0.05 per million tokens is an example unit price for the low-price cache tier, not the standard input price. If you estimate using the standard input price of CNY 0.5–2 per million tokens shown in Section 5, and assume a higher output price (for example, around CNY 5–15 per million tokens), annual gross revenue could rise to roughly CNY 360 million–900 million. So “tens of millions to hundreds of millions” is not a fixed conclusion; it depends heavily on the cache discount, the standard input price, the output price, and the call mix.

    After deducting compute, bandwidth, operations, redundancy, and failure costs, net surplus is usually below gross revenue. A full deployment of high-end accelerators can cost millions or even tens of millions, so rolling out more training compute on the back of low-price API revenue is generally not a stable business loop.

    4. Why does cache billing lower the unit price?

    Context caching, or prompt caching, is the key variable for understanding an API bill. Input tokens for a large model are usually billed under two categories:

    • Cache-hit input: repeated or reusable context is already cached, so processing cost is lower; it is typically charged at a discount to the standard input price.
    • Cache-miss input: requires full recomputation and is charged at the standard input price.

    Note that a cache hit does not happen automatically; it usually requires several conditions: the prefix must match token-for-token from the start of the request; a vendor-defined minimum cacheable length must be met; the cache has a limited lifetime and may be evicted, so a hit is not guaranteed; and some vendors charge separately for cache writes or storage. Refer to the vendor’s documentation for specifics.

    As an example of one vendor’s published pricing (around 2025, subject to the official pricing page): the cache-hit price may be only a small fraction of the standard input price. Cache discounts and lifetimes vary by vendor, from minute-level to hour-level, with some strategies lasting longer; for procurement, refer to the latest statement on the vendor’s official pricing page.

    1. # Single-request cost/revenue estimate (illustrative)
    2. total_input = cache_hit_tokens + cache_miss_tokens
    3. # To derive from the hit ratio:
    4. # cache_hit_tokens = total_input × hit_ratio
    5. effective_cost =
    6. cache_hit_tokens * input_price * cache_discount
    7. + cache_miss_tokens * input_price
    8. + output_tokens * output_price
    9. # cache_discount: the cache-hit discount rate.
    10. # For example, 10% of the standard input price means cache_discount = 0.1.

    When the cache discount is deep, the lifetime is long, and the hit rate is high, a large share of tokens settles at the discounted rate. Users get a good experience, but the vendor’s revenue per request is significantly diluted. This is one common reason for “impressive usage, thin margins.” So a price change is not only about adjusting standard input/output prices — it can also be an opportunity to revisit the cache billing strategy.

    5. Is it still cheap after the increase? A horizontal price comparison

    A counterintuitive but important point: after a price increase, the absolute price may still be competitive against other vendors.

    The ranges below are illustrative only and should not be used directly for procurement comparisons. Unit: CNY per million tokens; illustrative, around 2025, non-official, excluding tax.

    Category Cache-hit input (example) Standard input (example) Output (example)
    Open-source model after the increase ~0.01–0.05 ~0.5–2 ~2–10
    Other top-tier platforms ~0.05–0.2 ~1–3 ~5–20

    The ranges in this table are not real quotes; for procurement, use the real-time prices on the official pricing page, contract discounts, and the actual call mix.

    From this relative price table, even after a several-fold increase, the open-source model may still sit at a relatively low point on cache price, input price, and output price. Three judgments lie behind this:

    1. Compared with the past: the increase is clear, enough to discourage low-value, bursty batch calls.
    2. Compared with competitors: the price may still be lower, retaining appeal for cost-sensitive developers.
    3. For resource allocation: price filtering shifts scarce compute from “spreading wide” toward “serving high-value scenarios + feeding back into training.”

    This looks more like a “trade price for volume” rebalancing: moving away from extreme low prices in exchange for scale, back to reasonable pricing to protect supply and iteration.

    6. Pay-as-you-go or subscription?

    The vendor bills mainly on usage; whether it offers monthly or volume-based subscriptions should be confirmed against the plans described in its official documentation. For developers, choosing a billing model depends on your usage curve:

    • Volatile workloads: such as bursty scraping or intermittent batch jobs — pay-as-you-go is more flexible and less wasteful.
    • Stable, high-load workloads: such as 24/7 online services — evaluate subscriptions, resource reservations, or volume packages for more predictable costs.

    The total-cost difference between the two models is usually not dramatic. What matters is accounting for historical call volume, peaks, peak-to-trough ratio, and retry rate before committing to a volume package. If a workload has batch or asynchronous tasks, also confirm batch discounts, latency requirements, and how failed retries are billed.

    7. How is vision capability introduced? The multimodal increment

    This round of updates may bring vision understanding. A typical approach is to release a separate experimental version with image-modality recognition on top of the existing text model. Text capability is roughly on par with the main release, with the main increment concentrated in vision understanding; refer to the official documentation for specifics.

    1. # Capability comparison (illustrative, not measured; refer to official evaluation reports)
    2. Code/terminal benchmarks: possibly a slight increase
    3. Software engineering / repo-level tasks: possibly a relatively clear improvement
    4. Automation/execution benchmarks: possibly a slight increase

    Note that the ranges above are illustrative, not measured conclusions. Different benchmarks, prompts, and toolchains all affect results. For a technical evaluation, go back to the benchmark name, score, timestamp, and source cited in the evaluation you reference, and avoid treating illustrative data as a basis for procurement.

    The choice should weigh three things: whether you have a stable image task; how image tokens are billed (by image block, resolution, size, or platform policy); and whether uploaded images are retained or used for training — check the data retention and privacy policies.

    This “ship a separate experimental version first, then merge it back into the main model” engineering strategy isolates risk and makes staged validation easy; the cost is that during the transition, users must explicitly select the vision version to use image input.

    8. How can developers optimize cost? A billing example and a switching checklist

    Once you understand the logic, you can optimize on two dimensions — cost and technology:

    1. Optimize context reuse: put stable, unchanging instructions, system prompts, and documents first to raise the cache hit rate.
    2. Control output length: output tokens usually cost more, so trimming redundant output has a significant effect on cost.
    3. Match the billing model: use pay-as-you-go for volatile workloads; evaluate subscriptions or resource reservations for stable ones.
    4. Watch for cache-policy changes: when a vendor changes prices, cache discounts and lifetimes may change too.
    5. Choose the modality on demand: enable the vision version only when there is a genuine image-understanding need.
    6. Build cost observability: use logs, monitoring, and alerts to track call volume, token consumption, and failed retries.

    Billing example before and after cache-hit optimization

    Assume monthly input of 1 billion tokens (1,000 million tokens) and output of 20 million tokens (20 million). Standard input price CNY 1 per million tokens, output price CNY 5 per million tokens, cache-hit discount rate 0.2.

    1. Before optimization:
    2. Cache hit = 200M × 1 × 0.2 = CNY 40
    3. Cache miss = 800M × 1 = CNY 800
    4. Output = 20M × 5 = CNY 100
    5. Total = CNY 940
    6. After optimization:
    7. Cache hit = 600M × 1 × 0.2 = CNY 120
    8. Cache miss = 400M × 1 = CNY 400
    9. Output = 20M × 5 = CNY 100
    10. Total = CNY 620
    11. Savings after optimization ≈ CNY 320, a reduction of about 34%.

    Should you switch vendors after a price increase? A checklist

    • Confirm the latest prices on the official pricing page: input, output, cache-hit, cache-miss, and retry billing.
    • Confirm the cache policy: discount rate, lifetime, and support for long-context reuse.
    • Recompute the monthly bill using your real token mix, not just the unit price.
    • Assess migration cost: convert prompt adaptation, evaluation regression, fine-tuning assets, and compliance and security into effort and budget.
    • Set budget alerts: apply tiered limits by project, application, and call volume.
    • Prepare alternatives: keep at least one replaceable model or vendor and run A/B tests.
    • For stable workloads, consider resource reservations; for volatile workloads, keep pay-as-you-go.

    Common questions and pitfalls

    What cache hit rate makes it worthwhile?
    Caching is more worthwhile when the standard input price is high, the stable prefix is long, and the cache lifetime covers your business cycle. If the prefix is short, the hit rate is low, or cache writes/storage are charged separately, the benefit may be offset.

    Is long context billed in segments?
    Most platforms bill by token, but long context may be affected by the context window, a minimum cacheable length, or segmentation strategies; check the documentation and the actual bill.

    How is image input billed?
    Images may be billed by conversion to tokens, resolution, size, or number of blocks, which varies widely across platforms; check the official billing documentation first.

    Are retries billed?
    Failed retries can still consume input/output tokens, especially for long-context tasks, so include retry cost in your monitoring.

    Are rate limits and quotas adjusted at the same time?
    A price increase may come with changes to TPM, RPM, or concurrency quotas; update load tests, alerts, and scaling plans accordingly.

    Are batch or asynchronous tasks cheaper?
    Some vendors offer discounts for batch or asynchronous calls, but confirm latency, failed-retry handling, quotas, and how results are delivered.

    Conclusion

    Free open source plus higher call prices is not necessarily a business contradiction; it is more likely a rational rebalancing under compute constraints. It reallocates the vendor’s scarcest resource — the card-hours of high-end accelerators — between inference and training, using price as a lever for load shaping.

    For developers, rather than fixating on “did the price go up or not,” it is better to understand the underlying cost structure and cache billing mechanism, and to make more robust long-term technology choices between the open-source ecosystem and cheap-rate dividends by optimizing how you call, matching the billing model, and building cost monitoring.

    If you remember only one thing: put stable prefixes first, monitor your hit rate and retry rate, and recompute the bill based on your real token mix.

    Pricing, cache policies, and version capabilities are time-sensitive; refer to the latest announcements on the vendor’s official documentation.