LLM API Pricing Breakdown: The Real Cost of Input, Output, and Cache Hits

    This article breaks down how large language model API pricing actually works and why reading the price sheet at face value often leads to wrong cost decisions. It explains the three components of a single API call—input tokens, output tokens, and cached input—and shows why output can dominate the bill for coding, long-document, and agentic workloads. It then covers the inference mechanics behind the input/output price gap, how to judge generational price hikes by cost per successful task rather than unit price, how subscription quotas compare with pay-as-you-go billing, best practices for cache-friendly prompt orchestration, and a framework for real-time, attributable cost governance.

    1. Start With the Unit Price: A Single Call Is Actually Billed in Three Parts

    Most mainstream large-model APIs today use pay-as-you-go billing based on “token usage × unit price,” but the bill actually consists of three parts:

    • Input: all prompt tokens in the request, including system instructions, tool definitions, conversation history, retrieved documents, and the user’s question.
    • Output: tokens generated by the model, including the response body, tool-call arguments, and any reasoning traces that are produced and returned.
    • Cached Input: input tokens that hit the server-side prefix cache, typically billed at a fraction of the normal input price (often one-tenth to one-twentieth). It is a subset of the input and is deducted from the input count at settlement, not billed on top of it.

    Looking at just the three numbers—input, output, cache—it is easy to conclude “input is 20, output is 100, cache is 1, so output is the most expensive.” For real workloads like coding, long documents, and agents, this intuition is often wrong.

    Assume a given model generation charges $20 input, $100 output, and $1 cache hit (all in USD per million tokens). For a request with 100,000 input tokens and 2,000 output tokens, the cost is:

    1. cost = input_tokens × [(1 - cache_hit_rate) × input_price + cache_hit_rate × cache_price]
    2. + output_tokens × output_price
    Cache hit rate Input cost ($) Output cost ($) Total ($) Output share
    0% 2.00 0.20 2.20 9%
    50% 1.00 0.20 1.20 17%
    90% 0.20 0.20 0.40 50%
    100% 0.00 0.20 0.20 100%

    The conclusion is straightforward: the higher the cache hit rate, the closer input cost approaches zero—and the larger output’s share of the bill becomes. When almost all input is cached, cost is determined almost entirely by output tokens.

    Look at it another way. If the cache hit rate is only 10%, the average price per “billed input token” is about $19.1, still close to the regular input price. Whether the cache discount actually materializes depends on how “prefix-friendly” your request orchestration is (see Section 5).

    Next, here is a per-thousand-calls estimate for typical workloads (using the same prices, all illustrative assumptions):

    Workload Input tokens/call Output tokens/call Cache hit rate Input cost ($) Output cost ($) Total ($) Cost per 1,000 calls ($)
    Online Q&A 800 400 10% 0.014 0.040 0.054 54
    Code completion 4,000 80 60% 0.032 0.008 0.040 40
    Long-document analysis 60,000 800 30% 0.840 0.080 0.920 920
    Agentic coding 120,000 5,000 85% 0.360 0.500 0.860 860

    These numbers reveal an easily overlooked fact: in coding scenarios, the output unit price is 5× the input price, which sounds dramatic. But in agentic coding—a workload with many iterative rounds and repeated tool calls—output cost actually exceeds input cost and becomes the largest line item on the bill. The reason is that this kind of workload regenerates the plan, code, and commands every round, while the context keeps accumulating and is amortized through prefix caching.

    2. Why Output Costs More and Cache Costs Less: A Look at Inference Mechanics

    The price gap is not arbitrary—it reflects the different hardware characteristics of two computation phases.

    • Prefill (processing input): the entire input is fed to the model at once; the matrix operations are highly parallel and GPU compute utilization is high.
    • Decode (generating output): tokens are generated one at a time autoregressively. Producing each token requires reading the full model weights and the existing context cache again, so it is mainly constrained by memory bandwidth. Parallelism is low and hardware utilization is low, so the compute cost per token is significantly higher than in prefill.

    This is why output costs more than input. A cache hit means the request prefix’s key-value cache (KV cache) can be reused directly, eliminating that portion of prefill computation, so providers have room to offer a steep discount.

    Conversely, “output at 5× input” only means the model wants to charge more on “read-heavy, write-light” workloads and places tighter constraints on output-heavy scenarios. But engineering techniques like batch inference, inference-acceleration frameworks, KV cache reuse, and speculative decoding continually change the real compute ratio between input and output, loosening the static price gap. Generational upgrades bring stronger capabilities, and if providers raise prices at the same time, the upward drift in each generation’s “absolute unit price” is understandable.

    It is also worth stressing: the cache is not a “free lunch on by default”—it is an engineering capability with explicit terms.

    • Strict prefix matching: usually based on prefix hashing of the token sequence, matching at the granularity of a single token at finest. A change to any token in the system prompt, tool definitions, or few-shot examples causes everything after it to “miss.”
    • Minimum length threshold: there is generally a minimum cacheable length (often hundreds to a thousand-plus tokens), so short requests cannot hit.
    • Cache lifetime: usually a relatively short time-to-live (on the order of minutes), and it may be evicted early under capacity pressure.
    • Write overhead: some services charge a cache-write fee at a rate higher than normal input, so a one-off call can cost more than not using the cache at all.
    • Billing basis: some services bill the hit portion at a certain fraction of input tokens; others (especially where reasoning is visible) may also charge separately for returned reasoning content. When calculating, you must check each contract’s terms one by one.

    3. Judging Generational Price Hikes: Unit Price Is Not Cost

    Faced with information like “this generation’s API price is higher overall than the last,” concluding directly that “it is a lot more expensive” is not rigorous. What procurement and architecture decisions really care about is cost per successful task:

    1. cost per successful task = (Σ cost of each call) / number of successfully completed tasks
    2. = average calls per task × average cost per call / task success rate

    This breakdown shows that a unit-price increase is offset or amplified by three forces:

    • Higher success rate: the new model passes complex tasks on the first try more often, reducing retries, fallbacks, and human intervention, which lowers the number of calls.
    • Context strategy changes: stronger long-context ability leads developers to stuff in more material at once, driving up input token usage.
    • Reasoning behavior changes: if the model’s “think-as-it-goes” content is returned and billed, or if reasoning counts toward output, output token volume can multiply.

    Here is an illustrative example (method demonstration only):

    • Previous generation: unit-price index of 1, task success rate 60%, average attempts per task of 1/0.6 ≈ 1.67, effective cost index of 1.67.
    • New generation: unit-price index of 2, success rate up to 90%, average attempts down to about 1.11, effective cost index of 2.22.
    • The unit price doubled, but effective cost rose only about 33%; if the success rate reaches 95% (1/0.95 ≈ 1.05), the effective cost index is about 2.11, an increase of only around 26%.

    This is a case where a price hike can still be “worth it.” Conversely, if the price merely doubles with no clear capability gain, effective cost per unit doubles in step. So the basis for judgment should not be the unit-price table from a launch event, but measured data on a fixed test set:

    • Task success rate (e.g., test-case pass rate, compile pass rate);
    • Average input, output, and cached tokens consumed per task;
    • Retry count and human rework rate;
    • End-to-end latency: interactive latency is itself a cost to the experience.

    Here is reusable cost-accounting pseudocode that can plug directly into your existing call wrappers and logging:

    1. # Price unit: USD / million tokens; cached_tokens is a subset of input_tokens
    2. PRICES = {
    3. "default": {"input": 20.0, "output": 100.0, "cache": 1.0},
    4. "mini": {"input": 2.0, "output": 8.0, "cache": 0.2},
    5. }
    6. def call_cost(usage, model="default") -> float:
    7. p = PRICES[model]
    8. uncached = max(usage.input_tokens - usage.cached_tokens, 0)
    9. return (uncached * p["input"]
    10. + usage.cached_tokens * p["cache"]
    11. + usage.output_tokens * p["output"]) / 1_000_000
    12. def task_cost(records):
    13. """records: all call records generated by one task; each has usage / model / success"""
    14. total = sum(call_cost(r.usage, r.model) for r in records)
    15. success = any(r.success for r in records)
    16. return {"cost": total, "success": success}
    17. def cost_per_success(tasks):
    18. spent = [t for t in (task_cost(r) for r in tasks) if t["success"]]
    19. return sum(t["cost"] for t in spent) / max(len(spent), 1)

    With this accounting, you can benchmark candidate models and different generational versions on a fixed test set, directly comparing cost per successful task instead of comparing sticker prices.

    4. Subscription Quotas vs. Pay-as-You-Go: Why the Perceived Gap Closes Fast

    Many teams assume “subscription plan = cheaper” when purchasing, but subscription plans and API pay-as-you-go billing are two entirely different constraint systems:

    Dimension API pay-as-you-go Subscription quota plan
    Billing Billed separately by input/output/cache tokens; pay for what you use Fixed periodic payment for a usage allowance
    Usage boundary Constrained by account balance and rate limits Constrained jointly by total quota and rate caps
    Quota allocation Usually a shared balance May split into separate quota pools by scenario, e.g., chat, coding
    Burst capacity Obtained by raising rate limits or requesting higher quotas Constrained by rate caps and peak-reset policy
    Best fit Production workloads with volatile usage that need horizontal scaling Everyday interactive use by individuals and small teams

    Back to the prices set at the start (input $20, output $100, cache hit $1), a rough calculation of theoretical cost for a few typical workloads:

    • A single call with 10,000 input tokens and 2,000 output tokens costs about $0.40;
    • An agent call with 100,000 input tokens and 5,000 output tokens costs about $3.5;
    • A deep-analysis task with 500,000 input tokens and 20,000 output tokens costs about $22.

    None of these looks outrageous on its own. The problem is that typical usage under a subscription is repeated iteration: retrying after failures, continuous multi-file edits, repeatedly resubmitting long contexts—so single-call cost gets multiplied by a dozen or even dozens of times. Suppose a subscription plan can redeem roughly the equivalent of 20 million input tokens plus 1 million output tokens per month, which under ideal conditions is worth about $222—basically on par with API pay-as-you-go pricing. Add the experience loss from output limits, and the advantage is further erased.

    Beyond usage, there are two easily overlooked constraints:

    1. Rate limits: tokens per minute (TPM), requests per minute (RPM), and concurrent requests.
    2. Quota reset mechanism: rolling window or fixed window, plus burst capacity. Some platforms temporarily enlarge quotas for burst traffic, but this headroom is “temporary elasticity” and cannot serve as a basis for capacity planning.

    This leads to a few common misconceptions:

    • Looking only at the sticker price: a high unit price is “wrapped” by the quota in subscription scenarios and does not feel expensive; but once you exceed the quota and switch to pay-as-you-go, cost can jump by an order of magnitude.
    • Counting tokens but not time: the hidden cost of a plan is waiting. When there are many tasks, wait times stretch out, and the loss in productivity often exceeds the money saved.
    • Ignoring resets: if a quota runs out but resets a few minutes later, it is perceived as a brief wait; if it runs out right before a task peak and the reset time is far off, the experience falls off a cliff.

    The conclusion: subscription plans and APIs are not an either/or of “which is cheaper”—they serve different purposes. Interactive individual use can prioritize a plan; production system integration, batch tasks, and scenarios needing elastic scaling or stable latency should center on the API, and you should stress-test its rate limits with real workloads in a test environment.

    5. Cache-Friendly Request Orchestration Best Practices

    Whether the cache discount materializes depends on whether orchestration follows these principles:

    1. Fix the prefix by layering: order from “the more stable, the earlier”: system instructions and output rules → tool definitions (keep serialization order stable) → long-document or codebase context → few-shot examples → this round’s user input.
    2. Place dynamic content last: timestamps, request IDs, random numbers, and personalized info all go after the stable prefix; when you need to show the current time, fetch it via a tool call.
    3. Append to the prefix in multi-turn conversations: each turn only appends new content at the end. Once you rewrite, summarize, or trim the history, the previously established prefix chain is invalidated entirely.
    4. Segment and cache long documents: split by relatively stable sections or files, and reuse the same prefix when the question changes.
    5. Track the hit rate: bring the cache hit rate into monitoring, and investigate prompt drift and template changes when it falls below a threshold.
    6. Control output length: constrain the answer format and length, set a maximum output length, and avoid verbose explanations that inflate output cost.
    7. Standardize tool calls: have the model get real-time information by calling tools rather than splicing volatile content into the system prompt.
    8. Route by tier: handling simple requests with a smaller model first, then escalating to a stronger model based on confidence or complexity, is an effective way to suppress average cost.

    6. Cost Governance: From “Reviewing the Bill at Month-End” to “Real-Time Attribution”

    The most direct impact of a generational price hike is on budget. A governance framework can start with three lines of defense:

    • Attribute to business dimensions: record the business identifier (tenant, project, feature module), the model used, all three token counts, and task success/failure in call logs, pushing cost down to each feature or even each user.
    • Set quotas and circuit breakers: assign token and monetary budgets to each feature module, alert when approaching a threshold, and downgrade to a smaller model or queue for execution once exceeded.
    • Build budget regression tests: bring cost testing into the CI pipeline. After prompt, tool-definition, or model-version changes, automatically rerun the test set, and fail the pipeline once per-task cost or cost per successful task exceeds a baseline threshold.

    On cloud platforms such as Baidu AI Cloud, general-purpose capabilities like log services, monitoring and alerting, message queues, and container platforms can be combined to support this closed loop:

    • Call logs flow through a message queue into a log service;
    • Monitoring and alerting handle real-time reminders for hit rate and budget thresholds;
    • Batch tasks are queued and scheduled on a container platform.

    This way, “cost” becomes an engineering metric that can be continuously measured, just like “availability.”

    7. Conclusion

    1. See the structure: a single call consists of three parts—input, output, and cache hit—and the more fully the cache is hit, the more the cost structure tilts toward output.
    2. Compare effective cost: to judge whether a generational price hike is worth it, compare cost per successful task, not unit price.
    3. Distinguish positioning: subscription plans and API pay-as-you-go have different constraints, and perceived cost is determined jointly by “usage × available rate × wait time.”

    For developers, the real controllable levers have never been waiting for the next price cut, but continuously managing three variables: cache hit rate, output length, and number of calls.