How to Choose an LLM API: From Token Pricing to Per-Task Cost

    Choosing an LLM API by token unit price alone can be misleading: capability upgrades don't always change the price, but they can change your bill. This article explains why the metric that matters is cost per effective task, walks through the three billing components (input, cached input, output) and the three rules that govern them, and gives a reusable method for translating token prices into real spending, including a sensitivity analysis of output-token growth. It then covers the two main cost levers — improving cache hit rate and compressing output through format constraints, staged generation, batching, and tiered model routing — followed by a four-step evaluation framework based on business success rate rather than leaderboard scores, and the cloud infrastructure needed to make cost observable and attributable. All prices in the examples are illustrative and used only to demonstrate the calculation method; refer to the provider's current pricing rules when making real decisions.

    1. Introduction: Prices Don’t Change, But Your Bill Might

    When a batch of new models opens up their APIs, developers often notice a pattern: model capabilities improve, but the official token price doesn’t necessarily go up or down. So “upgrading comes at no extra cost” starts to feel like an obvious conclusion. But what really drives your business spending is usually not the unit price — it’s how much it actually costs to complete a qualified task.

    A few things tend to show up in public benchmarks and engineering practice:

    1. Capability gains can come with longer outputs. For the same task, if a new model produces more output tokens — and output is usually the most expensive billing item — the actual cost can rise rather than fall.
    2. The gap between top models is narrowing. The selection question is shifting from “who is stronger” to “who is more stable and cheaper.”
    3. Per-task cost is diverging. Even with similar capability scores, differences in reasoning style, output length, and retry behavior mean the final bill can differ noticeably.

    The core metric for model selection, then, should not be just “price per million tokens” but “cost per effective task.” This article covers the common API-selection questions — how to choose, common mistakes, and how to calculate — and offers a reusable estimation method.

    Prices in this article are illustrative and used only to explain the calculation method. For real decisions, follow the pricing rules the model provider publishes at the time.

    2. Reading the Token Bill: Three Billing Items and Three Rules

    Mainstream LLM APIs typically bill by token. A typical price structure looks like this (per million tokens, in USD):

    Billing item Unit price Relative to input price
    Input 1.40 100%
    Cached input 0.26 ~18.6%
    Output 4.40 ~314%

    This structure carries three rules that matter a great deal for cost optimization.

    Rule 1: Output is usually the most expensive

    For most models, the output unit price is several times the input unit price. In scenarios like long-form generation, code completion, and multi-turn Agent calls, output tokens often make up the bulk of the bill. Optimizing output length is usually the fastest way to cut costs.

    Rule 2: The cache discount is large

    Input tokens that hit the cache are billed at a much lower price. In the table, cached input is about 18.6% of the standard input price — roughly an 80% discount. For long-context tasks, cache hit rate is one of the most important controllable variables in the cost structure.

    Rule 3: The input-to-output ratio is determined by the task

    With the same prices, different tasks have very different cost structures:

    Task type Input characteristics Output characteristics Dominant cost
    RAG Q&A Long, includes retrieved snippets Short, answer and citations Input
    Code completion Medium context Short completion snippet Input
    Long-form generation or rewriting Medium Long Output
    Document understanding and summarization Very long Medium Both input and output
    Long-running Agent Very long, includes tool definitions and history Medium per turn, longer cumulatively Input and output stacked

    The practical principle: for input-dominated tasks, prioritize raising the cache hit rate; for output-dominated tasks, prioritize compressing output length.

    3. Per-Task Cost Estimation: Translating Unit Price into Actual Spending

    3.1 Basic formula

    Suppose one successful call consumes I input tokens and O output tokens, of which C are cache-hit input tokens. Let p_in, p_cache, and p_out be the per-million-token prices for input, cached input, and output. The single-call cost is:

    1. Cost_call = (C × p_cache + (I - C) × p_in + O × p_out) / 1,000,000

    From there, distinguish two measures:

    1. Strict measurement: cost per effective task = total call cost ÷ number of successful tasks. Total call cost should include successful, failed, and retry calls; the number of successful tasks is the count of tasks that finally pass validation or meet the business standard.
    2. Budget estimation: for capacity planning, approximate with a parameterized formula. Let r be the average number of extra retries or failed calls per initiated task, and s the success rate:
    1. Cost_effective ≈ Cost_call × (1 + r) / s

    Note carefully: r and s must not double-count failure cost. If r already includes failed or retry calls, and s is also used to spread the cost of failed tasks, calibrate against “total call cost ÷ number of successful tasks” from the actual bill. If failed-call costs differ significantly from successful-call costs, track them separately.

    3.2 Estimation example

    Take a code Q&A scenario: 2000 input tokens, 2000 output tokens, no cache hit.

    Using the illustrative prices:

    1. Input cost = 2000 × 1.40 / 1,000,000 = 0.0028 USD
    2. Output cost = 2000 × 4.40 / 1,000,000 = 0.0088 USD
    3. Base cost = 0.0028 + 0.0088 = 0.0116 USD

    Output tokens account for about 75.9% of the total cost here.

    If each initiated task produces an extra 0.1 retries or failed calls on average, and the success rate is estimated at 95%:

    1. Task cost with retries = 0.0116 × 1.10 = 0.01276 USD
    2. Cost per effective task = 0.01276 / 0.95 ≈ 0.01343 USD

    So the example can be written as:

    1. input_tokens = 2000
    2. output_tokens = 2000
    3. price_in = 1.40
    4. price_cache = 0.26
    5. price_out = 4.40
    6. # Base cost: no cache
    7. base = (input_tokens * price_in + output_tokens * price_out) / 1_000_000
    8. # base = 0.0116
    9. # Each initiated task produces an extra 0.1 retries or failed calls on average
    10. per_task = base * 1.10
    11. # per_task = 0.01276
    12. # 95% success rate; budget-basis cost per effective task
    13. effective = per_task / 0.95
    14. # effective ≈ 0.01343

    In practice, count “total call cost” and “number of successful tasks” directly on the gateway or logging side, and use that as the final cost per effective task.

    3.3 Sensitivity analysis of output growth

    Suppose a new-generation model keeps the same unit price and the same input tokens, but output tokens increase by 20%. In the example, input stays at 2000 and output rises from 2000 to 2400:

    1. Input cost = 0.0028 USD
    2. Output cost = 2400 × 4.40 / 1,000,000 = 0.01056 USD
    3. Base cost = 0.0028 + 0.01056 = 0.01336 USD

    Following the budget basis with retries and success rate:

    1. Cost per effective task ≈ 0.01336 × 1.10 / 0.95 ≈ 0.01547 USD

    So a 20% increase in output tokens, in this output-dominated example, raises the base cost from 0.0116 USD to 0.01336 USD — an increase of about 15.2%.

    More generally, let r_out be the share of output cost in total cost. A 20% increase in output tokens then produces a cost increase of:

    1. Δ ≈ 20% × r_out

    Different tasks are affected to different degrees:

    Task type Output share (illustrative) Cost increase after 20% output growth
    Code Q&A ~75.9% ~15.2%
    Multi-turn Agent ~64% ~12.8%
    Document summarization ~54% ~10.8%
    Long-document understanding ~35% ~7%
    Short question classification ~10% ~2%

    So when evaluating a new model, don’t just look at capability scores — re-test output length on your own business data. The longer a task’s output, the more sensitive it is to output growth.

    4. The First Cost Lever: Improving Cache Hit Rate

    4.1 How cache hit rate affects the bill

    The cache usually hits by prefix matching. Only when the prefix is completely identical can subsequent tokens be billed at the cache price. So in long contexts, the earlier parts are more likely to hit the cache.

    Take a single call with 4000 input tokens and 500 output tokens:

    Cache hit rate Base cost (USD) vs. no cache
    0% 0.00780 —
    50% 0.00552 down ~29.2%
    80% 0.004152 down ~46.8%
    95% 0.003468 down ~55.5%

    The calculations:

    1. 0%: 4000 × 1.40 + 500 × 4.40 = 7800, i.e., 0.00780 USD
    2. 50%: 2000 × 1.40 + 2000 × 0.26 + 500 × 4.40 = 5520, i.e., 0.00552 USD
    3. 80%: 800 × 1.40 + 3200 × 0.26 + 500 × 4.40 = 4152, i.e., 0.004152 USD
    4. 95%: 200 × 1.40 + 3800 × 0.26 + 500 × 4.40 = 3468, i.e., 0.003468 USD

    Note that if a task has a very high input share and a tiny output, a 95% cache hit sharply lowers the input-side cost. Looking only at the input portion, a 95% hit brings input cost down to about 22.7% of the original — a reduction of about 77.3%. But the total-cost reduction still depends on the output share.

    4.2 Four cache optimization strategies

    1. Prefix stabilization. Put unchanging content — system instructions, tool definitions, knowledge base snippets, few-shot examples — at the very front of the prompt; put dynamic content like timestamps, session IDs, and user input at the end. For multi-turn conversations, maintain history in an “append-only, no rewriting” way to avoid cache invalidation from changes to preceding content.

    1. import json
    2. from dataclasses import dataclass, asdict
    3. @dataclass
    4. class ToolSpec:
    5. name: str
    6. description: str
    7. parameters: dict
    8. @dataclass
    9. class ChatMessage:
    10. role: str
    11. content: str
    12. def build_messages(system_prompt: str,
    13. tools: list[ToolSpec],
    14. history: list[ChatMessage],
    15. user_input: str) -> list[ChatMessage]:
    16. """Build a message list with a stable prefix + mutable suffix to improve prefix cache hit rate."""
    17. messages = [ChatMessage("system", system_prompt)]
    18. # Keep tool definitions stable, using standard JSON serialization
    19. for tool in tools:
    20. tool_json = json.dumps(asdict(tool), ensure_ascii=False)
    21. messages.append(ChatMessage("user", tool_json))
    22. # Append-only history, no rewriting
    23. messages.extend(history)
    24. # Put the current user input at the end
    25. messages.append(ChatMessage("user", user_input))
    26. return messages

    2. Batch-shared prefix. If a batch of tasks shares the same instructions, tool definitions, or document content, schedule and run them together so the prefix is continuously hit within the cache’s validity window, avoiding a cold start every time.

    3. Template version management. Keep prompt templates under version control. Evaluate every change for its impact on cache hit rate, output length, and task cost, so frequent edits don’t cancel out the cache benefit.

    4. Include hit rate in monitoring. On the gateway side, record each call’s cached token count, uncached token count, and hit rate, aggregated by business line, scenario, and model. A sudden drop in hit rate often signals a change in prefix structure or template.

    5. The Second Cost Lever: Compressing Output and Optimizing the Call Chain

    5.1 Constrain the output format

    Setting a clear output structure for the model effectively reduces token fluctuation and downstream parsing cost:

    1. Use field-level schema definitions.
    2. Set upper limits on field lengths.
    3. Use controlled enum values instead of free text.
    4. Automatically validate structured output; on failure, retry only the failed part rather than regenerating the whole thing.

    The more constrained the output, the more stable the token consumption — and the lower the cost of manual review and parsing.

    5.2 Staged generation

    Complex generation tasks can be split into two stages:

    1. The first stage outputs a brief plan first — for example, a list of key points or a JSON structure.
    2. The second stage fills in the details in a constrained format and self-checks whether all key points are covered.

    Constraining output length at each stage separately helps suppress the “gets longer and longer” problem. General conversational products can also set a default response length in the system instructions, keeping an explicit trigger for long outputs.

    5.3 Batch processing and asynchronization

    Tasks like offline summarization, batch labeling, and data cleaning usually have low real-time requirements and can be done at a lower unit price via batch APIs. With a message queue, deferrable tasks can be scheduled to off-peak hours, lowering cost while protecting the service quality of the online path.

    5.4 Tiered model routing

    Not every request needs the strongest model. Build a tiered model pool by task difficulty:

    Task type Recommended tier
    Classification, extraction, format conversion Low-cost small model
    Regular dialogue, summarization Mid-tier model
    Code, complex reasoning High-capability model
    Long-running Agent High-capability model, with strengthened tool calling and retry strategies
    1. def select_model(prompt: str, need_tools: bool,
    2. history_len: int, tier: str) -> str:
    3. """Illustrative model routing strategy."""
    4. if tier == "simple":
    5. return "small"
    6. if history_len > 8 or not need_tools:
    7. return "mid"
    8. if need_tools and tier in {"complex", "agent"}:
    9. return "large"
    10. if tier == "code":
    11. return "large"
    12. return "mid"

    The core of tiered routing is to let simple tasks go to cheap models and reserve high-capability models for the scenarios that truly need them. Routing rules need continuous calibration: misjudgment pushes simple tasks to expensive models and wastes cost, while insufficient judgment lowers the success rate of complex tasks and increases retry cost.

    6. Evaluation Framework: From Leaderboard Scores to Business Success Rate

    Capability leaderboards are useful for initial screening, but the final decision should rest on your own evaluation system. Build it in four steps:

    1. Build a business evaluation set. Sample from production logs by scenario in layers, covering high-frequency, long-tail, and edge cases, and desensitize the data.
    2. Run two metrics in parallel. For each task, record both “whether it passes validation” and “how many tokens it consumed and how much it cost,” yielding a task success rate and an average cost per task.
    3. Fixed-load comparison. Run the new and old models on the same evaluation set and compare the change in output-token distribution per task type. If one task type’s output rises overall, the new model is likely more verbose and the prompt needs adjusting.
    4. Gray release with continuous recalculation. Roll out the new model to a small traffic gray release first, validate the evaluation conclusions with real traffic, then ramp up gradually. Recalculate after model or pricing-rule changes to avoid runaway costs.

    A reference evaluation table is below. Costs are computed with the same illustrative prices; if a model’s prices differ, replace the price parameters.

    Task ID Scenario Model Input tokens Cache hit Output tokens Validation result Cost (USD) P95 latency (ms)
    T-001 Code Q&A large 2000 0 2000 Pass 0.0116 3200
    T-002 Document summarization mid 8000 6400 600 Pass 0.006544 4100

    T-002 is calculated as:

    1. Uncached input: 8000 - 6400 = 1600
    2. Input uncached cost: 1600 × 1.40 / 1,000,000 = 0.00224 USD
    3. Cache hit cost: 6400 × 0.26 / 1,000,000 = 0.001664 USD
    4. Output cost: 600 × 4.40 / 1,000,000 = 0.00264 USD
    5. Total cost: 0.00224 + 0.001664 + 0.00264 = 0.006544 USD

    A composite score can be designed as a weighted combination of success rate, cost, and latency. For “lower is better” metrics like cost and latency, use min-max normalization:

    1. Normalized score = (max value - current value) / (max value - min value + tiny value)

    If a task has the lowest cost and lowest latency, its normalized score approaches 1; otherwise it approaches 0. You can also use threshold mapping — for example, set a target cost and target latency, after which the score drops quickly once the threshold is exceeded. The final score should factor in business weights, not just a single leaderboard score.

    7. Cloud Deployment: Making Cost Observable and Attributable

    Model tuning is just one layer of the application. To make cost truly visible and controllable, you also need cloud infrastructure support. Combine the following general capabilities as needed:

    1. Object storage: hold evaluation sets, prompt-template versions, and archived call logs as the raw data for cost accounting.
    2. Message queue: handle peak shaving and valley filling for batch async tasks, scheduling deferrable work to off-peak hours.
    3. Container platform: deploy a unified model-calling gateway that centrally implements model routing, rate limiting, retry, and degradation strategies.
    4. Log service: structurally record each call’s model identifier, input token count, output token count, cache hit amount, latency, and cost, tagged by business line, team, and scenario.
    5. Monitoring and alerting: set thresholds and fluctuation alerts for cost per effective task, cache hit rate, success rate, and P95 latency. When cost goes abnormal, you should be able to quickly locate the specific caller.
    6. Key management: centrally manage model-service credentials, avoid hardcoding, and allocate quotas and audit permissions per caller.

    The unified gateway layer is the core hub of this system. Only when all model calls go through a single, centralized exit can cost data be collected, attributed, and governed.

    8. Conclusion: The Next Phase of Competition Is Per-Task Cost

    As top-model capability scores converge, the focus shifts from “who is smarter” to “equally smart, who is cheaper and more stable.” For developers and enterprises, three moves matter most:

    1. Translate unit price into task cost. Measure cost per effective task with real business data instead of staring only at the token unit price.
    2. Build optimization into the architecture. Stable prefixes, batch reuse, output constraints, and tiered routing usually deliver more direct gains than frequently switching models.
    3. Turn cost into an observable metric. With a gateway, log service, and monitoring alerts, make every dollar of cost traceable and attributable.

    Common API-selection mistakes cluster into a few types: looking only at unit price and not task cost, looking only at leaderboards and not business success rate, ignoring output-length drift, changing cache prefixes frequently, missing or double-counting failure and retry costs, and misrouting simple tasks to high-cost models. Avoid these pitfalls, and a team can — while keeping quality intact — turn model cost from a “black-box bill” into a “manageable metric.”