Model API Token Prices Are Unchanged, But Your Bill Is Rising: Task-Level Cost

    This article explains why model API bills can increase even when per-token prices remain unchanged. It shows how longer reasoning outputs, multi-step agent workflows, tool calls, retries, and failures affect the real cost of each task. It also introduces a cost model based on cost per successful task, explains how to compare models beyond sticker price, and describes practical ways to control spending with caching, model routing, budget guards, and observability.

    When a model API’s per-token price stays the same, why does the monthly bill still go up?

    This is often the first question teams ask when switching from an older model to a newer one. After a new generation of Model A opened API access, its overall intelligence score rose from 53 to 60, while its token price remained unchanged from the previous generation. On the surface, this looks like better capability without a direct price increase. In practice, however, the real bill is not determined by unit price alone. Longer reasoning outputs, more complex agent chains, more tool calls, and more failed retries can all increase the actual cost of a single task.

    This article does not ask which model is the strongest. Instead, it answers three engineering questions:

    • Why can the bill rise even when the per-token price stays unchanged?
    • How can you tell whether a supposedly “cheaper” model is actually cheaper in practice?
    • How can caching, routing, budget guards, and observability bring the cost per unit of intelligence down to an acceptable range?

    1. The Phenomenon: Capability Improves While Price Stays Flat

    The figures below are compiled from public sources and should be read as an example. The reference date is June 2026. The task set covers short question answering, code generation, long-document summarization, and multi-step agents. The cost per task includes input tokens, output tokens, and failed retries, but does not assume cache hits. Monetary values are rounded to four decimal places. For comparison, some models are referred to as Model A, Model B, and Overseas Model C. These labels are used only to explain the evaluation method and do not constitute a selection conclusion.

    Billing item Example price (USD per million tokens) Notes
    Input 1.40 Regular input tokens
    Output 4.40 Model-generated tokens, about 3.1 times the input price
    Cached input 0.26 Input tokens that hit the cache, about 19% of the regular input price

    Using the same example evaluation, the cost per task is as follows:

    Model Intelligence score Cost per task Relative baseline
    Model A, new generation 60 About USD 0.68 Baseline
    Model B 60 About USD 0.84 About 24% higher
    Model A, previous generation 53 Not disclosed —
    Overseas Model C — About USD 1.23 About 81% higher

    Two conclusions stand out.

    First, the same score does not mean the same price. Model A’s new generation and Model B have similar intelligence scores, but the example cost per task of Model A is about 19% lower. Compared with Overseas Model C, the cost per task of Model A is about 55% of Model C’s cost, which means potential savings of about 45% on the same task set.

    Second, an unchanged unit price does not mean an unchanged task cost. The example evaluation shows that the new generation consumes about 20% more output tokens per task than the previous generation. If a team builds its budget only from the price list, it is likely to underestimate the real bill.

    Therefore, evaluating model cost should not stop at asking, “How much does it cost per million tokens?” It should also ask:

    • How many API calls does each task require?
    • How many input tokens are consumed?
    • How many output tokens are generated?
    • How many retries occur?
    • How many tasks finally succeed?

    2. The Real Cost Model: Unit Price Is Only the List Price

    The list-price cost of a single API call can be written as follows:

    1. Call cost = regular input tokens × input price
    2. + cached input tokens × cached input price
    3. + output tokens × output price

    In engineering practice, one detail matters: many logs report prompt_tokens as the total input, while the portion that hits the cache is usually placed in the cached_tokens field. When calculating cost, you should separate regular input from cached input. Otherwise, the cost may be overestimated or underestimated.

    At the task level, retry count and success rate must also be considered:

    1. Total cost per task = call cost × (1 + retry count)
    2. Cost per successful task = total cost per task ÷ task success rate

    Calculating the cost per successful task is important because failures, timeouts, truncations, and format-validation errors can still generate API charges. If you only look at the cost of successful samples, you may miss the hidden cost caused by failures.

    2.1 Four Variables That Determine Cost

    Variable Typical change How it affects cost
    Output token volume Longer reasoning chains or more verbose structured output Multiplied by the output price; usually the largest lever
    Task chain length From single-turn Q&A to multi-step tool use Input and output costs grow roughly linearly with the number of turns
    Cache hit rate Stable long prefixes or repeated context in multi-turn conversations Savings equal to input share multiplied by the price gap
    Retries and failures Timeouts, rate limits, format-validation failures Failed calls are still billed, and they also lengthen the task

    What really drives the bill is often not the unit price, but how many tokens each task needs and how many of those tokens can hit the cache.

    2.2 Cost Structure Differences Across Task Types

    Let the input be I and the output be O, both measured in million tokens. Let the regular input price be p_i = 1.40, the output price be p_o = 4.40, and the cached input price be p_c = 0.26.

    Task type Input I Output O Input token share Calculation (USD) Cost per task
    Short Q&A / text classification 0.003 0.0002 94% 0.003 × 1.40 + 0.0002 × 4.40 About USD 0.0051
    Code generation (about 500 lines) 0.006 0.003 67% 0.006 × 1.40 + 0.003 × 4.40 About USD 0.0216
    Long-document summarization (about 100,000 characters) 0.100 0.002 98% 0.100 × 1.40 + 0.002 × 4.40 About USD 0.1488
    Multi-step agent task 0.080 0.008 91% 0.080 × 1.40 + 0.008 × 4.40 About USD 0.1472

    Note: the “input token share” in the table means the share of input tokens in total tokens, that is, I / (I + O). It is not the share of input cost, nor the share of output cost. Later, when discussing output inflation, we will use a different measure: output cost share.

    A few observations follow:

    • Input-dominated tasks, such as classification, extraction, and long-document summarization, often have input token shares above 90%. For these tasks, the priority should be reducing input length or using prefix caching.
    • Output-dominated tasks, such as code generation, are affected strongly by the output price, which is more than three times the input price. The main cost-reduction approach is to reduce invalid output, constrain formats, and avoid overly long explanations.
    • In agent tasks, every turn carries the tool definitions and historical messages again. Input tokens are therefore repeatedly amplified. The combination of output inflation and more turns is the main reason why long-running task costs can rise sharply.

    3. Output Inflation: Why the Bill Rises Even When the Unit Price Does Not

    The additional output tokens from a newer model often come from three sources:

    • Explicit or implicit reasoning placeholders.
    • More detailed explanations generated to improve success rates on complex tasks.
    • Redundant whitespace, repeated fields, or unnecessary explanations in structured output.

    These three sources do not have the same value. The first two may lead to a higher task success rate, so they can be considered effective investment. The third may simply be waste.

    However, a 20% increase in output does not necessarily mean a 20% increase in the bill. Let the output growth rate be r, and let the share of output cost in total cost be s_o. Then the approximate cost increase is:

    1. Cost increase ≈ r × output cost share

    Using the examples above, we can estimate:

    Task type Output cost share Cost increase when output rises by 20%
    Short Q&A / text classification 17% About 3%
    Code generation 61% About 12%
    Long-document summarization 6% About 1%
    Multi-step agent task 24% About 5%

    That is, the same “20% output increase” can raise the bill by anywhere from about 1% to about 12%, depending on the business scenario.

    But this is only an ideal estimate when the number of turns stays unchanged. In agent scenarios, there is an amplification effect: once the context in a single turn becomes longer, the input and output costs of later turns may also increase. For long-running tasks, the real increase can be clearly higher than a linear estimate.

    On the other hand, stronger capability may also create “negative inflation.” If a task succeeds on the first attempt, requires less code rework, or needs fewer tool-call trials, the number of invalid tokens may decrease. Part of the reason why Model A has a lower cost per task than a similarly scored model may come from this kind of offsetting effect.

    So, “unit price × token volume” can only estimate the list-price cost. When comparing different models, a more reliable common yardstick is the cost per successful task.

    4. Evaluation Framework: Compare Models Using Cost per Successful Task

    4.1 Six-Step Evaluation Method

    1. Define the task set: Sample representative tasks from production logs. A typical range is 200 to 500 tasks, covering common input lengths, output lengths, and difficulty levels. If production logs are unavailable, an open evaluation set can be used for a cold start.
    2. Freeze evaluation conditions: Use the same system prompt, tool definitions, temperature, and other sampling parameters. Pin or record the model version so that evaluation results do not drift when the service is updated.
    3. Run multiple samples: Run each task at least three times. Record token usage, latency, whether a retry occurred, and the retry reason. Calculate the mean and P95.
    4. Determine success: Use automatically executable checks to determine whether a task succeeds. Examples include unit-test pass rate, JSON Schema validation, exact answer matching, or manual spot checks.
    5. Calculate unit cost: First calculate the total cost per task, then divide it by the success rate to get the cost per successful task.
    6. Run sensitivity analysis: Evaluate costs when the cache hit rate is 20%, 40%, 60%, and 80%, and when the failure rate rises from 2% to 10%.

    A task-set template should contain at least the following fields:

    1. task_id, business_scene, input_length, output_length_limit,
    2. tool_count, difficulty_level, success_criterion, max_retries,
    3. budget_limit, expected_latency

    4.2 Cost Calculation Example

    1. import json
    2. from dataclasses import dataclass, field
    3. # Example pricing in USD per million tokens. Always check the latest public platform pricing.
    4. PRICING = {
    5. "baseline": {"input": 1.40, "output": 4.40, "cached_input": 0.26},
    6. "new": {"input": 1.40, "output": 4.40, "cached_input": 0.26},
    7. }
    8. @dataclass
    9. class CallRecord:
    10. model: str
    11. prompt_tokens: int # Input tokens, including cached hits
    12. completion_tokens: int # Output tokens, including reasoning content
    13. cached_tokens: int = 0 # Cached input tokens among prompt tokens
    14. attempts: int = 1 # Total attempts, including the first call
    15. success: bool = True
    16. extra: dict = field(default_factory=dict)
    17. def record_cost(rec: CallRecord) -> float:
    18. p = PRICING[rec.model]
    19. fresh = max(rec.prompt_tokens - rec.cached_tokens, 0)
    20. return (
    21. fresh * p["input"]
    22. + rec.cached_tokens * p["cached_input"]
    23. + rec.completion_tokens * p["output"]
    24. ) / 1_000_000 * rec.attempts
    25. def summarize(records: list[CallRecord]) -> dict:
    26. total = sum(record_cost(r) for r in records)
    27. ok = [r for r in records if r.success]
    28. return {
    29. "task_count": len(records),
    30. "success_rate": len(ok) / len(records),
    31. "cost_per_task": total / len(records),
    32. "cost_per_success": total / len(ok) if ok else None,
    33. }
    34. # Example: compare the previous generation, a new generation without cache hits,
    35. # and a new generation with cache hits for the same short Q&A task.
    36. old = CallRecord("baseline", prompt_tokens=3000, completion_tokens=600)
    37. new = CallRecord("new", prompt_tokens=3000, completion_tokens=720) # Output increases by 20%
    38. new_cached = CallRecord(
    39. "new",
    40. prompt_tokens=3000,
    41. completion_tokens=720,
    42. cached_tokens=1440 # About 48% of input hits the cache
    43. )
    44. for name, r in [("old", old), ("new_cached", new_cached), ("new", new)]:
    45. print(name, r.cached_tokens, f"{record_cost(r):.6f}")
    46. print(summarize([old, new_cached, new]))

    Example output:

    1. old 0 0.006840
    2. new_cached 1440 0.005726
    3. new 0 0.007368
    4. {'task_count': 3, 'success_rate': 1.0, 'cost_per_task': 0.006645, 'cost_per_success': 0.006645}

    Three observations:

    • After output increases by 20%, the new generation without cache hits costs about 7.7% more than the previous generation.
    • If about 48% of the input hits the cache, the cached version is about 22.3% cheaper than the uncached new version.
    • In the same example set, the cached version is also about 16.3% cheaper than the previous generation.

    This shows that savings from cache hits can sometimes change the bill structure more than simply switching models.

    5. Caching: The Most Cost-Effective Lever

    The price of cached input is only about 19% of the regular input price, which is equivalent to an approximately 81% discount on the cached portion. This is one of the optimization items with the best return on effort.

    Savings can be estimated as follows:

    1. Savings = cached input tokens × (input price - cached input price)
    2. Cache hit rate = cached input tokens ÷ total input tokens

    For example, suppose each call has 6,000 input tokens and the entire prefix hits the cache:

    1. Savings = 6000 × (1.40 - 0.26) ÷ 10^6 ≈ USD 0.00684

    This is equal to 81% of the cost that would be charged for all input tokens at the regular input price without cache hits. In real engineering practice, not all input will hit the cache, so savings vary roughly linearly with the cache hit rate.

    There are four prerequisites for using caching well:

    1. Keep the prefix stable: Long and fixed content such as the system prompt, tool definitions, and few-shot examples should be placed at the front and kept byte-for-byte consistent. Timestamps, random IDs, and changing copy can invalidate the cache.
    2. Check the cache lifecycle: Cache lifetimes differ across platforms, and some platforms require a minimum token count. In very long conversations, keep the prefix unchanged and append new content when possible.
    3. Design for cache misses: Treat “miss” as the normal case when setting budget limits and fallback paths. This prevents cache-hit-rate fluctuations from directly hurting cost.
    4. Handle multi-tenancy and privacy: User data must be properly isolated and masked. Using caching does not mean relaxing data security. Storage and access policies for sensitive information should be designed separately.

    6. Architecture Practices: From Pay-as-You-Go Billing to Task-Level Governance

    6.1 Tiered Model Routing

    Not every task needs the strongest model. Routing by task complexity can reduce cost while keeping the success rate roughly unchanged.

    1. from enum import Enum
    2. class Tier(Enum):
    3. SMALL = "small" # Deterministic tasks such as classification, extraction, and format conversion
    4. STANDARD = "std" # Regular Q&A, summarization, and general code generation
    5. FRONTIER = "front" # Complex reasoning, multi-step agents, and security audits
    6. def route(task: dict) -> Tier:
    7. if task["kind"] in ("classify", "extract", "format"):
    8. return Tier.SMALL
    9. if task["steps"] > 5 or task["risk"] == "high":
    10. return Tier.FRONTIER
    11. return Tier.STANDARD

    Tiered routing should not be based on guesswork. It should be validated with data:

    1. Measure the success rate and cost per successful task for each tier on the same task set.
    2. Only move a task to a lower tier when the success-rate drop is acceptable and the cost per successful task is significantly lower.
    3. Put the routing rules themselves under version control so they do not silently break when models or business requirements change.

    A fallback strategy should also be designed. If a lower-tier model has a success rate below a threshold, or if its cost per successful task is higher than a higher tier’s, the request should automatically fall back to the FRONTIER tier.

    1. def route_with_fallback(task: dict, stats: dict) -> Tier:
    2. tier = route(task)
    3. if tier != Tier.FRONTIER:
    4. s = stats.get(tier.value, {})
    5. success_rate = s.get("success_rate", 0)
    6. cost_per_success = s.get("cost_per_success", float("inf"))
    7. if success_rate < 0.95 or cost_per_success > 0.08:
    8. return Tier.FRONTIER
    9. return tier

    6.2 Budget Guards and Fail Fast

    Before executing a task, set a token budget and cost ceiling. Requests that are obviously inefficient can be blocked early:

    1. def within_budget(task: dict,
    2. max_input_tokens: int = 200_000,
    3. max_cost_per_task: float = 0.05) -> bool:
    4. est = estimate_cost(task) # Estimate from input size and historical output distribution
    5. if est > max_cost_per_task or task["input_tokens"] > max_input_tokens:
    6. task["degraded"] = True # Use a lightweight model, compress context, or escalate to a human
    7. return False
    8. return True

    The value of budget guards is not only saving money. They also prevent abnormal tasks from dragging down the whole pipeline. For long documents, batch summarization, complex agents, and similar scenarios, capacity limits should be set in advance.

    6.3 Output and Chain Control

    • Structured output: Use JSON Schema to constrain the format and omit unnecessary explanation fields. This reduces output tokens and makes downstream validation more stable.
    • Context management: In multi-turn conversations, summarize historical content on a rolling basis and retrieve only relevant snippets when needed. Avoid sending the entire history repeatedly.
    • Long-running tasks: Set a maximum number of turns, a per-turn output limit, and an overall budget. Persist intermediate state so that a failed task can resume from the interruption point instead of starting over.

    6.4 Observability: Making Cost Attributable

    On the calling side, log the following fields and associate them with business identifiers:

    1. request_id, task_id, tenant_id, model, tier,
    2. prompt_tokens, completion_tokens, cached_tokens,
    3. attempts, latency_ms, success, error_code, cost_usd

    Example log:

    1. {
    2. "request_id": "req_20260622_0001",
    3. "task_id": "qa_10086",
    4. "tenant_id": "tenant_001",
    5. "model": "modelA-new",
    6. "tier": "std",
    7. "prompt_tokens": 3000,
    8. "completion_tokens": 720,
    9. "cached_tokens": 1440,
    10. "attempts": 1,
    11. "latency_ms": 2450,
    12. "success": true,
    13. "error_code": null,
    14. "cost_usd": 0.005726
    15. }

    In a logging service or monitoring and alerting system, it is recommended to aggregate by tenant, scenario, and model, and to set three types of alerts:

    • A sudden period-over-period increase in cost per task.
    • A drop in cache hit rate.
    • An increase in failure rate or retry rate.

    6.5 Unified Gateway: Avoiding Lock-In to a Single Model

    Introduce an adaptation layer between applications and model services. Move common logic such as prompts, tool definitions, and parameter validation down to the gateway. This has three benefits:

    • When a new model is launched, real tasks can first be replayed on shadow traffic to compare quality and cost before a gradual rollout.
    • When switching providers or deployment forms, business code does not need major changes.
    • Combined with private resource pools and hybrid deployment, sensitive data and high-frequency tasks can stay in your own environment.

    For example, Baidu AI Cloud provides a large-model platform that can be used to unify model access, online inference, model development, and private resource management. When combined with object storage, logging, and monitoring capabilities, teams can build a loop that covers onboarding, evaluation, launch, observation, and optimization.

    7. Risks and Governance: Cost Reduction Must Not Cross the Line

    • Quality baseline: Cost optimization must not come at the expense of accuracy or safety. Tasks related to security audits or compliance reviews should be fixed to high-capability models and retain human review steps.
    • Version locking: More capable models may be updated monthly. Locking a specific version helps ensure reproducible results, but teams should also regularly evaluate new versions to avoid staying on old versions for too long.
    • Quotas and rate limits: Input and output lengths, concurrency, and cached token counts usually have upper limits. Capacity planning should be done in advance for long documents and long sessions.
    • Portability: In parallel, verify whether open-weight models can replace online services. For large-scale usage, private deployment with fixed costs and online services billed by usage need a combined break-even analysis.
    • Comparison criteria: When comparing models, clearly state the task set, success criteria, whether cache hits are included, whether retries are included, the price date, and the version number. Anonymous or generalized comparisons can only serve as examples and should not be used directly as procurement conclusions.

    8. Conclusion

    The unit price of the model API has not gone up. The intelligence score has risen from 53 to 60, but output tokens may increase by about 20%. This tells us that “unit price” is only the starting point of the bill, not the endpoint. The real cost is determined by:

    • How many tokens each task needs.
    • How many of those tokens can hit the cache.
    • How many retries are required when failures occur.

    The next stage of competition is not only about “who is smarter,” but also about “who can complete tasks at a lower cost per successful task with the same level of intelligence.” For developers and enterprises, it is more important to build their own task-level evaluation sets and cost models than to chase leaderboard scores.

    There are four common mistakes:

    • Looking only at unit price, while ignoring cost per task and cost per successful task.
    • Ignoring hidden costs caused by retries, failures, timeouts, and truncations.
    • Keeping cache prefixes unstable, causing the cache hit rate to fluctuate.
    • Counting only successful samples and not spreading failure costs across the overall budget.

    Fixing these mistakes and continuously applying prefix caching, tiered routing, budget guards, and observability will help ensure that every unit of intelligence is spent where it matters most.