New Model, Same Price—Why Is Your LLM API Bill Higher?

    An API model upgrade can raise benchmark scores while keeping per-token pricing unchanged, yet your bill may still climb. This article explains why, from an engineering perspective: how to read public capability indices and turn score gains into business metrics, why output-token inflation and long-running agent tasks drive costs up, how cache discounts only set an upper bound on savings, and how to build a practical cost-accounting framework. It also includes an optimization checklist and code examples for tiered routing, context compression, structured output, and monitoring.

    Introduction: Better Capabilities, Same Price—So Why Is the Bill Rising?

    Model version upgrades in the industry are entering a new phase: the new version’s published benchmark capabilities clearly outperform the previous generation, yet the API unit price stays the same. For developers, this looks like a free capability upgrade—but the real bill doesn’t always move the way intuition suggests.

    Using one such version upgrade as a starting point, this article answers three questions from an engineering perspective:

    • How should you read public capability indices? Does a higher score mean higher business value?
    • How much can the “roughly 80% cache discount” actually save?
    • Why can the cost per task be lower than the previous generation, yet the actual spend per task end up higher?

    The ultimate goal is a practical cost-accounting framework and a set of optimization practices.

    1. How to Read Public Capability Indices

    A public capability index usually refers to a composite score that a third-party evaluation body derives by weighting many benchmark task categories. A higher score is direct evidence of the vendor’s iteration, but to translate it into an engineering decision, you need to break it down into three questions.

    1.1 What the Capability Index Means—and Where It Falls Short

    The greatest value of a public capability index is horizontal comparability: the same institution, the same time window, the same question set and weights, placing different models on the same measuring stick. But its limitations are also clear:

    • Scenario drift: Benchmark questions are mostly standardized knowledge Q&A, code-fragment completion, and logical reasoning. Whether they represent your business distribution depends on the overlap between the evaluation set and your business data.
    • Static data contamination: A model may have seen the public questions during training, inflating its score.
    • Answers only, not cost: The index measures whether the answer is “right,” not how many tokens were spent or how long it took.
    • Reasoning-chain compatibility: Some evaluations require the model to output a long reasoning process. Reasoning models have a natural advantage in sample efficiency, but those reasoning tokens are usually billed too and count toward output length.
    • Instruction-set differences: Evaluation bodies typically call models with a unified prompt. If a model is optimized for a specific official prompt system, third-party evaluations can only approximate its performance under other instruction sets.

    So a capability index is suitable for initial screening and trend judgment, but not as a direct selection criterion.

    1.2 Turning Score Gains into Business Metrics

    The sensible approach is to build your own evaluation system, recommended to include three categories of metrics:

    • Quality: task success rate, field-level accuracy, code test-case pass rate.
    • Cost: cost per task, cost per thousand tokens, cache hit rate.
    • Experience: time to first token, end-to-end latency, timeout rate.

    On top of this, you can define a “cost per unit of quality” for horizontal comparison:

    1. Cost per unit of quality = average cost per task ÷ task success rate

    If you define a composite score E = success rate × (1 − P95 latency / SLA latency ceiling), then the value metric is E ÷ average cost per task. This reflects both quality and the truly usable capability under latency constraints.

    1.3 Two Kinds of “Hidden Regressions” in Version Upgrades

    A stronger new version doesn’t mean every business scenario gets better. Watch out for these two regressions:

    • Output behavior changes: The new version often emits more tokens per task than the previous generation. Whether the output becomes more “verbose” directly affects the bill.
    • Call-convention incompatibility: New capabilities like thinking modes and context caching introduce new parameters. When the upstream SDK isn’t upgraded, these may be silently ignored—or trigger an outright error. Before launch, do a full-chain check of the request parameters.
    1. # Example: new-version capabilities depend on new parameters; verify the client passes them through before launch
    2. payload = {
    3. "model": MODEL_NAME,
    4. "messages": build_messages(task),
    5. "temperature": 0.2,
    6. "max_tokens": 2048,
    7. # The following capabilities must be forwarded by the new client version—verify each one before upgrading:
    8. "thinking": {"type": "disabled"}, # Disable extended thinking when you need precise control over output length
    9. "cache_control": {"type": "ephemeral"}, # Enable caching for stable context prefixes
    10. }

    2. Rethinking the Billing Unit of LLM APIs

    The mainstream pay-as-you-go model usually splits tokens into three categories: input, cached input, and output. The key is understanding the billing meaning and amplification effect of each.

    2.1 The “Unit-Price Amplification Effect” of Output Tokens

    Since inputs are mostly given values, cost optimization almost always lands on the output side. The reason is the serial dependency between output and input:

    • Serial generation: Autoregressive decoding generates tokens one at a time, and each new token depends on all prior tokens. Output can’t be processed in parallel batches the way input can, so the compute cost per unit of output is inherently higher.
    • Result dependency: The output is the final answer, so any redundancy is billed as-is.
    • Iteration amplification: In agent tasks, the model’s output is written back into the context. A round of redundant output reappears in the input of every subsequent round and is billed again, snowballing the cost.

    So when evaluating a new model, the tokens consumed per task matter more than the token unit price—especially the output tokens consumed.

    2.2 How Output Inflation Affects “Equivalent Capability”

    Take two models, A and B, that both score 60 on the AA intelligence index. General benchmark data shows that B’s cost per task is about 19% lower than A’s; but when B completes similar tasks, it outputs about 20% more tokens than its previous generation. Put together, these two data points form a classic trap.

    The key is to distinguish “same-generation horizontal comparison” from “cross-generation vertical comparison”:

    • Horizontal: Among models that both score in the 60 range, A and B—B is cheaper.
    • Vertical: Compared with the previous generation of B (53 points), the new version of B is a net capability gain; but if both cost the same, the extra output tokens the new version consumes turn into a higher cost per task.

    So “same unit price” does not mean “same cost.” To judge whether an upgrade is worthwhile, you must measure it against your own business samples:

    1. Upgrade gain ≈ (success-rate gain × business value per task) − (output-token increase × output unit price × task volume)

    3. The Real Value of Cache Discounts: Salvageability Is the Ceiling

    “Roughly 80% off on cache hits” sounds tempting, but the nominal discount and the actual gain are not the same thing. Let the discount rate be d (in this case, the hit price is 19% of the input price, so d ≈ 0.81) and the output unit price be P_out. Then the upper bound of cache savings is:

    1. Savings ratio = cacheable input share × d × (P_in / P_out)

    Under the assumption of input 1.40 and output 4.40, P_in/P_out is about 0.318. Even if all input hits the cache, the savings are under 26%. The magnitude of savings depends mainly on the ratio between input share and output share: the lower the input share and the higher the output share, the less a cache can do.

    This metric (ignoring cache-miss and cache-write overhead):

    1. Cache coverage = cache-hit input tokens / total input tokens

    Implementation needs attention on four points:

    1. Stable prefixes: Caches generally match by token prefix. Put the system prompt, tool definitions, and few-shot examples first and keep them stable.
    2. Variable placement: Put dynamic content like timestamps, request IDs, and session state at the end, to avoid contaminating the prefix.
    3. Cache invalidation: The cache may be invalidated wholesale when the model version switches, the context structure changes, or the retention window expires.
    4. Hit-rate decay: In multi-turn conversations, the hit rate is usually near 100% after the first turn; in single-turn long-document tasks, repeated document sections can also hit. So monitor by segment.
    1. # Example: put stable content first and dynamic content last to improve prefix-cache hit rate
    2. system_prompt = STATIC_SYSTEM_PROMPT # Fixed and unchanging
    3. tools_schema = json.dumps(TOOL_DEFINITIONS, sort_keys=True) # Keep key order fixed when the definition is stable
    4. few_shot = STATIC_EXAMPLES # Fixed and unchanging
    5. messages = [{"role": "system", "content": system_prompt},
    6. {"role": "system", "content": tools_schema},
    7. {"role": "system", "content": few_shot}]
    8. # Dynamic content (user input, retrieval results, timestamps) is appended at the end
    9. messages.append({"role": "user", "content": dynamic_context})

    4. An Underestimated Bottleneck: The Cost Structure of Long-Running Agent Tasks

    The cost of a single-turn call is simple (input + output), but the cost model of a long-running agent task is far more complex:

    1. Total task cost = Σ(per-turn input tokens × input unit price × (1 − cache discount this turn)
    2. + per-turn output tokens × output unit price)

    Let the stable prefix be P tokens per turn, the newly added tokens be Δ per turn, and the total number of turns be N, with a cache discount rate of d. Then the total input tokens for the task are approximately:

    1. T_in ≈ N × P + Δ × N × (N + 1) / 2

    Task cost grows quadratically with the number of turns. If a single-turn task also involves R tool calls and failed retries, the number of turns often inflates further. This leads to three direct conclusions:

    • The longer the stable context prefix and the more turns there are, the more significant the benefit a cache brings.
    • Suppressing the growth of newly added context per turn (Δ) is just as important as compressing the prefix (P).
    • Whether a single quote goes up or down matters less than accounting for the cost per task.
    1. from statistics import mean
    2. def simulate_agent_task(turns, prefix, delta, p_in, p_out, discount=0.0):
    3. """Example: cost accounting for a long-running agent task (excludes tool calls and retries)"""
    4. cost = 0.0
    5. context = prefix # Stable prefix, eligible for cache hits
    6. for step in range(turns):
    7. cached = min(step, turns) # Simplified: the prefix is fully cacheable after the first turn
    8. fresh = context - cached * prefix
    9. cost += (fresh * p_in * (1 - discount) + cached * prefix * p_in * (1 - discount))
    10. # Model output: thinking + tool calls + reply, estimated per step
    11. output = 600
    12. cost += output * p_out
    13. context += delta + output # Next-turn input = prefix + cumulative additions this turn
    14. return cost

    On this basis, three layers of governance are recommended to control cost:

    1. Tiered routing: Route high-frequency light tasks like intent recognition and field extraction to a low-cost tier, and complex reasoning and code generation to a high tier. Routing rules can start with rules, then use a small model for classification.
    2. Context compression: Summarize and structure historical turns, keeping only the state relevant to the current subtask.
    3. Budget and circuit breaking: Set a token limit, a maximum number of turns, and a timeout for each task; downgrade or escalate to a human when a threshold is triggered.
    1. # Example: tier models by task complexity
    2. def route_model(task):
    3. if task.estimated_tokens < 800 and task.risk_level == "low":
    4. return CHEAP_MODEL # Lightweight model: format conversion, information extraction
    5. if task.need_tool_chain and task.steps > 8:
    6. return AGENT_MODEL # Tier with stronger agent capability: long-running multi-step tasks
    7. return STANDARD_MODEL # Default tier

    5. Engineering Optimization Checklist: Turning Unit-Price Advantages into Total-Cost Advantages

    Optimization Approach Main benefit
    Structured output Constrain with JSON Schema, keep field order stable Reduces explanatory redundancy, output is parseable
    Output length control Set explicit length limits; disable unnecessary deep thinking Curb output inflation, lower latency
    Batch mode Send offline tasks through the batch async API Noticeable discounts in some scenarios
    Context compression Summarize history, trim repeated segments Reduce Δ and P, ease quadratic growth
    Tiered routing Send light tasks to light tiers, hard tasks to strong tiers Control average cost per task
    Cache prefix optimization Stable content first, dynamic content last Improve hit rate
    Monitoring and budget Monitor usage by token type, scenario, model Cost is attributable and can be alerted

    Structured-output example:

    1. import json
    2. TASK_SCHEMA = {
    3. "type": "object",
    4. "required": ["action", "summary", "risk"],
    5. "properties": {
    6. "action": {"type": "string", "enum": ["allow", "block", "manual_review"]},
    7. "summary": {"type": "string", "maxLength": 200},
    8. "risk": {"type": "string", "enum": ["low", "medium", "high"]},
    9. },
    10. }
    11. # Prompt: output JSON only, no explanations, to suppress redundant output
    12. prompt = (
    13. "Determine whether the log excerpt below contains suspicious behavior; output JSON only:" + json.dumps(SCHEMA_HINT) +
    14. "\nLog:" + log_excerpt
    15. )

    For monitoring, it’s recommended to bucket usage metrics by “model version / cache hit or miss / token type / business scenario”:

    1. [metrics]
    2. model=a-v2, cache=hit, type=input -> billed input (cached price)
    3. model=a-v2, cache=miss, type=input -> billed input (standard price)
    4. model=a-v2, type=output -> output tokens (including thinking content)

    At the same time, set up three types of alerts: the mean output tokens exceeding a threshold relative to the baseline, a drop in cache hit rate, and cost per task exceeding budget.

    6. Usage Boundaries in Security Scenarios

    The new model’s enhanced capabilities in programming and security should first be applied to defensive work: code auditing, dependency vulnerability identification, security log aggregation, configuration baseline checks, and so on. This kind of work amplifies the team’s capabilities, but keep in mind:

    • Human review is still needed: Model output should be treated as a lead, not directly as a conclusion or as a basis for action.
    • Security evaluations have side effects: Some security evaluations ask the model to perform code rewriting under a specific objective and set a strict token budget. This kind of evaluation naturally favors models with long reasoning times and verbose output. A high score may come from being “more willing to try” rather than being “more correct,” so judge it in combination with your business false-positive and false-negative rates.
    • Abuse control: Users should follow the service provider’s usage policies and content-safety requirements. When sensitive samples are involved, follow data-minimization and de-identification principles; where necessary, adopt private or dedicated deployment, and retain traceable records through log auditing.

    Conclusion: From “Comparing Capability” to “Comparing Cost per Unit”

    When multiple models stand on the same tier of public capability indices (e.g., AA 60 points), the competitive focus shifts from “who is smarter” to “who can push cost lower at the same intelligence level.” For engineering teams, this means three things:

    1. Build your own evaluation and cost-accounting system: Measure models by cost per unit of quality, not just by token unit price.
    2. Treat caching, structured output, tiered routing, and output-length control as infrastructure—not one-off optimizations.
    3. Keep running regression tests: Re-account on your own business samples every time you upgrade.

    On large-model service platforms such as Baidu AI Cloud Qianfan, developers can access multiple vendors’ models through one unified interface, compare cost per task and business quality under the same interface, and continuously track cost changes with general capabilities like monitoring alerts and log analysis. At the same capability tier, what usually determines the size of the bill is not the unit price, but the engineering implementation.