LLM Cost Calculation: Why Token Prices Drop but the Bill Doubles

1. Start with the conclusion: the same change can save money and increase spending

A common model-iteration narrative goes like this: output speed improves by about 30%, the output-side unit price is reduced by 20%, and the change looks like a clear price cut. But if you switch the metric from unit price to the total cost of completing a real task, the conclusion immediately reverses.

The table below uses representative benchmark tasks to illustrate the relationship. The numbers are only for showing orders of magnitude and do not represent any specific product.

Observation	Old version	New version	Change
Average output tokens per task	118K token	274K token	about +132%
Output-side unit price, relative	1.00	0.80	-20%
Total cost per task	12.99	26.50	about +104%

The key is to run a consistency check. If we consider only the output side, a 20% price cut combined with a 1.32× increase in output volume should raise output cost by about 86%. The actual total cost rises by 104%. The extra ~10% suggests that the input side also expanded: scaffold system prompts, tool definitions, historical trajectories, and returned reasoning content are all billed as input.

This leads to the first judgment in this article: a vendor price cut and a rising bill are not contradictory. The billing unit, per million tokens, and the value unit, a completed task, have decoupled in this generation of models. Unit price is a marketing-facing metric; total task cost is the engineering fact on the delivery side.

2. Get the math right: a cost model you can copy

Do not look only at the price list. A real bill should be written in the following form:

Total bill = Σ_tasks [ (effective input tokens × P_in + output tokens × P_out)
                × (1 + retry rate)
                × (1 - cache discount) ]
       + fixed overhead (vector retrieval, embeddings, orchestration, cold storage, and logs)

The following pseudocode can be used to generate comparable metrics on your own task set:

def task_cost(in_tok, out_tok, p_in, p_out, retry_rate=0.0, cache_hit=0.0):
    billable_in = in_tok * (1 - cache_hit)
    return (billable_in * p_in + out_tok * p_out) * (1 + retry_rate)
def unit_economics(tasks, pricing):
    total, ok = 0.0, 0
    for t in tasks:
        r = run(t)                      # Record actual usage instead of estimating
        total += task_cost(r.in_tok, r.out_tok,
                           pricing.p_in, pricing.p_out,
                           r.retry_rate, r.cache_hit)
        ok += 1 if r.accepted else 0    # Count a task as successful only after acceptance
    return {"avg_cost_per_task": total / len(tasks),
            "cost_per_successful_task": total / max(ok, 1),
            "success_rate": ok / len(tasks)}

The two metrics that must be viewed together are “average cost per task” and “cost per successful task.” When the success rate drops from 88% to 80%, the cost per effective output can deteriorate significantly even if a single call becomes cheaper. In real business, the latter often decides whether the budget can be maintained.

3. Why output volume becomes systematically longer: structural causes

This is not a personality problem of a specific model. It is the joint result of technical routes and business models.

1. Test-time scaling. Trading longer internal reasoning for higher accuracy is one of the most stable ways to improve capability. Chains of thought, self-checking, multi-branch exploration, and hypothesis refutation all show up as token consumption on the output side.

2. Intelligence and conciseness are structurally opposed in this generation of models. The capability vendors sell is “think more deeply,” while the billing unit is highly sensitive to “thinking more.” Higher capability and higher cost are driven by the same lever; you cannot take only one.

3. Incentive directions are aligned. Mainstream pricing is token-based, and the output side often has a higher unit price, so supplier revenue is positively correlated with output length. This is not a moral issue; it is a consequence of pricing design. Buyers must hedge with evaluation.

4. Agent scaffolds carry their own inflation. System prompts, tool schemas, retrieved snippets, multi-turn history, and trajectory replay make the input side grow too. The more automated a task becomes, the fatter the context becomes.

5. The mean trap of long-tail tasks. A doubling of average output volume is often contributed by a few extremely long tasks. Capacity planning based on averages severely underestimates tail overhead. Budgets should be set by quantiles, such as P50/P95/P99.

4. Three pitfalls in evaluation

• Pitfall 1: Look only at single-turn accuracy on leaderboards. Leaderboards are mostly single-turn, short-output, tool-free tasks, which differ greatly from business tasks with tool calls and long horizons. A/B testing with your own task set is necessary.

• Pitfall 2: Ignore retries and acceptance. First-pass success rate, second-pass success rate, and manual rework rate are the source of the (1 + retry rate) term in the cost function. If generation becomes cheaper but rework increases, the total bill can still worsen.

• Pitfall 3: Forget the asymmetry between input and output pricing and discount mechanisms. Prompt-prefix caching, batching, and context reuse can reduce cost enough to offset a lower unit price. Discount rules differ across providers, so you must test them instead of copying documentation.

5. Engineering cost governance checklist

(1) Build a reusable baseline. Fix 100 to 300 real-business tasks, fix sampling parameters such as temperature, and fix acceptance scripts. Rerun the baseline whenever a model or prompt changes. Without a baseline, any price comparison is an illusion.

(2) Use tiered routing. Let low-cost models handle extraction, classification, initial screening, and formatting. Send only final judgment and complex planning to high-capability models. A common approach is to let the cheaper model produce candidates and the more expensive model make the final decision, which can significantly reduce the share of high-cost calls.

(3) Add hard budget guards. Any automated loop must have token limits, time limits, step limits, and loop detection. When a limit is exceeded, the loop should stop and fall back:

def guarded_call(prompt, budget, validator, model):
    if budget.spent_tokens >= budget.cap_tokens:
        raise BudgetExceeded("Token budget exhausted")
    resp = model.generate(prompt,
                          max_output_tokens=budget.remaining_tokens(),
                          timeout=budget.deadline)
    if not validator(resp):
        resp = model.generate(repair_prompt(resp),
                              max_output_tokens=budget.repair_allowance)
    budget.commit(resp.usage)          # Commit accounting immediately instead of estimating later
    return resp

(4) Use three layers of caching. Prompt-prefix caching puts stable prefixes first and variables later; semantic caching covers frequent similar requests; result caching covers idempotent extraction and transformation tasks.

(5) Use cheap verifiers to suppress expensive generation. A large part of agent cost comes from invalid branches. When a deterministic verifier exists, use it as the sole judge, and discard illegal steps without feeding them into the context:

state = verifier.initial_goal()
for step in range(step_limit):
    if verifier.closed(state):
        break
    tactic = model.propose(state, context=verifier.recent(state, k=3))
    nxt = verifier.apply(state, tactic)   # The verifier decides whether the step is legal
    if nxt is None:
        continue                          # Drop the step and do not feed it back into history to avoid context bloat
    state = nxt

(6) Make cost observable and attributable. Attribute tokens and costs to four dimensions: session, feature, tenant, and version. Use a general logging service to carry call-chain details, monitor alerts on quantile changes in unit task cost, store complete traces in object storage for offline review, and run elastic runtime workloads on a container platform. Only when cost can be attributed to business units does optimization have a target.

(7) Hedge on the commercial side. Seek pricing terms based on tasks or successful outcomes, keep prompts and evaluation sets portable, and avoid deep binding between the evaluation system and a single supplier.

6. The viral “thousands of agents collaborate to complete formal proofs”: which layer matters

Such messages usually contain two mixed signals. They need to be separated.

The first layer is the essence of formalization. Translating proofs already written by humans into a formal language that can be checked step by step by a machine produces verifiability. It is an improvement in verification-engineering compression and regression protection, not the discovery of a new theorem. Its value is that assets such as mathematical libraries, protocol specifications, and compiler correctness proofs gain sustainable machine checking, and long-term maintenance costs drop significantly. Interpreting this as “mathematics has been conquered” is overextrapolation.

The second layer is the truly valuable signal: a large number of agents, without central orchestration, autonomously perform task decomposition, role claiming, conflict resolution, and retry after failure, and continue for dozens of hours. This means the system is moving from “a called tool” to “a self-organizing construction crew.” This is also the direction that most needs engineering investment: long-horizon memory, checkpoints and rollback, state consistency, cost accounting, and circuit breaking. Missing any one of them makes large-scale replication impossible.

At the same time, calculate its resource bill: thousands of concurrent execution units and dozens of hours of wall-clock time imply non-trivial token budgets, scheduling overhead, and failure rates. The reproducibility question is never “can it be done once,” but “is the cost per effective output acceptable.”

7. Replace the comparison standard

Here is a three-step path for teams:

1. Change metrics. Stop using price per million tokens for decisions. Use cost per successful task, cost per accepted change, and the number of end-to-end tasks that can be completed per unit of budget.

2. Change the process. Build your own task set, acceptance scripts, and canary comparisons. Turn model selection into a regression-based engineering experiment, not a one-off interpretation of benchmark news.

3. Change the guardrails. Make budget guards, tiered routing, caching, cheap verifiers, and cost attribution platform defaults, rather than scripts written by each business line.

Return to the opening judgment: per unit it became cheaper, but per task it became twice as expensive. Both statements are true. The difference is that the former is supplier pricing language, while the latter is your financial fact. Mastering total-cost accounting, building your own baseline, and encoding budget constraints into code will yield more long-term value than chasing the numbers from any product launch.