Why LLM Token Prices Fell 20% but Per-Task Costs Doubled: The Reasoning Token Tax

1. Unit Price Is Not Your Bill

When a model generation is upgraded, the most visible number the vendor publishes is the input and output price per million tokens. A 20% cut looks like an unambiguous win, yet many teams find their monthly bill rising after they ship the new model. The reason is not mysterious: unit price is a rate, and the bill is the result of rate × usage.

The identity can be written in one line:

per-task cost = Σ(token usage for each category × the corresponding unit price)

Reasoning models have one particular wrinkle. Before giving an answer, the model generates an internal reasoning process, and those tokens are typically billed as output. Much of the capability gain from a new generation comes from “letting the model think longer,” which creates a familiar pattern: output-side usage grows by a larger factor than output-side price falls, so total cost goes up.

Cost reduction has only three levers: lower the unit price, lower token usage, or raise the success rate of a single task (fewer reruns and retries). Developers do not control unit price, so the engineering work comes down to the other two.

2. Getting the Per-Task Cost Right: A Reusable Cost Model

Move cost calculation from reading the pricing table to an explicit function in your code, so every call can be attributed to a cost.

from dataclasses import dataclass
@dataclass
class TokenPricing:
    price_in: float    # Price per million input tokens (example values, for demonstration only)
    price_out: float   # Price per million output tokens; reasoning tokens are counted on the output side
def task_cost(tokens_in: int, tokens_out: int, p: TokenPricing) -> float:
    """Cost of a single task. tokens_out must include reasoning tokens."""
    return (tokens_in * p.price_in + tokens_out * p.price_out) / 1_000_000
def batch_cost(tasks: list[tuple[int, int]], p: TokenPricing) -> dict:
    per_task = [task_cost(i, o, p) for i, o in tasks]
    n = max(len(per_task), 1)
    return {
        "avg_cost": sum(per_task) / n,
        "p95_cost": sorted(per_task)[int(n * 0.95) - 1] if n > 1 else per_task[0],
        "total": sum(per_task),
    }

Two details are easy to miss. First, reasoning tokens must be counted in the output-side usage. Many teams’ instrumentation records only the length of the final answer, which systematically underestimates cost. Second, the average alone is not enough. Cost distributions are typically long-tailed, and the P95 per-task cost often determines the budget ceiling.

The following normalized example shows how a price cut can end up raising the bill. The numbers are for illustration only, with the previous generation’s top reasoning tier set to 100:

Configuration	Output price index	Average output tokens index	Per-task cost index
Previous generation · top reasoning tier	100	100	100
New generation · top reasoning tier	80	265	≈212
New generation · medium reasoning tier	80	115	≈92

The conclusion is clear. The 20% price cut is completely swallowed by 2.65× output growth, but dropping the reasoning tier from top to medium brings cost back below the previous generation immediately. The cost gap between tiers of the same model is often larger than the price gap across generations, which makes tiering the main driver of the bill.

3. Where Reasoning Token Bloat Comes From

• It scales with task difficulty. On simple tasks, long reasoning yields almost no benefit and is pure waste. On complex tasks it pays off, but with diminishing marginal returns.
• Context is billed over and over. In multi-turn conversations, the history is re-billed as input on every turn. Long document injection and tool results fed back into the context work the same way.
• Agent workflows amplify it. A single user request can trigger multiple “think → call a tool → observe the result” loops, so the cost of one request is multiplied by the number of loops.
• Failure retries cost you quietly. If a tier that is too low fails validation, the task escalates to a higher tier and is retried. The total cost can end up higher than if you had picked the high tier from the start.

This is why the measurement unit must shift from “cost per request” to “the cost of completing one task as the user perceives it.”

4. Tiered Usage: Treat the Reasoning Budget as a Tunable Resource

Most services expose reasoning effort as a set of tiers. A sensible default is to assign tiers by task type rather than maxing out everything.

Task type	Recommended tier	Rationale
Intent recognition, classification, field extraction, format conversion	Lowest	Input and output boundaries are clear, so long reasoning adds nothing
Single-file code completion, routine copy rewriting	Low to medium	Local reasoning is enough
Cross-file refactoring, complex bug localization, multi-step logical reasoning	Medium to high	Benefit is strongly tied to reasoning length
Open-ended analysis, long-horizon planning	High, with a hard budget cap	High benefit but high long-tail risk

Three engineering measures support this approach:

1. Explicit budget constraints. Set a maximum reasoning token count and a timeout for each call, so individual tasks cannot consume the entire month’s budget.
2. Difficulty routing (cascade calls). Start with a low-cost configuration, use deterministic validation (compilation, unit tests, schema validation, rule checks) to decide whether the result is acceptable, and escalate only if it is not. Basing the decision on validation results rather than subjective guesses is the key to whether this approach actually saves money.
3. Prioritize high-frequency tasks. Assign tiers to the Top-N task types by call volume, one by one. The payoff is far larger than tuning settings uniformly across the whole system.

5. Vendor-Agnostic Cost Reduction

• Context governance. Replace full-document injection with retrieval-augmented generation (RAG). Deduplicate, trim, and compress conversation history into structured summaries. Keep only the fragments a task needs.
• Output governance. Explicitly constrain output length, fields, and format, and turn off irrelevant explanatory content. Output tokens are usually the most expensive part.
• Cache reuse. If your service offers prompt prefix caching, put stable system prompts and tool definitions at the front of the prompt. This can significantly reduce repeated input charges. On top of that, add an idempotent result cache on the application side to cover high-frequency repeated requests.
• Invocation pattern optimization. Send batchable asynchronous jobs through a batch channel. Schedule concurrency so tasks do not sit in a queue and time out, which leads to wasted retries. Use retries with exponential backoff and a retry limit.

6. You Cannot Optimize What You Cannot Measure

The foundation of cost governance is persisting structured usage data for every call. Example fields:

{
  "trace_id": "8f2c...",
  "task_type": "code_review",
  "model_tier": "high",
  "tokens_in": 18240,
  "tokens_out": 3120,
  "reasoning_tokens": 2450,
  "cache_hit_ratio": 0.37,
  "latency_ms": 9820,
  "success": true,
  "retry_count": 1
}

Five key metrics are worth standardizing:

• Per-task cost. Track both the average and P95.
• Effective task cost = per-task cost ÷ task success rate. If retries and reruns eat up the savings from a lower tier, this metric will expose it immediately.
• Cost per 1,000 requests. This makes it easier to align cost with business revenue figures.
• Reasoning token ratio = reasoning tokens ÷ output tokens. This is a direct signal that a tier is set too high.
• Marginal cost = Δ per-task cost ÷ Δ eval score. This is the core criterion when choosing a model.

On Baidu AI Cloud, this system can be built by combining general-purpose capabilities. The application emits structured usage logs, which a log service stores in detail and makes searchable for analysis. Monitoring and alerting send real-time warnings when daily cost, P95 cost, or success rate crosses a threshold. Long-term data is archived to object storage for cost-quality regression analysis.

7. Selection Criteria: Convert Performance Gains into Cost per Unit of Performance

When you evaluate a new-generation model, look beyond benchmark scores and unit price, and follow these four steps:

1. Calculate the relative gain. Divide the percentage performance improvement by the percentage cost increase. The result is the cost increase you pay for every 1% of performance gain.
2. Look at the task distribution. If the upgrade only helps low-frequency complex tasks while a high tier inflates the bill for high-frequency ones, the overall value proposition can be negative.
3. Check the quality floor. Compliance review, financial risk control, and safety-related scenarios tolerate very low error rates. Letting cost give way to reliability is a reasonable choice there.
4. Run an A/B comparison. Use a fixed evaluation set and fixed business samples to run three groups (low, medium, and high tiers). Plot the cost-quality curve and pick the knee of the curve, not its highest point.

8. Conclusion Checklist

• Unit price is a rate, not a bill. Evaluate per-task cost and effective task cost.
• Reasoning tokens are billed as output and are the main source of hidden cost increases, so instrumentation must track them separately.
• The cost gap between reasoning tiers of the same model is often larger than the price gap across generations, which makes tiering the strongest lever.
• Replace manually chosen tiers with difficulty routing driven by deterministic validation.
• Set up usage instrumentation and cost alerts. Make cost observable first, then optimize.
• Use marginal cost as the selection criterion, and configure tiers differently by task frequency and quality tolerance instead of maxing out everything.