Stronger Models, Higher Token Costs: A Guide to LLM Cost Governance

I. A Common Industry Challenge: Capability vs. Consumption

Recently, a leading LLM provider faced a representative backlash following a new model release. While the model achieved top rankings on third-party evaluation benchmarks, users heavily complained that their subscription quotas were draining too quickly—hitting usage limits shortly after starting tasks. The vendor subsequently apologized, issued partial quota restorations for subscribers, and promised to reduce unnecessary sub-agent invocations and redundant outputs.

Beyond this specific incident, the event highlights a broader industry tension: model capabilities and token consumption are expanding simultaneously, while user quota expectations remain anchored to previous-generation products. For power users who integrate LLMs deeply into their coding and writing workflows as core productivity tools, this shift in consumption is palpable. A budget that previously handled ten tasks might now only cover five. Understanding the technical causes behind this perceived “shrinkage” is the first step toward effective cost governance.

II. Where Do Tokens Go? A Four-Layer Breakdown of Consumption

Most users intuitively gauge consumption based on visible text input. However, a real API call incurs far more token expenditure, which can be categorized into four layers:

1. Explicit Consumption: User input and model output. This is the only “visible” portion, but it typically accounts for less than half of the total tokens consumed.

2. Implicit Consumption: System prompts, tool and function definitions, and few-shot examples. In an Agent architecture, these fixed prefixes are repeatedly billed in every conversation turn and every sub-invocation. An agent registered with a dozen tools might consume thousands of tokens just for tool descriptions. When multiplied by the number of calls, this becomes substantial.

3. Amplified Consumption: Sub-agent invocation chains. A seemingly simple task might involve a planning agent decomposing the task, multiple execution agents processing it in parallel, and a summarizing agent integrating the results. Each hop requires carrying context, causing consumption to grow multiplicatively. In extreme cases, a few hundred words of visible output to the user might correspond to tens of thousands of tokens in internal routing.

4. Redundant Consumption: Overly long chains of thought, repetitive outputs, failed retries, and feeding entire original files back to the model rather than passing only relevant snippets. This type of consumption yields almost no value but acts as a primary “quota black hole.”

Consider an illustrative estimate: A user inputs 200 tokens and expects 500 tokens in output. Without an agent chain, a single call consumes about 700 tokens. However, if the task goes through 3 sub-agent calls, with each turn carrying roughly 2,000 tokens of context and tool definitions, the total consumption easily exceeds 8,000 tokens—an amplification of more than ten times.

III. Why Lower Unit Prices Don’t Mean Lower Total Costs

Understanding this consumption structure explains a counterintuitive phenomenon: API unit prices drop, yet users feel the service has become more expensive.

For pay-as-you-go API users, Cost = Unit Price × Consumption Volume. If the unit price drops by one-third, but the new model consumes twice as many tokens to complete the same task, the total cost actually increases by about one-third. Unit price is only one variable; consumption volume is the other. They must be evaluated together.

For subscription users, their quota is essentially a token-denominated budget. Model capability upgrades drive up per-task token consumption, meaning the number of tasks achievable within the same quota decreases—this is the mathematical reality behind “quotas draining faster.” A representative conclusion from heavy users perfectly captures this: Although the unit price is lower, the tokens required to complete a single task have increased, meaning the overall financial math offers little to no advantage.

This leads to a core selection principle: Evaluating model cost-effectiveness must occur at the “task” granularity, not the “token” granularity. The correct estimation formula is:

Unit Task Cost = Input Price × Task Input Tokens + Output Price × Task Output Tokens

Meaningful price comparisons can only be made after measuring actual token consumption on benchmark tasks.

IV. Quantification First: Installing a Dashboard for Token Consumption

The prerequisite for governing costs is visualizing them. It is recommended to establish unified instrumentation for every model invocation to track the following metrics:

• Tokens per task: Aggregated by business task, rather than per single API call.
• Input/Output ratio: Determines whether consumption is driven primarily by long contexts or long outputs.
• Sub-agent invocations: Directly correlates with the consumption amplification factor.
• Cache hit rate: Measures the effectiveness of prefix reuse.
• Failure retry rate: Retries are hidden consumption amplifiers.

Illustrative code:

# Task-level token instrumentation (illustrative)
def call_llm(messages, model="standard", max_tokens=1024):
    resp = llm_client.chat(model=model, messages=messages,
                           max_tokens=max_tokens)
    usage = resp.usage  # Common fields: prompt_tokens / completion_tokens
    metrics.record({
        "task_id": current_task_id,
        "model": model,
        "prompt_tokens": usage.prompt_tokens,
        "completion_tokens": usage.completion_tokens,
        "agent_depth": current_agent_depth,
    })
    return resp
# Unit task cost estimation (price unit: currency / million tokens)
def unit_task_cost(task_calls, price_in, price_out):
    tin = sum(c["prompt_tokens"] for c in task_calls)
    tout = sum(c["completion_tokens"] for c in task_calls)
    return tin / 1e6 * price_in + tout / 1e6 * price_out

Based on this, establish a “consumption baseline”: Before and after every model or version upgrade, use the same set of benchmark tasks to compare consumption changes. Many “upgrades make it more expensive” issues become immediately obvious under baseline comparison. Mainstream LLM platforms (such as Baidu AI Cloud, etc.) generally provide usage statistics and logging capabilities, which can be used to build team-level cost dashboards and alerts.

V. Six Engineering Optimization Strategies

1. Context Governance: Compress multi-turn conversation history using summarization and truncate early messages with a sliding window. Inject knowledge base retrieval results on demand, carrying only snippets relevant to the current query rather than entire documents.

2. Agent Invocation Convergence: Avoid splitting into multiple agents if a task can be completed in a single turn. Provide configurable depth levels for agents (e.g., “Fast / Standard / Deep”) to set user expectations regarding consumption. Define early termination conditions to stop iterating once the task meets the standard.

3. Output Constraints: Limit output length using max_tokens and reduce redundant phrasing with structured outputs (like JSON Schema). Explicitly state “output only the conclusion, not the reasoning process” in prompts. If necessary, route the chain of thought into a separate channel and meter it independently.

4. Cache Reuse: Persist stable system prompts and tool definitions as cacheable prefixes. Cached segments are typically billed at lower rates, offering significant benefits for agent-heavy scenarios.

5. Model Tiered Routing: Deploy a lightweight classifier upfront to assess task complexity. Route formatting or short summaries to lightweight models, invoking flagship models only for complex reasoning and long-horizon tasks. In practice, this can cut a substantial proportion of costs.

6. Failure Control: Set limits and backoff strategies for retries, and implement timeout circuit breakers. When retrying, carry only the minimal error-related context to avoid re-billing for the entire history.

VI. From “Comparing Unit Prices” to “Calculating Tasks”

For model selection and procurement decisions, replace simple “price sheet comparisons” with a three-step method:

Step 1: Build a benchmark task set covering core scenarios (code generation, long-text summarization, multi-turn QA, etc.) and track actual consumption on each candidate model.
Step 2: Estimate expenditure using “Unit Task Cost × Expected Monthly Task Volume,” then cross-validate this against subscription quota conversions.
Step 3: Incorporate consumption strategy transparency into the evaluation criteria. Whether sub-agent invocation behaviors are explicitly stated, whether call-level usage details are provided, and whether fallback mechanisms exist for anomalous consumption—these factors determine long-term cost controllability.

VII. Conclusion: Treat Cost Governance as Part of Agent Engineering

“Stronger models, less durable quotas” is not an issue isolated to a single vendor; it is the inevitable tension arising from simultaneous capability upgrades and consumption expansion. Vendors need more conservative default parameters and more transparent consumption strategies. On the developer side, four key practices—instrumentation, baselining, routing, and caching—dictate whether an application loses money as it scales or becomes more efficient over time. When tokens are managed as a first-class citizen in engineering, “powerful capabilities” and “affordability” can absolutely coexist.