Token Consumption in LLMs: Why Model Upgrades Inadvertently Increase Compute Costs

An analysis of excessive token consumption and compute optimization strategies for large language models—from underlying mechanisms to engineering practices.

1. The Contradiction Between Improved Model Capabilities and Accelerated Token Consumption

Recently, a well-known large language model sparked widespread discussion shortly after releasing a new-generation version. Users reported that while the new version showed significant improvements in code generation and long-horizon task handling, their subscription quotas were being consumed at an abnormally fast rate. Some users found their tasks approaching usage limits shortly after starting, and some argued that at the same price point, the model’s usable duration was far below the industry average. This is not an isolated case: as large models evolve from single-turn chat tools into complex productivity platforms, the contradiction of “the stronger the model, the faster the quota burns” is becoming increasingly prominent.

From a technical perspective, this contradiction stems mainly from three factors. First, to improve complex task handling, new-generation models often introduce more sub-agent call chains, multiplying the actual computation per task. Second, although longer text output improves generation quality, output tokens are typically billed far higher than input tokens. Third, users’ rising expectations of model capabilities lead them to embed models into more complex workflows, indirectly increasing call frequency.

2. Underlying Mechanisms: Why Model Upgrades Worsen Token Consumption

2.1 Hidden Costs of Sub-Agent Call Chains

Modern large models typically adopt multi-agent collaboration architectures to handle complex tasks such as 3D modeling and long-horizon code generation. Consider a typical code generation task:

# Pseudocode example: multi-agent collaboration for complex tasks
def execute_complex_task(user_query):
    # Main agent parses the task
    task_plan = main_agent.plan(user_query)
    # Invoke code generation agent
    code_draft = code_agent.generate(task_plan)
    # Invoke test agent for validation
    test_result = test_agent.validate(code_draft)
    # Invoke optimization agent for refinement
    final_code = optimize_agent.refine(code_draft, test_result)
    return final_code

On the surface, this is a single user request, but internally the system triggers at least four complete model inferences. Each call involves the full pipeline of input encoding, attention computation, and output generation, and passing intermediate results accumulates substantial context tokens. Without effective call-chain optimization, this “implicit computation” can easily drain quotas rapidly.

2.2 The Dual Cost Trap of Long-Text Output

The industry commonly uses separate billing for input and output. For example, one model API charges about 13 yuan per million input tokens and 68 yuan per million output tokens. When models increase output length to improve long-horizon task handling, the cost structure shifts significantly:

Assumed task scenario:
- Input context: 8K tokens (encoded user instructions + history)
- Output generation: 12K tokens (detailed code + explanation)
Cost calculation:
Input cost = 8 * 13 / 1000 = 0.104 yuan
Output cost = 12 * 68 / 1000 = 0.816 yuan
Total cost = 0.92 yuan
Compared with a simple task:
- Input: 2K tokens, output: 1K tokens
Total cost = 2*13/1000 + 1*68/1000 = 0.094 yuan

A single complex task can cost more than 10 times that of a simple task. If users are unaware of this difference, they can easily exhaust their subscription quota in a short time.

2.3 Context Windows and the Quadratic Complexity of Attention

When large models process long texts, the self-attention mechanism of the Transformer architecture scales quadratically with sequence length (O(n²)). This means processing a 32K context is not twice the computation of 16K, but four times. Some vendors limit context window size to control costs, but new-generation models often relax this limit to improve long-horizon capabilities, indirectly driving up compute consumption per inference.

3. Engineering Practices: System-Level Compute Optimization Strategies

To address these issues, developers and enterprise users need systematic optimization strategies that control costs while preserving model capabilities.

3.1 Call-Chain Optimization: Eliminating Unnecessary Sub-Agent Calls

Strategy 1: Merge homogeneous calls
Consolidate agents with overlapping functions to avoid redundant inference. For example, merge the code generation and optimization agents into a single “code generation and optimization agent,” using prompt engineering to complete generation and optimization within a single inference.

Strategy 2: Implement call circuit-breaker mechanisms
Set an upper limit on sub-agent calls per task. When the threshold is reached, force return of the current best result rather than continuing to iterate. This can be configured flexibly via a configuration file:

# Example task execution configuration
task_execution:
  max_subagent_calls: 5  # Maximum sub-agent calls
  max_output_tokens: 4096  # Per-call output token limit
  enable_early_stop: true  # Enable early termination
  early_stop_threshold: 0.85  # Result quality threshold

3.2 Token Consumption Optimization: Full-Pipeline Control from Input to Output

Input-side optimization: context compression and retrieval augmentation
Use techniques such as sliding windows and summary extraction to compress historical context. For example, summarize conversations exceeding a certain length first, retain key information points, then concatenate with the current user input as model input. Combined with retrieval-augmented generation (RAG), inject only background knowledge relevant to the current task rather than passing the full context.

Output-side optimization: structured output constraints
Constrain model output format through structured prompts to reduce redundant explanations. For example, require the model to return results in JSON format, avoiding repetitive natural-language preamble:

{
  "task_status": "completed",
  "result": {
    "code": "def example(): pass",
    "explanation": "Core logic explanation"
  },
  "next_steps": ["Run tests", "Merge to main branch"]
}

3.3 Architecture-Level Optimization: Caching and Asynchronous Processing

Semantic caching
Cache outputs for similar requests. When the semantic similarity between a new request and a cached request exceeds a threshold, return the cached result directly to avoid redundant inference. An efficient vectorized retrieval mechanism is needed to ensure cache hit rates.

Asynchronous task queues
Place complex tasks without real-time requirements into asynchronous queues and process them using idle model inference resources. This improves throughput and avoids extra overhead from resource contention during peak hours. Elastic scheduling can be implemented based on message queues and container platforms.

4. Enterprise-Level Cost Control: A Closed-Loop System from Monitoring to Alerting

4.1 Build a Multi-Dimensional Token Consumption Monitoring System

Developers need fine-grained token consumption monitoring that tracks not only totals but also consumption distribution by task type, call chain, and user behavior:

Example monitoring dimensions:
- Task type: code generation vs. text analysis
- Call chain: main agent consumption vs. sub-agent consumption
- Time: weekday peak hours vs. idle hours
- User: heavy users vs. light users

By integrating container platforms with logging services, real-time monitoring and historical retrospective analysis become possible, enabling rapid identification of abnormal consumption sources.

4.2 Implement Dynamic Quota Alerting and Rate Limiting

Build dynamic alerting mechanisms based on monitoring data. When a user’s or task’s token consumption rate grows abnormally, the system automatically triggers an alert and can optionally enforce rate limiting:

# Pseudocode example: dynamic quota alerting
class TokenUsageMonitor:
    def __init__(self):
        self.usage_history = {}
        self.alert_threshold = 1.5  # Abnormal consumption rate threshold
    def check_usage(self, user_id, current_usage):
        avg_rate = self.calculate_average_rate(user_id)
        current_rate = current_usage / time_elapsed
        if current_rate > avg_rate * self.alert_threshold:
            self.trigger_alert(user_id, current_rate, avg_rate)
            if current_rate > avg_rate * 2.0:
                self.apply_rate_limit(user_id, limit_type='soft')

4.3 Model Selection and Hybrid Scheduling Strategies

Enterprises should implement tiered scheduling based on task complexity to avoid over-provisioning:

• Simple tasks (e.g., short text summarization, simple Q&A): route to lightweight or small-parameter models
• Moderately complex tasks (e.g., routine code generation): use standard models
• Highly complex tasks (e.g., long-horizon architecture design): enable high-performance models

An intelligent routing layer can automate task distribution, minimizing compute costs while ensuring result quality. Model inference services offered by platforms such as Baidu AI Cloud typically support multi-model hybrid scheduling, which enterprises can configure flexibly based on their business scenarios.

5. Conclusion and Outlook

The “token black hole” phenomenon in large models is essentially a concentrated manifestation of the tension between leaps in model capability and compute cost control. As model architectures grow more complex and application scenarios deepen, raw API call volume or subscription quotas alone can no longer accurately reflect underlying compute consumption. Developers and enterprise users must establish fine-grained compute optimization systems at the architecture level, achieving controllable costs through call-chain optimization, token management, caching mechanisms, and dynamic alerting.

In the future, with architectural innovations such as sparse attention mechanisms and mixture-of-experts models, along with advances in hardware compute, per-token inference costs are expected to continue declining. But before such breakthroughs materialize, engineering optimization strategies will remain the key means of balancing model capability and operating cost. Developers need to cultivate “compute-aware” programming thinking, treating token consumption as one of the important metrics of code quality, and driving large model applications from “usable” toward “efficient and practical.”