Why Do Agent Credits Run Out of Control? Practical Cost Governance for LLM Agents

From a Billing Dispute to a Systemic Problem

A recent incident involving a major LLM assistant product made headlines: within a day of deploying a new model version, users reported that their credits were nearly exhausted after only a short period of task execution. The provider issued a public apology, replenished quotas for subscribers, and promised to reduce unnecessary sub-agent calls and verbose outputs. This is far from an isolated case. Any product that evolves from a “question-and-answer tool” to a “task executor” eventually hits the same wall—users’ intuition about consumption is still rooted in the chat era, while the actual consumption structure of an agent is an entirely different beast.

Beneath the surface, three interconnected engineering problems emerge:

1. Metering: How to define, measure, and present “how much was used” to the user.
2. Call governance: How to eliminate token-amplifying steps while still completing the task.
3. Trust: How to help users understand and accept unexpected consumption patterns.

Let’s examine each of these in turn.

Why Agents Burn Tokens at an Alarming Rate

1. The Multiplicative Effect of Reasoning Loops

In a chat scenario, the cost of a single query is roughly “input + output.” An agent, by contrast, operates in a loop: plan → call a tool → observe the result → plan again. Each iteration re-sends the entire context history to the model. If the context size per round is C and the agent runs for n rounds, the total encoding cost approaches n×C. Worse, this grows superlinearly as task complexity increases.

2. The Amplification Effect of Sub-Agent Calls

In a multi-agent architecture, a primary agent often decomposes a task into subtasks handled by sub-agents. These sub-agents don’t work in a vacuum—they carry their own system prompts, tool descriptions, and context fragments inherited from the parent task. A task split into three subtasks, each running three loops, results in the context being copied and re-encoded many times. The deeper the nesting and the higher the concurrency, the higher the redundant encoding cost. This is why “reducing unnecessary sub-agent calls” becomes a key optimization target for such services—it is not about saving small change; it is about shutting down the largest amplifier.

3. Verbose Tool Outputs

Tool responses—like web page snapshots, file contents, or execution logs—can easily run to thousands of tokens that enter the context untouched. Without truncation and structured extraction, a single unsuccessful retrieval step can consume a disproportionate share of the budget.

4. Uncontrolled Output Length

Without explicit constraints, models tend to produce exhaustive, explanatory responses. For agents, verbose intermediate steps slow down execution and directly inflate the bill.

5. The Sunk Cost of Failed Retries

In long-horizon tasks, a failure in the middle or late stages—without checkpointing or partial result persistence—often forces the entire chain to restart, invalidating all the tokens already spent.

System Design for Quotas and Rate Limiting

1. Metering: Token-Based, Request-Based as a Secondary Metric

Counting requests alone is almost guaranteed to misrepresent agent usage—“one task execution” can consume ten times more tokens than another. A more reasonable approach is to use tokens as the primary unit, supplemented by request rate and concurrency controls to prevent abuse.

2. Tiered Quotas

It is advisable to apply limits at multiple scopes, as shown below:

quota:                              # numbers are illustrative only
  account_monthly_tokens: 10000000  # account-level monthly quota
  project_daily_tokens: 1000000     # project-level daily quota
  session_task_budget: 200000       # per-task budget cap
  rate_limit:
    requests_per_minute: 60
    tokens_per_minute: 200000
    max_concurrent_tasks: 5

The point of tiering is to contain “blowouts” within the smallest possible scope: a runaway task should not bring down the entire account.

3. Rate Limiting Algorithms

A token bucket works well for scenarios that allow short bursts while constraining the long-term average rate. A leaky bucket suits cases requiring strong output smoothing. For agent concurrency, an additional cap on concurrent tasks is necessary. A simple token bucket implementation:

import time
class TokenBucket:
    def __init__(self, capacity: int, refill_rate: float):
        self.capacity = capacity        # bucket capacity
        self.tokens = float(capacity)
        self.refill_rate = refill_rate  # refill tokens per second
        self.last = time.monotonic()
    def allow(self, cost: int = 1) -> bool:
        now = time.monotonic()
        self.tokens = min(self.capacity,
                          self.tokens + (now - self.last) * self.refill_rate)
        self.last = now
        if self.tokens >= cost:
            self.tokens -= cost
            return True
        return False

4. Budget Guards: Move the Check Inside the Agent Loop

Rate limiting should not happen only at the gateway. By introducing a budget guard inside the agent loop, you can gracefully terminate near the limit, preserve partial results, and avoid hard failures mid-execution:

def run_agent(task: str, budget_tokens: int):
    used = 0
    ctx = build_initial_context(task)
    for _ in range(MAX_STEPS):
        est = estimate_input_tokens(ctx)
        if used + est > budget_tokens:          # budget is running low; wrap up early
            return finalize(ctx, mode="partial", used=used)
        resp = llm_call(ctx)
        used += resp.total_tokens
        if resp.is_final:
            return finalize(ctx, mode="done", used=used)
        ctx = append_and_trim(ctx, resp, tool_observation(resp))
    return finalize(ctx, mode="max_steps", used=used)

Engineering Practices for Cost Governance

1. Sub-Agent Call Governance

• Evaluate necessity before dispatch: Do not split a subtask that the primary agent can complete in a single round.
• Limit decomposition depth and concurrency width: For example, at most two levels deep and no more than N parallel subtasks.
• Return summarized results: Only write conclusions and key evidence back to the primary context, never the full process.

2. Context Management

• Truncate: Use a rolling window to keep the system prompt and recent critical information while discarding stale observations.
• Compress: Perform structured extraction on large tool outputs, keeping only field-level conclusions.
• Deduplicate and cache: Reuse identical prefixes (system prompts, tool lists) and leverage server-side prefix caching to reduce redundant encoding costs.

3. Output Constraints

• Explicitly set a maximum output length.
• Require structured output (e.g., JSON fields) to enforce conciseness by format.
• Instruct the model that intermediate steps should only contain conclusions and the next action.

4. Model Routing

Not every step needs the most capable model. Delegate planning and final decisions to a flagship model, and route summarization, formatting, and simple extraction tasks to lightweight models. This can significantly cut costs with minimal impact on task success rates.

5. Failure and Retry Handling

• Design idempotent tool calls for critical operations.
• Persist intermediate state at each step to support resumption rather than full re-execution.
• Set retry caps and exponential backoff to avoid amplifying load during failures.

Usage Transparency: Turning Quotas into a Product Feature

Beyond technical optimization, the real antidote to “quota incidents” is transparency and predictability:

1. Real-time usage display: Show consumed tokens and estimated remaining steps during task execution, not just at the end.
2. Pre-execution estimation and confirmation: Provide a range estimate for long, high-consumption tasks and allow users to choose between “full execution” and “light mode.”
3. Detailed usage breakdown: List token composition by task and by call so users can answer the question, “Where did my quota go?”
4. Graceful degradation at the limit: Clearly indicate the remaining quota and offer a low-consumption mode instead of abruptly terminating the task.
5. Dispute response: Replenishing quotas can stop the bleeding, but explainable rules and predictable consumption are the real foundation of long-term trust.

These capabilities do not have to be built from scratch. Logging services can record call details, monitoring and alerting can track quota levels and issue early warnings, and object storage can hold task intermediate artifacts. On cloud platforms like Baidu AI Cloud, such generic components can be composed directly to quickly assemble a full “meter-show-alert” pipeline.

Conclusion: The Stronger the Model, the Tighter the Budget

“The stronger the model, the harder the business” is more of a misperception. What becomes difficult is not the model itself but the failure of metering and call governance to keep pace with capability growth. The stronger the model, the higher the ceiling for single-task capability—and the more complex the tasks users hand over, the more intricate the consumption structure becomes. By treating quotas as a product feature—with tiered limits, budget guards, call governance, and transparent metering—agent services can achieve both cost control and user trust.