AI Agent Costs Out of Control? Demystifying the Token Bill

1. A Scenario That Makes Engineering Leaders Cringe

It is Friday afternoon. You assign a long-standing bug fix to a coding agent, define the objective, and head into a meeting. Three hours later, you return to find the task marked unresolved. Meanwhile, the token usage shown in the console is already equivalent to half a day of a junior engineer’s labor cost.

More importantly, the expense has no direct relationship to whether the task was completed. The agent may have wandered through the repository, repeatedly tried and discarded approaches, and read large numbers of irrelevant files. Every step still generated a real bill.

This scenario is playing out across many engineering teams. The phrase “the most expensive intern” captures the frustration well: an agent may incur the cost of a full-time employee while offering the reliability of an intern. It can act independently, but it has little awareness of the cost of its actions. Even when it performs poorly, it continues to collect its “salary.”

The root problem is not any single model or tool. It is the structural mismatch between how agents work and how model usage is billed. The first step toward solving the problem is understanding where the money actually goes.

2. Why Agents Are Naturally Token-Intensive: The Structural Cost of Iterative Workflows

Traditional human–machine interaction follows a simple question-and-answer pattern. Its cost is relatively predictable and easy to cap. Coding agents follow a different workflow:

plan, retrieve, read, modify, verify, and re-plan.

Each iteration usually requires a complete model call. Each call must carry enough context for the model to continue working, including:

• The task description
• Repository structure
• Relevant file contents
• Previous actions
• Tool results
• The agent’s execution history

This creates three structural sources of cost.

1. Repeatedly carrying the context

Each time the agent takes another step, it may need to provide the model with much of its previous “memory” again. After a dozen or more iterations, the total input token volume can be several times larger than the final output—and in some cases dozens of times larger.

2. Large tool responses

A file search may return extensive code fragments. A test run may produce thousands of lines of logs. Much of this information is useful only at the moment it is returned, but it can still be included in the context of later iterations and contribute to billing.

3. Exploratory trial and error without a brake

When an agent reaches a dead end—for example, after misidentifying the root cause of a defect or repeatedly modifying the wrong function—it does not necessarily stop. It may continue to “think” and experiment, allowing the bill to grow continuously.

This explains one counterintuitive conclusion from agent-related research: in coding tasks, the majority of token consumption often comes from the input side, not the output side.

Many models charge more per output token than per input token. However, a higher output price does not mean that output accounts for most of the total cost. Once input context is repeatedly expanded across multiple iterations, input usage becomes the dominant expense.

This finding challenges the assumption that cost reduction mainly means shortening model output. The primary target for optimization is often context management.

3. Why Costs Can Vary So Widely: Three Levels of Attribution

The next question is one many practitioners ask: if two agents are both fixing a bug, why might one task cost very little while another consumes hundreds of dollars?

The difference can be analyzed at three levels.

Level 1: Task characteristics

Repository size, ambiguity in the issue description, and the number of files that must be inspected all directly affect how much content the agent needs to retrieve and read.

The more complex the task and the larger the repository, the higher the expected consumption. This relationship is consistent with common intuition.

Level 2: Model strategy

Different models can behave very differently. Some favor a “quick and decisive” approach: locate the likely cause quickly and make the smallest possible change. Others favor an exhaustive investigation, reading extensively and validating repeatedly.

For the same task, the total cost of completion can differ by several times—or even by an order of magnitude—depending on the model.

A common cause of budget overruns is choosing a model based only on the price of an individual call rather than the total cost of completing the task.

Level 3: Runtime randomness

This is the most counterintuitive factor. The same model can cost dramatically different amounts on the same task with the same prompt.

An agent’s execution path branches as it works. A different search result in one iteration or a different tool response can send all subsequent steps in a completely different direction. One run may go directly to the answer, while another may repeatedly circle around the problem.

This also leads to a more uncomfortable finding: spending and task quality are not necessarily positively correlated.

When complete execution traces are examined, failed tasks often consume the most tokens because they repeatedly pursue an incorrect direction. High spending is more likely to reflect path uncertainty and task difficulty than result reliability.

In other words, giving an agent a larger budget does not necessarily make it perform better. The mechanism does not support that assumption.

4. From Post-Execution Reimbursement to Upfront Budgets

If token consumption varies so widely, can teams estimate the approximate cost before a task begins?

Yes.

Researchers have used task metadata—such as model type, issue description length, repository size, and historical task statistics—to train consumption prediction models. Across multiple scenarios, these models performed significantly better than human experts relying on intuition.

The underlying logic is straightforward: token consumption is random, but the randomness has structure. Across a large number of tasks, consumption distributions contain patterns that can be learned.

Predictability makes governance possible. Cost estimates can be incorporated into three stages of execution.

1. Budget approval

Estimate the expected consumption range before starting a task. If the projected cost exceeds the team’s budget limit, automatically downgrade the task by switching to a smaller model, splitting it into subtasks, or routing it to a human.

2. Runtime circuit breaking

If actual consumption deviates significantly from the estimate—for example, if the agent has consumed twice the predicted amount without converging—trigger an alert or stop execution.

This prevents an agent from continuing to burn tokens while stuck in an unproductive loop.

3. Model selection

When multiple models are available, route tasks based on the total cost of completing the task, rather than the price of a single model call.

5. Practical Cost Governance

Before execution: Put the budget in the task definition

Teams should treat token budgets the way they treat resource quotas for continuous integration: as hard execution constraints, not as soft metrics reviewed after the fact.

def run_with_budget(agent, task, budget_tokens):
    estimate = cost_estimator.predict(task)   # Predict the consumption range before execution
    if estimate.p95 > budget_tokens:
        task = decompose(task)                # Split the task or route it to a human
    for step in agent.run(task, max_iterations=budget_limit):
        if step.cumulative_tokens > budget_tokens:
            save_checkpoint(step)             # Preserve the state for human takeover
            raise BudgetExceeded(estimate)

During execution: Add brakes to the agent loop

• Limit steps and iterations: Set a maximum number of iterations for each task. Pause execution and request human intervention when the limit is reached.
• Detect loops: Monitor recent tool-call sequences and code changes. Look for patterns such as reverting a change and then applying it again, or repeatedly reading the same group of files.
• Validate incrementally: Instead of waiting until the agent completes the entire task, add checkpoints during execution. This helps detect directional errors early.

After execution: Make every token traceable

Complete trajectory logs are the foundation of cost governance. For each model call, record information such as:

• Input volume
• Output volume
• Tool type
• Associated task
• Execution stage

These records should be sent to a logging service and used to build cost dashboards.

When a task consumes an unusual amount of tokens, the team should be able to replay the trajectory and identify exactly which steps caused the overrun. Individual incidents can then be converted into improved rules, prompt templates, and execution policies.

Use the available technical levers

• Prompt caching: Stable system prompts and repeated context can be cached to avoid repeatedly processing or billing the same content, where supported by the model service.
• Context trimming: Summarize and filter tool results so irrelevant content does not enter the history. For long-running sessions, periodically replace the full history with concise intermediate conclusions.
• Retrieve instead of injecting everything: Let the agent retrieve relevant code fragments as needed instead of placing the entire repository into the context.
• Route by task difficulty: Send lightweight tasks such as defect localization, code explanation, and solution review to smaller models. Reserve flagship models for complex modifications.

Platform support

Teams that do not want to build all of this infrastructure from scratch can use general capabilities provided by large-model service platforms.

A complete governance loop may include:

• Request logs that record input and output volumes
• Monitoring and alerts when consumption exceeds thresholds
• Cost dashboards that aggregate usage by task, model, or team

Major cloud providers, including Baidu AI Cloud, provide general observability and quota-management capabilities for large-model services. The team’s responsibility is to connect internal task identifiers and cost tags to these systems so that each bill can be automatically attributed to a specific business function.

6. Changing the Organizational Mindset

Cost governance is not only a technical issue. It also requires changes in management practices.

1. From success rate to cost per successful task

An agent with a high success rate but twice the average consumption may have a higher overall cost than a less expensive agent with a slightly lower success rate.

Evaluation should therefore consider the cost of successful completion, not success rate alone.

2. From a staffing mindset to a cost mindset

An agent should be treated more like an outsourced engineering team that requires scheduling, budgeting, and acceptance criteria—not like an always-available assistant whose use has no marginal cost.

3. From black-box trust to observable trust

As agents become more autonomous, people must retain visibility into their costs and critical decisions. Approval gates and takeover mechanisms at important checkpoints should not be removed simply because the agent can act independently.

7. Conclusion

The value of this research is that it turns the vague complaint that “agents burn money” into an engineering problem that can be measured, attributed, predicted, and governed.

The industry is moving beyond the stage of impressive demos and alarming bills and entering a more mature period focused on efficiency and cost-effectiveness.

Teams that can turn token bills from a blind box into transparent, traceable accounts will be better positioned to scale AI agents. They can capture the productivity gains of autonomous systems without ending up with an expensive and unreliable “intern.”