Scaling LLM Agents: Capability Upgrades and Token Cost Governance

Introduction: Capabilities Near the Top, Quotas Draining in Minutes

Recent iterations of agent-oriented large language models have produced two curves pointing in opposite directions.

The first is the capability curve. New-generation models now match or approach leading closed-source models on coding, long-horizon tasks, and knowledge-intensive benchmarks. They have evolved from “able to answer questions” to “able to sustain work,” covering real production scenarios such as programming, financial research, and 3D content creation.

The second is the consumption curve. As models genuinely begin “working continuously,” token consumption grows by orders of magnitude. Many development teams report that a single complex agent task can burn through an entire day’s API quota in ten minutes. This is no accident—it is a structural characteristic of agent workloads.

Understanding the technical mechanisms behind these two curves is a prerequisite for sound model selection and cost governance in the agent era. This article covers four areas: training engineering, capability evaluation, cost structure, and governance practices.

Where the Capabilities Come From: The Engineering Logic of Post-Training Scaling

1. From Pre-Training Scaling to Post-Training Scaling

As the marginal returns of pre-training narrow, the industry is directing incremental compute toward post-training—scaling reinforcement learning (RL) to hundreds of downstream domains. The core idea: for each target domain (coding, agents, financial research, 3D content creation, etc.), build verifiable reward signals and interactive task environments, letting the model acquire genuine skills through massive “try–feedback–correct” loops rather than merely fitting surface patterns through instruction tuning.

2. Six Layers of Infrastructure for Large-Scale Joint Training

Scaling RL to hundreds of domains tests not a single algorithm but an entire systems stack. Industry practice typically calls for systematic upgrades in the following areas:

• RL framework: supports high-throughput rollout sampling, reward aggregation, and training stability control;
• Long-context inference: KV cache management for million-token contexts, context parallelism, and attention computation optimization—these directly determine training sampling efficiency;
• Training efficiency: raise compute utilization and provide fault-tolerant recovery so long-running jobs are not discarded midway;
• Sandbox infrastructure: code execution and tool calls must run in securely isolated environments to prevent unauthorized operations during training;
• Scheduler: orchestrates training and inference workloads across heterogeneous compute so that no resource sits idle;
• Environment quality-check platform: the most commonly overlooked layer. If the interactive environment itself is flawed, RL will exploit loopholes in the environment instead of learning real capabilities, producing inflated benchmark scores and underwhelming real-world performance.

3. Speed Tiers: Same Intelligence, Faster Output

A common industry practice is to ship a speed variant with intelligence on par with the standard version, using inference-side optimizations to boost output speed by roughly 2–3x at a correspondingly higher price. Whether it is worth paying for comes down to a single criterion: whether time cost or token cost dominates your scenario. In latency-sensitive settings such as interactive sessions and online coding, faster output translates directly into better experience and throughput; for offline batch jobs, the speed advantage is almost never realized, and paying double makes little sense.

How to Evaluate Models That Can “Work Continuously”

1. From Static Q&A to Interactive Environment Benchmarks

Traditional knowledge benchmarks measure what a model knows; terminal-environment benchmarks measure what it can accomplish. The model is placed in a realistic, resource-constrained terminal environment and must autonomously complete multi-step tasks, each with dedicated test cases—pass or fail is decided by whether the tests pass. This setup reproduces the constraints of real engineering work: tangled dependencies, long procedures, and limited resources. Its discriminating power far exceeds that of multiple-choice knowledge quizzes.

2. Key Metrics for Long-Horizon Evaluation

For long-horizon coding evaluations that run for hours, completion rate and test pass rate are not enough. Three often-overlooked dimensions deserve attention: token cost per task (economy), number of human interventions (autonomy), and context consistency (whether the model “forgets” earlier constraints in the later stages of a long task).

3. Three Recommendations for Building Your Own Evaluations

• Build a private evaluation set from your own real business tasks; public benchmarks can only filter out clearly unqualified models.
• Record cost metrics: the same task can cost several times more on one model than on another.
• Keep comparison runs identical in configuration, so prompt differences do not lead to wrong conclusions.

Why Token Costs Spiral: The Cost Structure of Agent Workloads

1. Three-Tier Pricing

The prevailing industry model prices per million tokens across three tiers: cache-hit input, cache-miss input, and output. The typical pattern: cache-hit input is the cheapest, cache-miss input is often an order of magnitude more expensive, and output costs several times more still. This makes agent applications far more cost-sensitive than conversational ones. In a chat application, the context of each request is roughly bounded; in an agent application, the context keeps expanding.

2. Context Bloat in the Agent Loop: The Mathematical Root of Cost Blowups

The standard agent loop is “think—call a tool—observe the result—think again.” The catch: every request must carry the entire prior history. Suppose each round adds Δ tokens (tool outputs such as logs, error stack traces, and directory listings are often huge). After N rounds:

Cumulative input tokens ≈ Δ × N(N+1) / 2

With 40,000 new tokens per round over 50 rounds, cumulative input reaches roughly 51 million tokens—and that is before counting output. This is the mathematical essence of “burning the quota in ten minutes”: context bloat makes input consumption grow quadratically with the number of rounds.

3. Other Cost Amplifiers

• Verbose tool output: unprocessed logs and stack traces are dumped straight into the context;
• Multi-agent collaboration: N agents each maintain their own independent context, multiplying consumption;
• 24/7 long-horizon tasks: continuous, never-ending consumption;
• Verbosity bias: some models tend to over-explain instead of giving conclusions directly.

Cost Governance in Practice: Six Actionable Techniques

1. Cache-First Message Organization

Place stable content first and volatile content last, reuse identical prefixes as much as possible, and push the cache hit rate as high as it can go:

# Anti-pattern: volatile content at the front, cache hit rate near zero
messages = [
    {"role": "user", "content": f"Current time: {now()}, task: {task}"},
    {"role": "system", "content": SYSTEM_PROMPT},
]
# Recommended: stable prefix + compressed history + latest observation at the end
messages = [
    {"role": "system", "content": SYSTEM_PROMPT},   # stable
    {"role": "system", "content": TOOLS_SPEC},      # stable
    {"role": "user", "content": task_spec},         # stable within a task
    *compressed_history,                             # periodic summaries
    {"role": "user", "content": latest_observation} # volatile
]

2. Context Compression and Externalized State

Summarize history at milestones, keeping only key decisions and the verbatim text of the most recent K rounds. Go further: treat the file system, databases, and object storage as “external memory,” keeping only pointers and summaries in the context and retrieving details on demand.

3. Task Decomposition and Sub-Agent Isolation

The main agent maintains only planning-level context, delegating execution details to sub-agents and receiving back only their conclusions. In effect, this splits a single quadratically growing long context into multiple linearly growing short contexts.

4. Tiered Model Routing

Route simple tasks (format conversion, information extraction) to lightweight models, and call the flagship model only for complex reasoning and long-horizon planning. Combining rule-based and semantic routing typically eliminates 30–50% of wasted spend.

5. Budget Control and Circuit Breakers

class AgentBudget:
    """Task-level token budget and circuit-breaker control"""
    def __init__(self, max_in, max_out, max_rounds=50):
        self.max_in, self.max_out = max_in, max_out
        self.max_rounds = max_rounds
        self.used_in = self.used_out = self.rounds = 0
    def precheck(self, next_input_tokens):
        if self.rounds >= self.max_rounds:
            raise RuntimeError("Max rounds reached; consider decomposing the task")
        if self.used_in + next_input_tokens > self.max_in:
            raise RuntimeError("Input budget nearly exhausted; trigger history compression or abort")
    def record(self, in_tok, out_tok):
        self.used_in += in_tok
        self.used_out += out_tok
        self.rounds += 1

Estimate before every call and trip the breaker when limits are exceeded—this is the last line of defense against a single task dragging down the overall budget.

6. Constraining Outputs and Tool Results

Cap maximum output length and enforce structured output to cut redundant phrasing; truncate and preprocess tool results before they enter the context (for example, keep only the tail of a log, or only diff summaries for code changes).

Harness Selection: Build or Buy

A harness that supports long-running agents (sandboxed execution, tool registration, state persistence, multi-agent collaboration) can be obtained in two ways. Managed agent services suit 24/7 long-horizon tasks and multi-agent collaboration, saving substantial operational investment. The self-built route uses general-purpose cloud capabilities as its foundation—for example, on Baidu AI Cloud, a container platform hosts the sandboxes and execution environments, object storage persists task state and checkpoints, logging services plus monitoring and alerting deliver end-to-end observability, and message queues decouple communication between agents. The core value of building it yourself is a controllable environment and data that never leaves your boundary. When choosing, score five dimensions: task duration and concurrency scale, sandbox isolation level, state persistence and checkpoint recovery, whether token costs can be attributed per task, and compliance and data-boundary requirements.

Conclusion

Post-training scaling has produced models that “can work continuously”—along with a structural surge in token consumption. Capability and cost are two sides of the agent coin. Engineering teams should treat token cost as a first-class citizen in architecture design: cache-first organization, context governance, tiered routing, and budget circuit breakers. The stronger the model, the more the engineering behind it needs to count every token.