When a Ten-Minute Run Burns the Entire Quota: The Real Cost Structure of Long-Horizon Agents

A model recently completed a version update with a clear focus: coding capability and long-horizon agent tasks. Around these two themes, the official release shared two sets of key benchmark data. Interestingly, what sparked the most discussion was not the scores themselves, but an accompanying user report—the quota was exhausted within ten minutes.

On the surface, this sounds like a complaint. In reality, it points to an often-overlooked engineering fact: when an agent shifts from single-turn Q&A to multi-step execution, the shape of token consumption changes fundamentally. A single long-horizon task may involve dozens of tool calls, multiple rounds of self-verification and rollback. The cost of a single task is no longer determined by prompt length, but by the number of execution steps and failure retries. Improvements in performance and speed often simultaneously amplify the rate of resource consumption per unit of time.

This article starts from the benchmark data, breaks down the technical implications of the capability improvements, and discusses how to approach quota planning and task design in long-horizon agent scenarios.

1. Terminal Task Benchmark: What Does 65.7% Actually Mean

On Terminal-Bench 4.0, this version scored 65.7%. Benchmarks of this type share a common characteristic: they require a coding agent to complete complex tasks in a realistic, resource-constrained terminal environment, with each task judged by dedicated test cases.

1.1 Why the Evaluation Environment Matters

Traditional code generation benchmarks mostly take the form of function-level completion or single-file fixes, where input and output boundaries are clear—the model only needs to generate a syntactically correct, logically sound code snippet to earn a high score. Terminal tasks are entirely different:

• The environment is stateful. A file modification in an earlier step affects all subsequent operations, and errors accumulate.
• Resources are constrained. Disk, memory, network permissions, and available command-line tools are deliberately limited. The model must understand what can and cannot be done in this environment.
• Evaluation is end-to-end. The dedicated tests do not look at what you wrote—they only check whether the final system state matches expectations.

This means a score of 65.7% is not a measure of “how well the code is written,” but a measure of “whether the job can get done in an environment with friction.” The latter is clearly harder.

1.2 Reading the Score in Context

Referring to results published in the same batch, another mainstream model scored 58.2%, and a model long regarded as a coding benchmark leader scored 66.4%. The relationship among the three is: this version has surpassed the former and trails the latter by less than 1 percentage point.

Two observations stand out from this distribution:

• The top tier is converging. When the gap between leading models compresses to within 1 percentage point, the discriminative power of a single benchmark approaches noise levels. The marginal significance of scores is declining, while the weight of engineering usability, stability, and cost structure is rising.
• “Approaching” does not mean “equivalent.” A 1-percentage-point gap in real projects can manifest as systematic failures on specific task types—for example, dependency resolution involving multi-stage builds, or debugging steps that require precise understanding of terminal error semantics. When selecting a model, validate with your own task set rather than relying on public leaderboards.

1.3 Long-Horizon Coding Benchmark: The Gain from 55.95% to 61.2%

In an internal long-horizon coding benchmark, this version scored 61.2%, a notable improvement over the previous generation’s 55.95%. This benchmark covers multiple engineering domains and focuses on evaluating three capabilities: multi-step execution, tool calling, and end-to-end problem solving.

From 55.95% to 61.2%, the absolute gain is roughly 5 percentage points, and the relative gain approaches 10%. In long-horizon tasks—where “every additional step adds another chance of failure”—such an improvement typically comes from a combination of several directions:

1. Planning stability: reducing mid-task cases of “forgetting the goal” or “repeating ineffective operations”;
2. Error recovery: correctly reading error messages after a command fails and adjusting strategy, rather than retrying in place;
3. Context management: maintaining tracking of key state across long conversations and large volumes of tool output;
4. Tool-calling precision: more accurate parameter construction, reducing invalid calls caused by format issues.

It is worth emphasizing that long-horizon capability improvements are often non-linear. When single-step success rate rises from 95% to 97%, the overall success rate difference on a 30-step task is significantly amplified. This is also why long-horizon benchmarks carry more reference value.

2. Faster, Then Faster to Burn Through: The Coupling of Speed and Cost

The release notes mention “faster.” In agent scenarios, speed improvements do not bring entirely good news.

2.1 How Speed Changes the Consumption Curve

Suppose a task requires an average of 40 model calls, with each call consuming a certain amount of input and output tokens. When inference speed improves and per-call latency drops, the number of calls that can be completed per unit of time rises, so:

• In the same ten minutes, more steps can be executed;
• For the same task, completion time is reduced;
• But in the same ten minutes, more quota is consumed.

This is the technical explanation for the “ten-minute burn” phenomenon. It is not that the quota got smaller—it is that the consumption rate has been amplified by speed. In interactive use, this effect is especially pronounced: user wait times shrink, operating pace quickens, task submission frequency rises, further pushing up peak consumption.

2.2 Breaking Down the Cost Structure of Long-Horizon Tasks

The cost of a typical long-horizon agent task can be roughly decomposed into four parts:

Total cost ≈ planning calls × N_plan
           + tool execution feedback processing × N_tool
           + error retries × N_retry
           + context replay × N_context

Among these, N_retry and N_context are the two most easily underestimated. Each retry after a tool call failure re-carries the historical context; each new round of observation results makes the input of subsequent calls longer. This means cost growth with respect to step count tends to be super-linear, not linear.

2.3 Practical Advice: Manage Quota as an Engineering Resource

• Layer your tasks: separate exploratory tasks from deterministic ones. The exploration phase can allow failures and retries, but should cap the maximum number of steps; the deterministic phase should execute with a smaller context window.
• Set step limits and early termination conditions: define a clear maximum iteration count and success criteria for the agent to avoid meaningless loops.
• Compress observation results: summarize or structurally trim tool output before feeding it back, avoiding raw logs repeatedly entering the context.
• Weigh batch versus serial: parallel tasks simultaneously raise instantaneous consumption; when quota is tight, consider switching to serial scheduling.
• Establish a cost baseline: record call counts and token usage for typical tasks to form a predictable budget model, rather than relying on a “gut feeling.”

3. After the Capability Boost: Which Tasks Truly Benefit

Combining the focus areas of both benchmarks, the task types that benefit most clearly can be summarized as follows:

Clear beneficiaries:

• Engineering tasks requiring multi-step builds and verification, such as dependency installation, build failure troubleshooting, and test-fix loops;
• Operations tasks requiring configuration and script orchestration in constrained terminal environments;
• Automated workflows that pass state between multiple tools, such as going from log analysis to configuration changes to verification.

Still requiring caution:

• Tasks with extremely high requirements for precise numerical values or strict formats, where a 1-percentage-point gap may fall exactly on the critical path;
• Ultra-long task chains (dozens of steps or more)—even with a high single-step success rate, the overall success rate still decays rapidly;
• High-concurrency batch scenarios, where cost and stability risks are simultaneously amplified.

4. A Practical Checklist for Developers

1. Validate with your own task set, not just public leaderboards. Prepare 20–50 tasks covering core business paths, and record success rate, average step count, and average consumption.
2. Estimate per-task cost first, then do capacity planning. Translate “ten-minute burn” into “average consumption per task and how much peak concurrency can be supported.”
3. Add guardrails for the agent: maximum steps, timeouts, retry limits, and dangerous command allowlists.
4. Treat context as a budget: regularly trim and summarize tool output to prevent uncontrolled historical replay.
5. Distinguish interactive from batch scenarios: interactive use values response speed; batch processing values unit cost. Their quota strategies should differ.

Conclusion

The signal from this upgrade is clear: the capability curve of long-horizon agents is still rising, and the gap between leading models is converging into a range that requires fine-grained validation to distinguish. At the same time, the stronger and faster the capability, the more prominent the structural problem of resource consumption becomes. “Burning through the quota in ten minutes” is not a capability defect—it is the inevitable result of usage patterns and cost models not yet matching. For developers, what is truly valuable is not chasing that 1 percentage point on a leaderboard, but building a measurable, controllable, and predictable agent engineering system—so that capability improvements genuinely translate into delivery efficiency, rather than surprises on the bill.