Evaluating LLM Capabilities for Long-Running Coding Tasks and Designing Usage Quotas
Long-running, autonomous coding sessions have become the mainstream way developers use AI coding assistants, but they raise new questions about model capability, token cost, and rate-limit design. This article breaks the problem into three layers: capability, cost, and governance. It explains why multi-hour tasks test LLMs differently from single-turn benchmarks, defines six evaluation dimensions and layered metrics with a reproducible testing workflow, shows how to route work between a flagship reasoning model and a lightweight executor, and examines how short-window and long-window quotas jointly constrain long-running tasks. It closes with engineering practices such as task decomposition, state persistence, budget-aware degradation, and automated quality gates that make overnight runs predictable.
Introduction: When a coding assistant has to work through the night
A developer set a clear goal for an AI coding assistant before bed, hoping it would run autonomously overnight. When he checked the next morning, the first short-window rate limit had already been hit; by the time he woke again, his long-window cumulative allowance was exhausted too.
Two days later, in a fresh cycle, he opened several coding sessions in a single morning: some at a higher reasoning effort level, others at the maximum, and he wrote “delegate concrete work to the lightweight model whenever possible” into the project-level context file. By late morning he had burned roughly a third of his short-window allowance and about a quarter of his long-window allowance. At that pace, the long-window quota would run out that same afternoon.
Two things in this feedback are worth noting. First, the developer’s usage had already shifted to “long-running task + high reasoning effort + primary/secondary model division of labor” — which is exactly the mainstream way people use AI coding tools today. Second, what frustrated him was not just the absolute size of the quota, but that when and how fast the quota would run out was almost impossible to predict in advance.
If we treat these tools as a complete engineering system, the problem splits into at least three parts:
- Capability layer: what models are good at, and where they fall short, in long-running coding tasks;
- Cost layer: how to allocate tasks and set reasoning effort so quality goes up while consumption goes down;
- Governance layer: how vendors design quota policies and how users plan their usage so long-running tasks finish reliably.
This article addresses all three layers.
1. Why long-running tasks are the new standard for measuring coding ability
1.1 From “one question, one answer” to “hand over a goal and walk away”
Early benchmarks of LLM coding ability relied mostly on single-function completion, single-file repair, or a handful of programming problems from standardized datasets. These evaluations have unified scoring criteria and are easy to reproduce, but they differ sharply from real development work: a real task is often a vague goal that spans multiple files, multiple languages, and existing engineering constraints, requiring the model to gather information, break down steps, verify results, and change course when things go wrong.
What developers do more often today is hand over a goal and walk away, letting the tool keep working for a long time until it finishes or hits the quota ceiling. This “long-running task” changes what we demand of a model in a fundamental way:
- A single correct turn captures only an instant; long-running tasks test the overall outcome after many consecutive decision steps;
- Context pollution, faulty edits, and wrong assumptions introduced at each step accumulate forward, so a small local error can send a whole night’s work off course;
- Once interrupted midway, the reasoning and generation already spent is not refunded, yet context must be rebuilt from scratch, significantly amplifying cost.
1.2 Capability gaps in long-running tasks differ from single-turn evaluation
In long-running tasks, the following abilities carry noticeably more weight:
- Long-horizon goal retention: after hundreds of tool-calling rounds, still not drifting from the original goal or forgetting earlier constraints;
- Engineering context understanding: accurately grasping cross-file, cross-module call relationships and not breaking existing interface contracts when making changes;
- Reasoning depth and quality: reaching correct conclusions on problems with no ready-made answer, and keeping the reasoning concise rather than being led astray by meaningless branches;
- Error recovery: locating root causes and self-correcting when tests fail, compilation breaks, or requirements are misunderstood, rather than repeatedly patching symptoms;
- Output quality: whether changes are readable and maintainable, whether documentation reads naturally, and whether other developers can quickly understand them.
1.3 No single model is strongest on every dimension
In practice you often see this: one class of models is better at open-ended reasoning — algorithm design or formal verification that has no standard solution and needs multi-step derivation; another writes more naturally in natural language and keeps its reasoning concise; yet another performs reliably on code-engineering tasks. Because the dimensions differ, no single ranking can capture them all.
It’s worth stressing that this capability landscape shifts quickly. A new release, an adjustment to reasoning effort levels, or a change in context window can invalidate past conclusions. Evaluation should therefore be done “by task type, by version, and against your own codebase,” not against a static leaderboard.
2. Evaluating capabilities in long-running coding tasks: dimensions, metrics, and methods
2.1 Six core evaluation dimensions
| Dimension | What it examines | How to observe it |
|---|---|---|
| Task understanding & planning | Whether it grasps the real need and decomposes it sensibly | Human review of the execution plan |
| Engineering context handling | Cross-file edits, interface compatibility | Regression tests, compilation results |
| Reasoning quality | Correct conclusions on open-ended problems; concise reasoning | Comparison to reference answers, review of the reasoning |
| Error recovery | Whether it can locate and fix the root cause after failure | Recovery rounds after fault injection |
| Instruction following | Whether it respects the conventions in context files | Counting violations |
| Output quality | Readability, maintainability, documentation naturalness | Blind human scoring |
2.2 Layer metrics by task length
| Metric | Definition | Significance |
|---|---|---|
| First-pass rate | Share of tests that pass without human intervention | Correctness of a single interaction |
| Final completion rate | Share of goals completed within a given budget and step count | Overall long-running performance |
| Regression rate | Share of new failing tests out of the total | Engineering safety awareness |
| Average fix rounds | Rounds between first failure and eventual pass | Self-recovery ability |
| Average consumption | Mean tokens spent per task | Basis for cost prediction |
| Cost per effective change | Effective lines changed / cumulative consumption | Direct measure of value for money |
| Human interventions | Number of times human correction was needed | Degree of automation |
Looking only at first-pass rate can under- or over-estimate a model: one that keeps self-correcting but takes longer may have a lower first-pass rate yet a higher final completion rate.
2.3 A reproducible evaluation workflow
To make evaluation reproducible, fix the model version, reasoning effort, context size, tool permissions, and random seed, and repeat each task multiple times before taking statistics. A reference workflow looks like this:
TASKS = load_tasks("repo_samples") # Sample from your own repos: bug fixes, small features, refactors, cross-module changesdef evaluate(model, effort, task, repeat=3):results = []for _ in range(repeat): # Repeat runs to observe stabilitysandbox = Sandbox(repo=task.repo) # Isolated repo copy and runtime environmenttrace = agent_loop(model=model, reasoning_effort=effort,goal=task.goal, workspace=sandbox, max_steps=task.max_steps)report = sandbox.run_tests() # Unified tests and static checks as objective criteriaresults.append({"passed": report.all_passed,"regression": report.new_failures,"steps": trace.step_count,"tokens": trace.total_tokens,"human_edits": trace.interventions,})return aggregate(results) # Completion rate, regression rate, average consumption, stability variance
2.4 Reasoning effort levels: stronger isn’t always better
Most coding assistants offer multiple reasoning effort levels, from a fast mode up to the maximum. A higher level gives the model more time to reason, at the cost of higher latency and token consumption in tandem.
One notable pattern is that some models have long, branch-heavy reasoning traces. On open-ended problems, multi-branch exploration lowers the chance of missing the correct solution; but on well-bounded local fixes, redundant reasoning both slows responses and increases consumption. So instead of always maxing out the effort level, pick it by task complexity:
- Single-file, low-risk changes: low effort;
- Cross-module refactors that need design trade-offs: medium-to-high effort;
- Open-ended derivation, formal verification, complex algorithm design: maximum effort.
The effect of the effort level on success rate should be quantified against your own evaluation set, not set by gut feeling.
3. Primary/secondary model collaboration: let the “thinker” plan and the lightweight model execute
The usage in the opening case is typical: high-effort reasoning sessions find bugs, review code, and handle overall design, while concrete edits are delegated to the lightweight model as much as possible. This is a tiered scheduling strategy.
3.1 Why tiered scheduling works
- Different cost structures: flagship models have a higher per-token cost and deeper reasoning; lightweight models respond fast at a lower unit price and suit high-concurrency execution;
- Context isolation: running subtasks in separate sessions keeps irrelevant information out of the main model’s context, reducing pollution;
- Contained risk: lightweight edits stay within a defined scope, are easy to verify quickly with automated tests, and can be rolled back directly when needed.
3.2 A routing strategy sketch
router:default: flagship # Flagship model as fallback when no rule matchesrules:- when: risk == high or ambiguity == highassign: flagship # High risk or vague goal: flagship plans- when: task.type == "lint_or_locate"assign: light # Mechanical tasks: lightweight model executes- when: context_tokens > 200000action: shard_by_module # Context too large: shard by module before dispatching- when: short_window.used_ratio >= 0.7 or long_window.used_ratio >= 0.8action: degrade # Quota tight: lower effort or switch to lightweight model
3.3 Project-level context file: write the division of labor into the repo
Many coding assistants let you place a project-level instructions file at the repository root to describe project structure, coding conventions, build and test commands, and collaboration rules. Writing “which work can go to the lightweight model and which must be gated by the main model” into it makes the division of labor stable, rather than relying on ad-hoc instructions each time.
A sample snippet:
## Task division convention- Flagship model owns: solution design, cross-module changes, complex bug localization, final code review- Lightweight model owns: local bug fixes, log triage, single-file small features, style-convention edits- Changes made by the lightweight model must pass unit tests and static checks before review- Changes to database schema, public interfaces, or auth logic must not be executed directly by the lightweight model
3.4 Risks of tiered scheduling
Tiered scheduling is not free:
- Without test coverage, lightweight edits may introduce hidden defects;
- Splitting subtasks too finely increases coordination overhead and total token consumption;
- Reviewing subtask results with the main model itself consumes quota.
The sound approach is risk-tiered authorization: hand low-risk, automatically verifiable tasks to the lightweight model; let the main model lead and personally sign off on high-risk, hard-to-verify tasks.
4. Availability of long-running tasks under quota mechanisms
4.1 Why tiered rate limiting and periodic quotas are needed
Back to the opening case: layering a short-window limit over a long-window quota is a common design in today’s AI coding tools. Understanding the technical logic behind it helps decide how to schedule tasks.
- Limited server-side compute elasticity: a single high-effort reasoning task may occupy inference resources for tens of minutes; if a few users run unlimited concurrency, queues lengthen sharply and affect everyone;
- Cost grows non-linearly with reasoning depth: reasoning models do substantial internal computation before producing a formal answer, so a higher effort level and a longer task make each task more expensive;
- Multi-tenant fairness: when total capacity is finite, resources can only be allocated fairly by time — that is, capping how much each user may use within a given period;
- Encouraging sustained usage: fixed-window limits cause a burst of releases at window boundaries, while rolling windows are smoother; multi-layer limits also bring “long-term total usage” under constraint.
4.2 How the typical mechanisms work
| Mechanism | Principle | Characteristics |
|---|---|---|
| Short-window quota | Counts usage within a period measured in hours | Suppresses burst consumption, eases compute peaks |
| Long-window quota | Accumulates usage over a week or month | Controls long-term total cost; the decisive constraint for long-running tasks |
| Rolling window | Counts usage over a fixed duration ending at the current moment | Avoids concentrated releases at period boundaries |
| Priority & queuing | Dynamically adjusts task order by load | Shaves peaks and fills valleys, but long-task wait time is unpredictable |
Note that platforms differ in how they define windows, reset quotas, and convert consumption across reasoning effort levels; refer to the terms of service for specifics. Only general principles are discussed here.
4.3 The long-window quota is the real bottleneck for long-running tasks
The short-window limit decides “how long you can work before an interruption”; the long-window quota decides “how long you can work in total over a week.”
The opening case illustrates the point well: just over two hours into the morning, roughly a quarter of the long-window quota was gone. Three reasons:
- Many sessions were open, most at high reasoning effort;
- The sessions ran in parallel, so resource usage stacked up;
- The project convention routed work to the lightweight model, but the main model still had to review tasks, review results, and make decisions — overhead that can’t be eliminated.
This yields a rule of thumb:
Available time ≈ remaining long-window quota ÷ consumption rate per unit time
And that consumption rate depends on the number of sessions, the reasoning effort level, and task complexity. When high-effort sessions run in parallel, the rate can be several times that of low-effort sessions, which is why the long-window quota gets drained so quickly.
4.4 A simplified implementation of the dual constraint
The pseudocode below shows how the dual constraint of “short-window rate limit + long-window quota” takes effect together:
class QuotaGuard:"""Dual-window quota guard (illustrative): check short window first, then long window"""def __init__(self, short_window, long_window, price_table):self.short_window = short_window # e.g., usage over the last 5 hoursself.long_window = long_window # e.g., usage over the last 7 daysself.price_table = price_table # Cost conversion factors by model and reasoning effortdef before_request(self, model, effort, max_tokens):cost = estimate_cost(model, effort, max_tokens)# Reject if either window is at its limit; allow only when both passself.short_window.assert_capacity(cost)self.long_window.assert_capacity(cost)return cost
In engineering, short windows are usually implemented with sliding logs or token buckets, and long windows with counters plus scheduled resets. For users, what matters is not the implementation details but the takeaway: both windows must be satisfied at once; whichever fills first is the current bottleneck.
5. Engineering practices for long-running tasks: make “running overnight” manageable
5.1 Task decomposition: make long tasks interruptible and resumable
- Split tasks at an “independently verifiable” granularity: each subtask has a clear goal and completion criterion, e.g., a specific test going from failing to passing;
- Set a budget cap for each subtask (max steps, max tokens) to prevent getting stuck in a loop on a single problem;
- Describe the goal specifically enough — constraints, acceptance criteria, and files that must not be changed — to reduce wasted consumption caused by ambiguity.
5.2 State persistence: the prerequisite for resuming from a checkpoint
Once a long-running task is interrupted by rate limiting, if it can’t be resumed the earlier consumption becomes sunk cost. It’s advisable to write the execution process to a persistent state file:
{"goal": "Fix the order module timeout and add regression tests","plan": [{"id": 1, "desc": "Locate the root cause of the timeout", "status": "done", "evidence": "Logs and call-chain analysis"},{"id": 2, "desc": "Implement timeout retry and fallback", "status": "done", "commit": "a1b2c3d"},{"id": 3, "desc": "Add concurrency regression tests", "status": "pending"}],"token_usage": {"short_window_ratio": 0.94, "long_window_ratio": 0.78},"resume_from": 3}
When resuming, handing this state together with a summary of the relevant code to the model saves the cost of re-investigation.
5.3 Budget-aware auto degradation
Add a quota check to the task script: when either window’s usage ratio crosses a threshold, automatically lower the reasoning effort, hand subsequent subtasks to the lightweight model, or move the task into a paused queue to continue after the window recovers. This turns “suddenly hitting the limit” into “graceful degradation.”
def budget_aware_dispatch(task, guard, policy):if guard.short_window.used_ratio >= 0.9 or guard.long_window.used_ratio >= 0.9:return defer(task) # Move to a paused queue, wait for the window to recoverif guard.long_window.used_ratio >= 0.7 and task.risk == "low":return dispatch(task, model="light") # Quota tight and risk low: switch to lightweight modelreturn dispatch(task, model="flagship", effort=policy.effort_for(task))
5.4 Observability and automated quality gates
Execution logs, token consumption, tool-call records, and test results for long-running tasks should all be collected uniformly, making it easy to later review “where the quota went.” For enterprise teams, general cloud capabilities — object storage, logging services, and monitoring alerts, such as those offered by Baidu AI Cloud — can centrally store run logs and usage metrics, alert you before the quota nears its limit, and elastically scale compute resources on demand to ease queuing pressure during peak hours.
On quality, configure an automated gate for each subtask:
Local static checks → unit tests → integration tests → code review → merge↑ On any failure, revert that subtask's changes and log the reason for the model to fix
Only changes accepted by the gate count as “effective output,” which lets the results of long-running tasks actually land in the codebase.
5.5 Recommendations for platform-side quota design
From a developer-experience standpoint, quota mechanisms can be improved in these ways:
- Predictable: show the remaining rolling-window amount and the estimated time to exhaustion at the current rate;
- Early warnings: alert before hitting a threshold, rather than interrupting when a task is halfway through;
- Recoverable: support resuming from a checkpoint after an interruption, so earlier investment isn’t wasted;
- Finer granularity: price by reasoning effort level, or let users buy long-running task quota separately;
- Sensible ratios: the proportions among multi-layer quotas should match real usage patterns, avoiding a long-window quota that is too tight relative to the short-window one.
For consumers, rather than hoping for unlimited quota, it’s better to build an evaluation and budgeting system centered on task completion rate and cost per effective change. This both helps with model selection and turns quota into a plannable resource rather than a randomly appearing obstacle.
