Long-Running AI Coding Agents: Capability Boundaries and Evaluation Practices

    This article examines how coding agents should be evaluated for real engineering work rather than one-shot code generation. It covers the shift toward long-task execution, feedback loops, context management, the benefits and costs of multi-agent orchestration, a minimal evaluation workflow, cost accounting under compute quotas, six stability dimensions, and practical integration practices.

    1. The evaluation paradigm has shifted: from generating snippets to completing engineering tasks

    Over the past two years, the dominant way to assess code-generation ability has been to ask models to complete function-level code or solve single-turn algorithm problems, with metrics such as build pass rate or unit-test pass rate. However, real software development tasks are rarely single-turn. They usually involve requirement clarification, confirmation of repository conventions, multi-file modifications, dependency installation, runtime verification, and error-driven self-repair. If any step fails, the entire chain may become unusable.

    Therefore, the capability of a coding agent should be observed along at least three orthogonal dimensions:

    1. Task decomposition: Can it turn a vague requirement into an ordered set of verifiable sub-steps?
    2. Long-horizon iteration: Can it keep the goal from drifting across multiple read/write and trial-and-error cycles, and adjust its approach based on feedback?
    3. Environment awareness: Can it understand the existing directory structure, framework versions, script entry points, and runtime state, instead of generating code from thin air?

    Single-shot generation quality is only an expression of the intersection of these dimensions. Using it alone to evaluate agent-level products often leads to misleading conclusions.

    2. Long-task iteration: the dividing line for agent-level coding

    2.1 Technical characteristics of “long tasks”

    Tasks that an agent must handle usually have the following characteristics: they exceed what the context can hold in a single pass; state must be persisted across rounds; external dependencies are unstable; and failure causes often must be inferred from runtime output. These characteristics mean that success cannot come from “writing it correctly in one shot.” It depends on a feedback loop.

    2.2 The verification loop is the physical foundation of iterative capability

    Without automatically executable verification, so-called “self-iteration” is merely the model freely associating. For an agent to iterate, every sub-task must be machine-checkable as success or failure: it must compile, tests must run, type checks must pass, APIs must be reachable, and logs must show no anomalies.

    A minimal usable iteration loop usually looks like this:

    1. def agent_loop(task, workspace, verifier, max_rounds=20):
    2. plan = decompose(task) # requirements -> ordered sub-steps
    3. for step in plan:
    4. for round_idx in range(max_rounds):
    5. diff = propose_change(step, workspace.read_context())
    6. workspace.apply(diff) # controlled write with snapshots
    7. result = verifier.run(step) # compile/test/smoke verification
    8. if result.passed:
    9. break
    10. step = repair(step, result.logs) # converge using failure output
    11. else:
    12. workspace.rollback(step) # exceed round limit; roll back to avoid pollution
    13. raise UnresolvedTask(step, result.logs)
    14. return summarize(workspace.commit())

    There are four engineering points in this pseudocode that are often overlooked: writes must go through a controlled channel and leave snapshots; verification must be repeatable and idempotent; iteration must have an upper bound to prevent infinite loops; and failures must allow rollback instead of leaving half-finished work in the workspace.

    2.3 Context management determines long-horizon stability

    In long tasks, context length is not the only bottleneck; context relevance matters more. In practice, three types of strategies can be combined:

    • Layered summarization: Compress early exploration conclusions into structured notes such as decisions, constraints, and ruled-out options. Later rounds carry only the notes rather than raw logs.
    • Retrieval on demand: First locate relevant files and symbols, then load only the minimal snippets, instead of stuffing the entire repository into the context.
    • Contracts first: First fix the interfaces, data structures, and configuration items, then let the agent fill in implementations. This can significantly reduce direction drift during multi-round changes.

    Error accumulation is an inherent risk in long tasks: every step may be “roughly correct,” but after ten steps the work may already have diverged from the original intent. Therefore, turning requirements into explicit, verifiable contracts is more effective than repeatedly telling the model to “stay consistent.”

    3. Multi-agent orchestration: applicability boundaries and engineering costs

    “Multi-agent parallelism” is often promoted as a capability upgrade, but it is essentially an architectural choice that trades parallelization for throughput. It yields net positive value only in specific scenarios.

    3.1 Scenarios that truly suit parallelism

    • Independent, separable sub-tasks, such as batch file migration or multi-module test completion.
    • Large-scale codebase inspection, such as searching for specific patterns across repositories and aggregating a checklist.
    • Multi-source information aggregation, such as extracting several public materials into a unified structure and then cross-checking them.
    • Cross-review division of labor: one agent implements, while another focuses exclusively on finding faults.

    3.2 Scenarios that should not be parallelized

    When requirements are highly serial, shared mutable state is extensive, or verification costs are much higher than implementation costs, multi-agent orchestration only amplifies conflicts and rework. In such cases, “one agent plus rigorous verification” is often faster and cheaper.

    3.3 The cost list

    Increasing parallelism brings three types of costs: context synchronization cost, when sub-agents see local information inconsistent with the main goal; merge cost, when the same file is concurrently modified and causes semantic conflicts; and cost amplification, when the number of rounds grows approximately linearly with the number of agents. In addition, observability in multi-agent systems becomes significantly worse. Debugging requires the complete scheduling trace, not just the final conversation.

    A schematic orchestration skeleton:

    1. def swarm(orchestrator_task, max_parallel=4):
    2. subtasks = split_by_independent_module(orchestrator_task)
    3. results = []
    4. with ThreadPoolExecutor(max_parallel=max_parallel) as pool:
    5. for group in batch(subtasks, max_parallel):
    6. futures = {pool.submit(worker_agent, t, shared_contract) : t
    7. for t in group}
    8. for f in as_completed(futures):
    9. results.append((futures[f], f.result())) # return results to the orchestrator for unified merging
    10. return orchestrator_merge(results) # conflict detection and harmonization

    The key design is that all sub-agents share the same contract, including interface definitions, naming conventions, and data formats, while only the main agent performs merging. Conversely, if sub-agents decide interfaces independently, the merge stage will almost inevitably require rework.

    4. A reusable minimal viable evaluation process

    Take a typical requirement such as “build a financial reporting season information monitoring application” as an example. Evaluating a complete delivery is more meaningful than evaluating a single-turn output. It is recommended to observe the following stages:

    1. Requirement-to-specification conversion: Write the colloquial requirement into a structured task description, and observe whether the agent proactively asks about missing items.
    2. Frontend skeleton delivery: Observe layout fidelity, whether interaction states such as loading, empty, and error are complete, and whether accessibility meets expectations.
    3. Data contract landing: Observe whether it defines data structures before implementation, or keeps changing them during coding and causes repeated adjustments.
    4. Backend enhancement: Add APIs and persistence on top of the existing frontend, and observe whether the agent can understand the frontend conventions it previously generated.
    5. Runtime verification: Observe whether it proactively writes and runs tests, and whether it can locate problems itself based on failure logs.

    A directly reusable task specification template:

    1. task: monitor-reporting-season
    2. goal: Provide a unified dashboard that aggregates disclosed reports by company and date and marks their status.
    3. constraints:
    4. - Only use the standard library or dependencies already approved by the team.
    5. - All external requests must set timeouts and retry limits.
    6. - Do not hardcode any credentials.
    7. acceptance:
    8. - Empty data, timeout, and missing fields must all produce clear UI feedback.
    9. - Core flows must have automated tests that can be run locally with one command.
    10. out_of_scope:
    11. - User authentication and multi-tenancy are not implemented for now.

    The value of this template is that it turns “how well it was done” into a verifiable proposition. Without acceptance criteria, reviews degrade into debates over personal taste.

    5. True accounting of quota and compute cost-performance

    In the agent era, the price per request alone is insufficient to describe cost. A more reasonable metric is the total cost of completion, which includes effective rounds, wasted trial-and-error rounds, human intervention time, and rework cost.

    It can be roughly written as:

    1. Total cost = Σ(input + output compute cost per round) + human review hours × hourly cost + rework cost
    2. Effective output = number of tasks that pass acceptance
    3. Cost-performance = effective output / total cost

    This formula explains a common paradox: if one approach leads by about ten percent on benchmarks but its unit cost is several times higher, the cost per unit of effective output may actually increase, reducing overall productivity. The value of abundant quota is not merely “being able to use more”; it allows the agent to adopt a verify-before-converge strategy: run more tests, perform more cross-checks, and these are precisely the sources of long-task quality.

    Engineering mitigation is task classification: low-risk, automatically verifiable tasks can be batch-processed by agents; high-risk, cross-system changes should have their modification scope restricted and require human approval; meanwhile, caching retrieval results and reusing completed context can reduce wasted rounds.

    6. Six evaluation dimensions for agent stability

    To bring agents into daily engineering practice, it is recommended to use a set of fixed scenarios for regression-style evaluation. The dimensions are as follows:

    Dimension Observation point Typical failure behavior
    Instruction following Whether constraints and scope boundaries are strictly respected Adding unrequested features or dependencies
    Long-horizon consistency Whether interfaces and naming remain self-consistent after multiple rounds of changes Incompatible data structures between early and later stages
    Exception handling Whether it can recover when dependencies are missing, ports are occupied, or the network fails Repeatedly executing the same failed command
    Environment awareness Whether it can understand directory conventions, framework versions, and build scripts Generating code that does not match the project’s technology stack
    Safety Whether high-risk operations require confirmation and credentials are not leaked Directly executing delete or push operations
    Recoverability Whether snapshots are preserved and the workspace can be rolled back to a usable state The workspace becomes unusable after a failure

    These six dimensions form a scorecard that better predicts an agent’s performance in real projects than any single benchmark score.

    7. Deployment recommendations and common anti-patterns

    Anti-patterns that should be actively avoided:

    • Feeding an oversized task all at once: Context and verification both go out of control, and failure rates are extremely high.
    • Not writing acceptance criteria: Reviews become subjective, and iteration cannot converge.
    • No version control in the workspace: If something goes wrong, rollback is impossible, making trial and error more expensive.
    • Handing over sensitive credentials: Residues in logs and contexts are a common leakage path.
    • Letting the agent freely decide architecture: Without constraints, it easily generates structures that “look complete but are hard to evolve.”

    Corresponding practice recommendations:

    • Commit in small steps, and make each step verifiable and rollbackable.
    • Write tests and contracts first, then let the agent implement.
    • Set human confirmation points for high-risk operations and follow the principle of least privilege.
    • Include agent output in CI: formatting checks, static analysis, and unit tests should all run automatically.
    • Preserve execution traces for retrospective analysis rather than looking only at the final result.

    8. Integrating agents into existing engineering systems

    Agent output must ultimately land in an operable system. Whether using a self-hosted cluster or public cloud infrastructure such as Baidu AI Cloud, the common approach is to use a container platform for unified delivery and scaling, object storage for static assets and artifact archiving, a log service to centrally collect agent execution traces and application logs, and monitoring/alerting with thresholds for failure rate, latency, and quota consumption. This turns “whether the agent is stable” from subjective perception into a continuously observable metric.

    Conclusion

    The key to judging a coding agent is not how elegant its single-turn generated code is, but whether it can continuously converge through feedback loops during long tasks, and whether it can turn compute into accepted output under quota constraints. Building your own evaluation method around task decomposition, verification loops, the costs of parallel orchestration, cost accounting, and the six stability dimensions is more valuable in the long term than chasing the short-term benchmark conclusions of any single product.