Evaluating Coding Agents: Long Tasks, Multi-Agent Systems, and Cost-Effectiveness

1. Background: Evaluation Standards in the Agent Era Are Changing

In the past, evaluating a Large Language Model (LLM) typically meant looking at its public benchmark scores. However, as Agents become deeply involved in daily software development, the explanatory power of a single benchmark score is declining. A model that scores 10% higher on a benchmark but costs twice as much per unit might actually result in a net decrease in productivity when deployed in a real production environment. The reason is straightforward: Agents operate in a “multi-turn, long-horizon, high-frequency invocation” mode. Token consumption far exceeds that of one-shot Q&A scenarios, meaning price elasticity is multiplied exponentially.

Consequently, an increasing number of teams are shifting their evaluation focus to four questions that better reflect real-world engineering scenarios:

• Can it stably execute and self-repair during a multi-step, hours-long task?
• Can it produce a structurally complete, directly runnable application skeleton from a single-sentence requirement?
• Can it process splittable batch tasks in parallel via multi-agent collaboration?
• What is the comprehensive cost per unit of effective output (rather than per unit token)?

The following sections will explore these four questions one by one.

2. Highlight 1: Long-Horizon Task Iteration Capability

2.1 From “One-Shot Generation” to “Closed-Loop Iteration”

The typical workflow of early coding assistants was: the developer provides context, the model outputs code in one shot, and if it fails, manual rework is required. The key differentiator of the new generation of Agents is the formation of a closed loop: “Generate -> Execute -> Observe Results -> Correct -> Re-execute”. To determine if an Agent possesses true long-task capability, observe three signals:

1. Self-Feedback: Upon execution failure, can it autonomously read error messages, locate the cause, and adjust its approach, rather than simply throwing the error back to the user?
2. Task Memory: Across multiple rounds of modifications, can it maintain consistency regarding project structure and existing conventions (naming styles, directory norms)?
3. Test-Driven: Can it proactively run tests, or even write its own acceptance test cases, using “tests passed” as the completion standard rather than “code written”?

2.2 A Scenario Worth Testing Independently: Legacy Project Enhancement

Compared to “building a demo from scratch,” a test with greater engineering value is placing the Agent into an existing project and requiring it to perform local refactoring, bug fixing, or feature enhancement. This type of task evaluates its understanding of existing code, its awareness of minimizing changes, and regression safety. Practical observations indicate that some next-generation Agents perform significantly better in the “automatically running application tests and continuing to fix based on results” phase than earlier tools. The number of manual interventions has dropped notably, which is directly related to the maturity of the closed-loop design.

2.3 Reference Implementation of the Iteration Loop

During evaluation, you can build a minimal Agent loop to observe its behavior pattern:

# Minimal Agent iteration loop (illustrative)
while not task.done and steps < MAX_STEPS:
    plan = agent.propose(task, memory)
    result = sandbox.run(plan.commands)   # Execute in an isolated sandbox
    feedback = result.stdout + result.stderr
    if result.exit_code == 0 and run_tests():
        task.done = True
    else:
        memory.append(reflect(feedback))  # Self-reflection, correct the next step

Focus on observing three metrics: convergence steps, repeated error rate, and whether oscillation occurs (where fixes make things worse).

3. Highlight 2: 0-to-1 Generation of Runnable Application Skeletons

3.1 How to Design the Test Task

A highly practical test task is providing a natural language requirement (e.g., “Build a dashboard for tracking key business metrics, including sample data, filtering, and alerts”) and having the Agent directly construct the application on the client or web side. You can then score it across four dimensions:

• Requirement Coverage: Are all requested features implemented? Is there any cutting corners?
• Style and Interaction Quality: Is the visual completeness and interaction pattern logical?
• Runnability: Does it work out of the box, or require extensive manual patching?
• Structural Soundness: Does the component division and state management leave clear interfaces for subsequent backend integration?

3.2 Considerations for Horizontal Comparison

When conducting horizontal comparisons, it is recommended to keep three things constant: the same requirement description, the same scoring rubric, and the same maximum interaction turns. Additionally, pay attention to two easily overlooked cost factors: First, session resource consumption—some models consume an abnormally high amount of quota on a single task, even triggering rate limits and causing task interruption. Second, style drift—does the output style diverge after multiple follow-up questions? Both factors directly impact the judgment of “whether it can be safely handed over for long-horizon tasks.”

3.3 Frontend-First, Backend-Follow Collaboration Mode

Even if the current output only covers the frontend, as long as the skeleton quality is high enough, subsequently having the Agent enhance the backend will be more targeted: the API contract is implicitly defined in the frontend calls, and the backend only needs to be filled in according to that contract. This is an efficient path for “human-machine collaborative development”—humans handle acceptance and direction, while the Agent handles implementation and iteration.

4. Highlight 3: Multi-Agent Collaboration

4.1 Division of Labor: Single Agent vs. Cluster Mode

A single Agent is suitable for continuous tasks with strong context dependencies. However, when naturally parallelizable batch tasks arise (e.g., “summarize key operating indicators from this period’s financial reports of various listed companies”), the value of cluster mode becomes apparent: an orchestrator decomposes the task, distributes it to multiple sub-agents for parallel execution, and finally aggregates the results. The benefits are not just speed, but also fault isolation—a single sub-task failure does not affect the overall system.

4.2 Architecture Diagram

            ┌─────────────┐
 Request ─▶ │  Orchestrator│
            └──────┬──────┘
   Split/Dispatch  │  Aggregate
      ┌────────────┼────────────┐
      ▼            ▼            ▼
 ┌────────┐  ┌────────┐  ┌────────┐
 │SubAgent1│  │SubAgent2│  │SubAgent3│  ← Parallel execution, independent context
 └────────┘  └────────┘  └────────┘

The orchestration logic can be expressed with minimalist pseudocode:

subtasks = orchestrator.split(requirement)
results = await asyncio.gather(*[worker.run(t) for t in subtasks])
report = orchestrator.merge(results)

When evaluating cluster capabilities, focus on: whether the task decomposition granularity is reasonable, whether the result formats of sub-agents are stable and aggregatable, and whether the total cost increases or decreases compared to single-agent serial execution.

5. Cost-Effectiveness: The Underestimated Core Metric

In Agent scenarios, cost-effectiveness is sometimes more critical than the model’s raw capability. A simple but effective measurement method is:

1. Fix a set of representative tasks (several each for 0-to-1 generation, legacy fixes, long tasks, and batch aggregation);
2. Record the iteration rounds, total token consumption, number of manual interventions, and final quality score for each task;
3. Calculate the “comprehensive cost per unit of effective output” rather than just looking at the unit price.

In practical observations, some products provide rather generous quotas; even after a week of heavy usage, the weekly consumption remains low, achieving an experience of “basically not worrying about limits at flagship-tier capability.” Others show astonishing consumption on a single task. If a task exhausts the quota on an expensive model but fails to complete, yet smoothly converges on another model of comparable capability, then a few percentage points of benchmark lead is meaningless in engineering terms.

6. An Actionable Evaluation Process

Converging the previous content into a checklist, teams can directly execute as follows:

1. Build a Task Set: Cover four categories: 0-to-1 generation, legacy project repair, cross-hour long tasks, and parallelizable batch tasks;
2. Define Metrics: First-pass success rate, average iteration rounds, manual intervention count, requirement coverage, total token cost, task interruption rate;
3. Specialized Stress Testing: Stability (whether drift occurs during continuous long runs), instruction adherence (whether complex constraints are followed), exception handling (recovery capability during dependency missing or network failures), environment perception (accuracy in recognizing the operating environment such as OS and dependency versions);
4. Logging and Replay: Retain complete execution traces to support issue review and regression comparison;
5. Build Infrastructure: Use containers to provide isolated sandboxes, and use logging services with monitoring and alerting to track the consumption and results of each run. The evaluation pipeline can be built entirely on general-purpose cloud capabilities (such as the container platforms, logging services, and monitoring/alerting provided by Baidu AI Cloud) without relying on specific proprietary components.

7. Conclusion

The focus of the new generation of coding Agents is shifting from “higher benchmark scores” to “more stable closed loops, stronger collaboration, and better unit cost.” For developers, the most valuable approach is not chasing every new release, but establishing an evaluation process of your own: use real tasks to test long-horizon iteration, use parallelizable tasks to test multi-agent collaboration, and use a fixed task set to calculate comprehensive costs. Once the evaluation has a unified yardstick, technology selection naturally has a solid foundation.