Background: Adapting Evaluation Methods for Autonomous Agents

Early AI coding tools functioned primarily as “completion” assistants: suggesting a few lines of code at the cursor for developers to review and accept. Evaluation during this phase was straightforward—focusing on single-generation quality and completion acceptance rates. Today, coding agents have evolved into “task executors.” Given a natural language requirement, an agent must autonomously read the codebase, decompose tasks, write code, run tests, and self-correct based on errors until delivering a runnable product.

The evaluation focus has subsequently shifted. The correctness of a single-turn generation is now just the baseline. What truly differentiates agents are four dimensions:

• Long-task iteration capability: Maintaining goal consistency across multi-step, cross-file tasks.
• Multi-agent collaboration: Orchestrating parallel or sequential sub-agents for complex compound tasks.
• Environment awareness and stability: Understanding existing engineering structures and toolchains while stably handling unexpected issues.
• Cost-efficiency: Token consumption, quota usage, and wait times required to complete identical tasks.

We will explore these dimensions using a typical scenario: building an “earnings season monitoring web application” via natural language.

Long-Task Iteration: The Agent’s “Endurance” Metric

1. Defining Long-Task Iteration

Long tasks are engineering efforts that span hours, involve dozens of files, and require multiple rounds of decision-making. Essentially, it is a “plan-execute-observe-repair” loop: the agent generates a plan, executes it step-by-step, gathers real feedback (compilation results, test outputs, runtime behavior) at each step, and adjusts subsequent actions accordingly. Compared to single-turn generation, long tasks introduce two new risks: error accumulation (small early errors amplified by subsequent steps) and goal drift (deviating from the original requirement mid-task).

2. Practical Observations

In a real-world development scenario, using only a natural language prompt, an agent directly generated a fully structured frontend application: layout, styling, and interaction modes were completed in one pass, covering almost all expectations. Although the output was temporarily frontend-only, its clear structure made subsequent incremental backend additions highly targeted—this “iterability” is a direct benefit of long-task capability.

Another standout experience was autonomous application testing: the agent started the app itself, observed actual behavior, located issues, fixed them, and re-verified, forming a complete loop. This “self-verification” capability was rare in previous tool formats.

3. Execution Loop Pseudocode

goal = parse_requirement(user_input)
plan = agent.plan(goal, context=scan_repo())
for step in plan:
    result = agent.execute(step)
    observation = run_and_observe(result)      # Compile, test, run application
    if not observation.ok:
        step = agent.repair(step, observation) # Self-correct based on real feedback
    if agent.check_goal_drift(goal, step):     # Prevent goal drift
        plan = agent.replan(goal, done=..., failed=...)

When evaluating an agent’s long-task capability, focus on three aspects: sufficiency of observation signals (whether it can actually run the app and get feedback), convergence of corrections (whether it avoids getting stuck repeating the same error), and effectiveness of drift detection (whether it consistently aligns with the original requirement).

Multi-Agent Collaboration: Swarm-Style Orchestration

1. The Ceiling of Single Agents

Compound tasks like “earnings season monitoring” encompass heterogeneous subtasks: data ingestion, document parsing, metric calculation, and visualization. Processing these sequentially with a single agent quickly dilutes the context, and a single failure can contaminate the entire workflow.

2. Three Common Orchestration Patterns

• Pipeline: Subtasks are passed sequentially; suitable for scenarios with strong dependencies.
• Fan-out/Merge: A planner decomposes the task, N executors process in parallel, and an aggregator merges the results.
• Hierarchical: A main agent holds the global goal, sub-agents manage specific modules, and validation occurs layer by layer.

3. Practice: Decomposing an Earnings Intelligence Task

Using “public company earnings intelligence monitoring” as an example, the decomposition looks like this:

task: Earnings season monitoring
subtasks:
  - name: ingest    # Data ingestion
    input: Announcement list
    output: Raw document collection
  - name: extract   # Structured parsing
    input: Raw documents
    output: Metric tables (Revenue/Net Profit/YoY/Guidance)
  - name: analyze   # Analysis and generation
    input: Metric tables
    output: Daily intelligence summary
  - name: render    # Frontend presentation
    input: Summary and metrics
    output: Web dashboard
orchestrator:
  strategy: fan-out + merge
  retry: Independent subtask retries

The benefit of a multi-agent architecture is not just parallel speedup, but fault isolation: a failure in the parsing stage does not affect ongoing ingestion, retry granularity is finer, and both overall throughput and robustness improve simultaneously.

Environment Awareness and Engineering Integration

In real-world development, most tasks do not involve building from scratch, but rather augmenting and fixing existing projects. This imposes four requirements on an agent’s environment awareness:

• Identify the tech stack: Package managers, testing frameworks, build scripts, directory conventions.
• Respect existing conventions: Code style, naming conventions, module boundaries, rather than tearing everything down.
• Handle unexpected situations: Providing workarounds autonomously for dependency conflicts, port collisions, test timeouts, etc.
• Instruction adherence: Compliance with constraints like “do not modify this module” or “maintain interface compatibility.”

The evaluation method is straightforward: pick an existing project you are highly familiar with, assign 3 to 5 fix or enhancement tasks, and comprehensively log the agent’s every step. Focus on whether it “reads before writing” and changes strategies after a failure. Agents with strong environment awareness will proactively explore the repository structure before acting, rather than guessing blindly—this behavioral difference is very easy to identify.

The Underestimated Dimension: Cost-Efficiency and Quota Economics

In the Agent era, an easily overlooked fact is that cost-efficiency sometimes impacts productivity more than model capability itself. Consider a straightforward deduction: Assume Model A’s benchmark score is 10% higher than Model B’s, but its unit cost is 100% higher. For a team running dozens of agent tasks daily, doubled costs force a shift from “running freely” to “running sparingly”—yet an agent’s value lies precisely in running freely: massive trial and error, parallel exploration, and automatic retries. Restricting API calls fundamentally restricts productivity. Recent industry debates over the pricing strategies of top-tier products stem exactly from this.

Therefore, evaluation should establish a “task-level cost” perspective rather than just looking at unit price:

• Average token consumption per task (including hidden costs of retries and self-corrections).
• Cost per unit deliverable, e.g., average spend for “generating a frontend page” or “fixing a bug.”
• Quota strategy: Whether high-priced model quotas are sufficient, and the risk of weekly/monthly rate limits truncating long tasks.
• Wait time: Wall-clock time lost due to queuing and rate limiting.

A simple tracking table:

Task	Iterations	Token Consumption	Wait Time	Completed
Frontend page generation	—	—	—	—
Bug fix in existing project	—	—	—	—
Data analysis script	—	—	—	—

Track this continuously for two weeks, and the cost-efficiency landscape will naturally emerge—more reliable than looking at any leaderboard.

An Actionable Evaluation Framework

Here is a minimum viable evaluation plan that can be completed within a week:

1. Fixed Task Set: 3 to 5 representative tasks—generating a small app from scratch, fixing an existing project, writing a data analysis script, long document comprehension and summarization, and a compound task requiring multi-step orchestration.
2. Unified Prompts: Run the same requirement descriptions across different models or tools to control variables.
3. Record Four Metrics: Completion rate, manual intervention count, wall-clock time, and task-level cost.
4. Subjective Scoring: Code readability, adherence to project conventions, and quality of error fixes.
5. Re-test Next Day: Re-run the same tasks after a few days to observe reproducibility.

Stability is often ignored but is closest to real production environments: production requires not “occasional brilliance” but “long-term reliability.”

Selection Checklist and Conclusion

Based on the analysis above, here is a selection checklist:

• Long tasks: Can it maintain goal consistency and achieve convergence in corrections for tasks lasting over 30 minutes?
• Collaboration: Does it support multi-agent orchestration? How are fault isolation and retry granularity?
• Environment awareness: Does it “read before write” and respect existing conventions on existing projects?
• Cost-efficiency: How are task-level costs, quota strategies, and the impact of rate limits on long tasks?
• Stability: Are results reproducible when the same task is re-run on a different day?

Tool selection ultimately serves engineering efficiency. Whether you choose an open-source framework paired with a model service platform’s API (such as the large model services provided by Baidu AI Cloud) or use an integrated AI coding assistant product, it is recommended to run through the evaluation process above before making a long-term commitment. After all, in the Agent era, the most expensive thing is not the model’s unit price, but the opportunity cost of an agent getting “stuck” on a critical task.