Evaluating Coding Agents: Evaluation Methods and Engineering Practices

1. Why Coding Agent Evaluation Needs a Fresh Look

In the past, code models were often evaluated by asking them to complete a function and then counting how many unit tests passed. That approach measures single-turn generation ability. Coding agents work differently: they need to read a code repository, run commands, modify files based on error messages, and verify the result again. The whole process may last dozens of steps. A high single-turn score does not mean the agent can reliably complete tasks in a real terminal.

Evaluation should therefore shift from “writing a correct piece of code” to “completing a full task in a controlled environment.” Before selecting or deploying a solution, teams need to answer three questions: Can the model perform multi-step operations in a real environment? Can it maintain its objective and state over long-running tasks? Are the cost and latency of a single task acceptable to the business?

2. Breaking Down the Evaluation Dimensions

2.1 Environment Realism

The evaluation environment should be close to production. It should include a complete file system, installable dependencies, executable shell commands, and error messages consistent with those from real projects. If the environment is too idealized, test scores will be higher than actual performance.

2.2 Long-Horizon Task Ability

Long-horizon tasks test planning, memory, and recovery. The main points to examine are:

• Whether the agent can break a large task into executable substeps;
• Whether it can locate the cause of an error in an intermediate step and roll back;
• Whether it still remembers initial constraints after the context grows longer, such as directories that must not be modified.

2.3 Multimodal Understanding

Some tasks require reading screenshots, interface images, or charts. Visual-input tasks should be counted separately during evaluation so that visual ability and coding ability are not mixed into a single score.

2.4 Cost and Latency

In addition to success rate, record the number of input and output tokens consumed by each task and the end-to-end elapsed time. These two metrics directly determine the cost of running the system at scale.

3. Building a Reproducible Evaluation Workflow

A single run can be heavily affected by randomness. The environment should therefore be fixed, runs should be repeated, and complete traces should be saved. The example code below shows a minimal evaluation framework. Its interfaces are hypothetical and must be replaced with those of the agent framework in use.

from dataclasses import dataclass
@dataclass
class TaskCase:
    task_id: str
    instruction: str
    workspace: str
    checker: str  # Path to the deterministic acceptance script
def run_trial(agent, case, sandbox, max_steps=50):
    """Run one task in a fresh sandbox and return the result."""
    env = sandbox.create(workspace=case.workspace)
    steps = 0
    try:
        for _ in range(max_steps):
            action = agent.next_action(case.instruction, env.observe())
            if action.type == "finish":
                break
            env.execute(action)
            steps += 1
        passed = env.run_checker(case.checker)
    finally:
        env.destroy()
    return {"task_id": case.task_id, "passed": passed, "steps": steps}
def evaluate(agent, cases, sandbox, repeats=5):
    """Run each task repeatedly and return the success rate for each task."""
    report = {}
    for case in cases:
        runs = [run_trial(agent, case, sandbox) for _ in range(repeats)]
        report[case.task_id] = sum(r["passed"] for r in runs) / repeats
    return report

Three points matter in practice:

• Use a fresh sandbox for every run so that files left by a previous run do not affect the result;
• Let a deterministic program determine acceptance rather than asking the model to evaluate its own result;
• Save complete traces so that failed samples can be replayed, making it easier to distinguish model capability limitations from environment configuration problems.

4. Cautions for Interpreting Leaderboard Scores

Public leaderboards provide useful references, but ranking solutions directly by score can easily lead to mistakes.

1. Task distributions differ. Different benchmarks cover very different task types. A high score on one benchmark does not mean strong performance in every business scenario.
2. Data contamination may exist. If test tasks have already appeared in training data, scores will be inflated. During selection, teams can add internal task sets that have not been publicly released.
3. Results fluctuate. The same model may show success rates that differ by several percentage points across multiple runs. Report the mean and standard deviation rather than a single result.
4. Evaluation settings affect results. Settings such as whether tool calls are allowed, the step limit, and the sampling temperature can all change outcomes. Comparisons must use consistent settings.

Anonymized comparison testing is a common practice. The systems under test are not labeled with their source, and evaluators score only based on output quality. This reduces judgment bias caused by brand preferences. Teams can borrow this approach for internal model selection.

5. Cost Accounting: Total Parameter Count Does Not Equal Per-Run Cost

Mixture-of-experts (MoE) models have a large total parameter count, but each inference activates only a subset of the expert networks. The computation for a single inference depends mainly on the activated parameters, not the total parameters. Therefore, a model with a large total parameter count does not necessarily have a high per-call cost.

Parameter structure is only one factor in cost. Actual expense can be estimated as follows:

Task cost = input tokens × input unit price + output tokens × output unit price, multiplied by the average number of calls per task.

When estimating cost, note the following:

• Agents call models repeatedly, and input tokens continue to accumulate across multiple rounds. The actual number of calls is usually much higher than in a single question-and-answer interaction;
• Output speed determines user waiting time. For batch tasks, throughput can be used instead of single-request latency for evaluation;
• A larger context window can hold more information at once, but long inputs are also billed. Input size should be controlled through retrieval and summarization.

6. Engineering Practices for Long Context

A context window measured in millions of tokens allows a model to read a large code repository at once. However, putting the entire repository directly into the input is not always optimal. Common practices include:

• Split by module and load only the files relevant to the current task;
• Use code search or vector retrieval to recall relevant snippets, then pass them to the model;
• Summarize historical conversations, preserving the objective, constraints, and completed steps.

Long context is suitable for global-understanding tasks, such as tracing call relationships and checking interface consistency. For local modification tasks, retrieval-based approaches are more economical.

7. Risk Controls Before Launch

Coding agents can execute commands, so permission design requires particular care.

• Least privilege: The agent can access only the directories required for the task and is not given production credentials;
• Sandbox execution: Commands run in isolated containers, with network access opened only according to an allowlist;
• Human review: Operations involving database changes, dependency upgrades, permission adjustments, and similar actions must be confirmed by a human;
• Monitoring and alerting: Record every operation through a logging service and set alerts for abnormally high call frequency and sudden increases in failure rate.

Gray release is also important. First let the agent handle low-risk tasks such as supplementing documentation or unit tests. After enough operational data has been accumulated, gradually expand the scope.

8. A Suggested Adoption Path for Teams

1. Identify the five to ten most common task types in the team and form an internal task set;
2. Use the framework above to run a baseline in a unified environment, recording success rate, cost, and elapsed time;
3. Compare candidate solutions under the same conditions, focusing on the distribution of failure sample types;
4. Determine the applicable scope based on the results and retain human involvement for high-risk operations.

The purpose of evaluation is to serve specific business needs and determine whether a solution can work reliably on the team’s real tasks.