AI Coding Agent Evaluation: Capability Breakdown and a Reproducible In-House Assessment Method

1. Why It Is Difficult to Answer “Which AI Coding Agent Is Better?”

“Which AI coding agent is better?” is one of the most frequently asked questions, yet it is also one of the hardest to answer credibly. The difficulty is not caused by the question itself, but by the fact that the answer depends on at least three types of variables:

1. Model capability: code understanding, instruction following, stable tool-call formatting, and effective use of information in long contexts.
2. Framework engineering, often called the harness: how context is organized and compressed, how retrieval is performed, which execution tools are available, how retries and validation loops are handled after failures, and where permission and approval boundaries are enforced.
3. Task distribution: maintaining an existing project, creating a new project from scratch, fixing a single bug, and refactoring across modules all place very different demands on the system.

If any one of these variables changes, the result of a side-by-side comparison can be reversed. The same model may perform noticeably differently on long-file modification tasks when paired with a different context-management strategy. The same framework may fail on multi-file refactoring tasks when paired with a model that has a smaller effective context window. Therefore, the right approach is not to ask “Which one is strongest?” but to run reproducible controlled experiments under a fixed task set and fixed framework constraints.

2. Capability Breakdown: Six Dimensions for Evaluating a Coding Agent

2.1 Effective Context, Not Just Context Window

Context window length is a nominal specification. What matters in practice is effective context: whether the model can reliably locate the relevant symbols, type definitions, and call sites when the input contains tens of thousands or even hundreds of thousands of tokens.

Common failure modes include:

• Missing a call site in a file outside the immediate edit scope.
• Losing critical constraints after context compression or summarization.
• Confusing variables or functions with similar names.

A direct way to test this is to prepare tasks that require cross-file reasoning, such as changing an interface signature referenced by multiple files. Observe whether the agent proactively searches for call sites. If it edits only the current file and declares the task complete, its context organization strategy or retrieval tooling likely has a gap.

2.2 Stable and Recoverable Tool Calling

The value of a coding agent comes from its closed loop:

read files → modify code → run build → read errors → correct code.

This loop is highly sensitive to the stability of structured output. A single invalid argument, a path escape, or a malformed multi-line patch can cause an entire task round to fail. When evaluating an agent, you should track:

• Tool-call success rate
• Number of retries caused by formatting errors

rather than only checking whether the final task was completed.

2.3 Planning and Self-Correction for Long-Horizon Tasks

Multi-step tasks require the model to maintain an internal plan and verify preconditions at each step.

Stronger behavior looks like this:

• Run tests first to establish the baseline failure point.
• Make the smallest necessary change.
• Validate again through regression testing.

Weaker behavior looks like this:

• Perform a large-scale rewrite.
• Run the full test suite only once.
• Fail to localize the problem when the tests fail.
• Enter an unstable loop of “change → error → change again.”

2.4 Controllability and Safety

An agent that can execute arbitrary commands in a terminal has a very different risk profile from a chat assistant. At least three issues should be confirmed:

1. Whether dangerous operations, such as deleting files, force-pushing, or modifying infrastructure configuration, require human confirmation.
2. Whether working directories and credentials are properly bounded.
3. Whether the agent can run in a container or a disposable environment.

These are engineering admission requirements. If they are not satisfied, the agent should not be allowed to operate directly on production code repositories.

2.5 Engineering Integration

Engineering integration includes:

• Whether the agent can run consistently in IDEs, terminals, and CI pipelines.
• Whether it can connect to internal documentation and code search systems.
• Whether it supports team-level policy controls, such as disabling certain commands or limiting writable paths.
• Whether session records are auditable.

Integration level often has a greater impact on real adoption than raw model scores.

2.6 Cost and Throughput Characteristics

Cost and throughput characteristics include:

• Token consumption per task.
• Support for prompt caching.
• Concurrency limits.
• Degradation behavior when rate limits are reached.

These factors determine whether the agent can be used for daily development across an entire team.

A cheap but unpredictable agent may end up costing more than an expensive agent that completes the task correctly on the first attempt, because rework and review time are also costs.

3. Build an In-House Evaluation: A Reproducible Assessment Pipeline

3.1 Build Your Own Task Set

Do not rely on public leaderboards. Their task distributions are often too different from your own codebase. A more reliable approach is to mine tasks from version-control history.

Select change records that satisfy both conditions:

• Tests failed before the change.
• Tests passed after the change.

Remove changes that only contain formatting updates or dependency upgrades. Keep around 50–200 tasks as your evaluation set. These tasks naturally come with automatically verifiable success conditions.

For each task, record:

• Task description: abstracted from the commit message, without solution hints.
• Files involved: used only for statistics, not as input to the agent.
• Validation command and expected result
• Difficulty label: single-file fix, cross-file change, or new feature.

3.2 Metric Design

A single “pass rate” is not enough for decision-making. At least five metrics should be collected:

Metric	Meaning	Purpose
First-pass rate	Percentage of tasks that pass validation on the first submission	Measures the combined strength of the model and framework
Average interaction rounds	Number of tool calls or conversation rounds required to complete a task	Reflects planning efficiency
Cost per task	Token usage and cost for completing or abandoning a task	Used for cost modeling
Human intervention rate	Percentage of tasks requiring human correction or additional writing	Reflects real usability
Rollback rate	Percentage of generated changes that are reverted	Reflects code quality and controllability

3.3 Isolate Variables

Controlled experiments must keep all variables fixed except the one being tested:

• The same toolset
• The same system prompt template
• The same decoding parameters
• The same temperature setting
• The same task order

Each task should be sampled 3–5 times, and you should record the distribution rather than a single point result. Coding tasks have high variance, and differences from a single run usually lack statistical meaning.

3.4 Minimal Evaluation Script Skeleton

The following example shows the key elements of an evaluation loop: forced validation, a retry limit, and instrumentation for cost and rounds.

from dataclasses import dataclass, field
@dataclass
class TaskResult:
    task_id: str
    passed: bool
    rounds: int = 0
    prompt_tokens: int = 0
    completion_tokens: int = 0
    needed_human: bool = False
    log: list = field(default_factory=list)
def run_one(agent, task, max_rounds: int = 12) -> TaskResult:
    """Execute one task with a closed loop where external tests are the only judge."""
    res = TaskResult(task_id=task.id)
    agent.reset(repo_snapshot=task.base_commit)   # Start every task from the same baseline.
    for _ in range(max_rounds):
        res.rounds += 1
        step = agent.step(instruction=task.description)
        res.prompt_tokens += step.usage.input
        res.completion_tokens += step.usage.output
        if step.tool_name == "run_tests" and step.passed:
            res.passed = True
            break
        if step.tool_name == "ask_human":
            res.needed_human = True
            break
    if not res.passed:
        # Important: clean up the environment to avoid cross-task contamination.
        agent.reset(repo_snapshot=task.base_commit)
    res.log = agent.transcript()
    return res
def aggregate(results: list[TaskResult]) -> dict:
    n = len(results) or 1
    return {
        "pass_rate":      sum(r.passed for r in results) / n,
        "avg_rounds":     sum(r.rounds for r in results) / n,
        "human_rate":     sum(r.needed_human for r in results) / n,
        "avg_in_tokens":  sum(r.prompt_tokens for r in results) / n,
        "avg_out_tokens": sum(r.completion_tokens for r in results) / n,
    }

Two easily overlooked details are important:

• Each task must start from the same baseline snapshot. Otherwise, leftover changes from the previous task will contaminate subsequent results.
• Validation must come from an external script, not from the model’s self-report. If you let the agent declare “completed” on its own, hallucinations will be counted as successes.

4. Four Key Priorities in Harness Engineering

If the goal is to build or improve a coding agent, the following four areas usually provide the highest marginal return:

1. Context layering and on-demand loading.
Do not fill the context with the entire repository at once. First use symbol indexes or retrieval to obtain candidate files, then expand them as needed. For early interactions, create structured summaries and retain a list of decisions and constraints to avoid losing requirements late in a long session.

2. Enforce a validation loop.
Make “build and tests must pass” part of the loop termination condition rather than relying only on prompt instructions. For patch-style tasks, also add minimal-diff constraints to prevent unrelated changes from entering code review.

3. Task decomposition and subagents.
For goals beyond a single round of capability, such as migrating across ten modules, first produce a reviewable implementation plan, then execute the work in slices and validate each slice independently. Pass constraints between slices through shared context summaries.

4. Observability.
Record the input arguments, return values, latency, and token usage of every tool call. Connect these logs to logging services, monitoring, and alerting systems, and build trend dashboards for failure rate, average rounds, and unit cost. This data supports both evaluation and online troubleshooting. If your applications and model services already run on Baidu AI Cloud, you can reuse its general capabilities, such as logging, monitoring, alerting, and container platforms, to collect this data without building a separate observability pipeline.

5. Cost Governance: Free Credits Can Distort Usage Behavior

Operational mechanisms such as free trial periods, subscription plans, and bonus credits can significantly change developer usage patterns. When credits are abundant, teams tend to let the agent “run the entire module by itself.” When credits become tight, they switch to “ask for a plan first, then execute manually.” This has two engineering implications:

• Do not infer long-term cost from trial-period experience. During evaluation, record task completion behavior under both “no credit pressure” and “with a budget cap” conditions.
• Implement budget controls on the agent side. Set round limits and token limits for each task. When the limit is exceeded, degrade the behavior to “output suggested patches instead of executing writes directly,” returning decision authority to the developer.

class TokenBudget:
    """Per-task budget guardrail that degrades instead of aborting."""
    def __init__(self, max_input=400_000, max_output=120_000):
        self.max_input, self.max_output = max_input, max_output
        self.used_input = self.used_output = 0
    def charge(self, usage) -> str:
        self.used_input += usage.input
        self.used_output += usage.output
        ratio = max(self.used_input / self.max_input,
                    self.used_output / self.max_output)
        if ratio >= 1.0:
            return "degraded"      # Only output suggestions; do not perform write operations.
        if ratio >= 0.8:
            return "throttled"     # Compress context and reduce retrieval scope.
        return "normal"

6. Selection and Rollout Checklist

• Use 30–50 tasks from your own codebase for the first round of screening instead of making decisions based on public leaderboards.
• Treat safety boundaries as admission requirements: confirmation for dangerous commands, path restrictions, and execution sandboxes. Satisfy these before evaluating performance.
• Keep the framework and parameters fixed during evaluation. Change only one variable at a time and sample each task multiple times.
• Focus on the combination of first-pass rate, human intervention rate, and cost per task rather than a single score.
• Run a small pilot: have a team of 3–5 people try the agent for two weeks, record real rollback rate and review burden, and then decide whether to expand usage.
• Build a continuous evaluation pipeline. Connect the task set to CI so that any model or prompt change automatically runs a regression evaluation.

Conclusion

Whether one agent is “better” than another depends heavily on your team’s codebase characteristics, task types, and cost constraints. The better question is not “Which one is strongest?” but:

Under our task distribution, given our cost and risk constraints, which combination of model and framework achieves the highest first-pass rate and the lowest human intervention?

Then validate the answer with a reproducible evaluation process.

The gap between models and frameworks is narrowing quickly. At this stage, a team’s most important differentiator is whether it has its own evaluation set and observability data.