AI Coding Agents: An Evaluation and Paid-Tool Selection Methodology

1. Get the question right: not “who to pay for,” but “along which dimensions”

The discussion of which AI coding tool to adopt is usually pulled in two directions: vendor-published benchmark scores, and the monthly subscription price. Both happen to be the least reliable inputs for that decision.

Before committing to any conclusion, fix five categories of constraints first. Without them, benchmark results are not comparable:

• Size of the code asset. A 10k-line monorepo and a multi-repo estate measured in millions of lines impose requirements on context retrieval that differ by an order of magnitude.
• Languages and framework stack. Symbol resolution quality in mainstream statically typed languages is usually clearly better than in niche stacks.
• Distribution of task types. Single-site bug fixes, cross-module feature development, test backfilling, dependency upgrades, and log-based troubleshooting differ enormously in difficulty structure.
• Failure cost the process can absorb. Whether auto-opening merge requests is acceptable, or whether every change must be reviewed by a human.
• Data and compliance boundaries. Whether source code, secrets, and production logs are allowed to leave your own network.

Three upfront conclusions, each argued in the sections that follow:

1. Capability convergence happens mainly at the shallow end. Single-file completion and standard problem solving are where the gap among leading products has already become small.
2. Real differences concentrate in four places. Localization accuracy under long context, tool-use strategy, stability and checkpoint resume on long-running tasks, and reviewability of the change set.
3. Cost differences rarely come from unit price. They come from the implicit amplification of failures and retries. The same model on a differently built harness can differ 2–3x in total cost.

2. Breaking “convergence” into a three-layer capability model

Comparing “who is smarter” in the abstract has no engineering value. Decompose the comparison into three layers, each with metrics you can actually measure.

2.1 Context layer: can it find the right place?

The core issue is not how large the window is, but whether the right information gets placed inside a limited context budget. This layer covers: codebase indexing and hybrid retrieval (keyword search + vector search + symbol index), cross-file call-chain resolution, use of build files and dependency manifests, and context compression and summarization strategy.

Measurable metric: localization hit rate — given a task description, does the initial set of files the agent reads contain the locations that genuinely need to change. In cross-repo refactoring work, this metric usually explains final success rate better than the model’s general reasoning ability does.

2.2 Execution layer: can it finish on its own?

This is the so-called agent loop: tool-set design (read file, write file, run command, search, run tests), loop termination conditions, self-repair on errors, parallel subtasks, and the ability to resume after a long task is interrupted.

The key design tradeoff is the termination condition. Terminating too early delivers a half-finished change; terminating too late keeps burning budget down a wrong path. In practice, three hard constraints usually need to be in effect simultaneously: maximum turns, wall-clock time, and token budget.

Measurable metrics: unattended completion rate, average turns, timeout rate, error self-repair success rate.

2.3 Governance layer: can it be accepted with confidence?

This layer is the easiest to overlook, yet it decides whether a tool can enter the production workflow: whether change granularity is easy to review, whether it can integrate with pipeline gates, whether permissions are least-privilege, whether operations are logged, and whether cost is observable.

An agent with strong capability but hard-to-review output can have a net negative return in a real team: the time a reviewer spends reading the diff eats up the time saved by generated code.

3. Why benchmark scores cannot be compared directly

A common mistake in these discussions is putting two vendors’ published scores side by side. At least four problems make that comparison meaningless:

• Different question sets. One may be a small, manually verified subset; the other a larger set that sits closer to real repositories. The denominators differ, the difficulty distributions differ, and the numbers have no shared scale.
• Saturation and contamination. Once problems enter public corpora, there is leakage risk between training and evaluation; vendor self-evaluation also carries sampling bias.
• Harness effect. Scores are strongly influenced by the evaluation framework — whether regression tests are provided, how many attempts are allowed, what the timeout is, whether builds may be run. Switching the same model to a different harness often changes the score by more than the gap between models.
• Task distribution mismatch. Public benchmarks are dominated by “an isolated commit fixes a single defect,” while real enterprise tasks are “cross-module feature development + compatibility with legacy code + passing an existing test suite.” The former only requires looking at something local; the latter requires understanding global constraints.

So there is exactly one correct use of these scores: to screen out clearly unqualified candidates, never to make the final selection.

4. Building your own evaluation: the only trustworthy answer

4.1 A six-step process

Task sampling → environment freezing → unified harness → metric computation → manual review → regression-set maintenance.

Sampling has to stay close to reality: pick 50–100 merged changes from historical merge requests, keep the original repository snapshot and the hidden tests, and drop entries that involve sensitive information. Freezing the environment means every task runs in an identical container snapshot with dependency versions pinned, so that “it works on my machine” cannot pollute the conclusions.

4.2 Task spec example

# eval/task_0007.yaml
id: "TASK-0007"
repo_snapshot: "svc-billing@sha:a1b2c3d"
goal: "Add proportional discounting to the billing service without changing existing reconciliation logic"
constraints:
  - "Must not change external interface signatures"
  - "Old configuration files must still load"
budget:
  max_turns: 25
  wall_clock_s: 900
  token_limit: 1_200_000
verifier:
  setup: "./scripts/bootstrap_env.sh"
  cmd: "./scripts/verify.sh"
  must_pass: ["unit", "integration", "lint"]
grading:
  hidden_tests: true
  forbid_editing_tests: true

The forbid_editing_tests line matters especially. Without it, a common shortcut for the agent is to modify assertions until tests pass, producing an artificially inflated pass rate.

4.3 Unified scoring harness example

# eval/harness.py
import json, subprocess, time
from pathlib import Path
def count_turns(trace_path: Path) -> int:
    return sum(1 for line in trace_path.read_text().splitlines()
               if json.loads(line).get("event") == "tool_call")
def run_case(spec: dict, runner_cmd: list[str]) -> dict:
    t0 = time.time()
    subprocess.run(runner_cmd + ["--task", spec["id"]], check=False)
    wall = time.time() - t0
    v = subprocess.run(spec["verifier"]["cmd"], shell=True,
                       capture_output=True, text=True)
    trace = Path(f"traces/{spec['id']}.jsonl")
    usage = sum(json.loads(l).get("tokens", 0)
                for l in trace.read_text().splitlines())
    return {
        "id": spec["id"],
        "resolved": v.returncode == 0,          # all hidden tests pass
        "turns": count_turns(trace),
        "tokens": usage,
        "wall_clock_s": round(wall, 1),
        "within_budget": wall <= spec["budget"]["wall_clock_s"],
    }

4.4 Metric definitions

Metric	Definition	Why it matters
First-attempt pass rate	Share of tasks where hidden tests pass on the first round	Measures real usability
Unattended completion rate	Share of the full run with zero human intervention	Determines whether automation is possible
Within-budget completion rate	Completes without timeout or budget overrun	Reflects execution efficiency
Human rework time	Minutes of review and patching per merge request	A net cost item that is often ignored
Rollback rate	Share of merged changes later reverted or causing regressions	Risk control
Cost per effective delivery	Total cost ÷ number of changes actually merged	The only metric that should drive the decision

Pay attention to the last row. If one option doubles output volume but also doubles the rework rate, cost per effective delivery may not change at all — while team fatigue rises noticeably.

5. Cost model: unit price is the least important variable

An estimate that ignores amplification effects looks like this:

task cost = input tokens × input price + output tokens × output price

A model closer to reality:

task cost = (input + output) × unit price × parallel amplification factor × retry amplification factor
          + manual review cost (minutes × loaded labor rate)

Three main sources of amplification:

• Cache hit differences. Repeatedly injected system prompts, repository summaries, and standards documents cost significantly less in long-context scenarios when they can be reused from cache; when they cannot be cached, every call recomputes them at full price.
• Repeated context injection. Every turn of the loop carries history and code snippets, so input tokens accumulate roughly linearly over a long task, while output accounts for only a small share.
• Parallel and retry amplification. Having the agent explore several fix paths simultaneously consumes budget in multiples; a 30% failure-retry rate means multiplying by another 1.3.

A rough example: a single task with 800k input tokens (about 70% of which is cacheable), 60k output tokens, a three-way parallel search producing roughly 2.4x amplification, plus a 30% retry rate adding another 1.3x. That means at identical unit prices, a poorly designed harness can cost more than twice as much per task as a well-designed one. So an evaluation must record both token consumption and actual cost, not just whether the task passed.

6. Form factor: four delivery models, each with its own home turf

Form factor	Strengths	Weaknesses	Good fit
In-editor inline completion	Low latency, does not break flow	Weak at cross-repo changes	Day-to-day coding, boilerplate
Terminal CLI agent	Long tasks, scriptable, orchestratable	Needs permission and sandbox governance	Bug fixing, bulk refactoring
Asynchronous pipeline integration	Batch-capable, parallel, gate-enabled	Long feedback cycle	Dependency upgrades, test backfilling
Cloud-hosted agent	No local environment, elastic scaling	Data egress needs assessment	Large one-off remediation programs

The steady state for most teams is a combination of the first three — in-editor completion plus a terminal agent plus pipeline integration — rather than a binary choice between two. Routing work by task type is cheaper than chasing one all-purpose tool.

7. Security and compliance checklist

• Data boundary. Whether source code, secrets, and production logs leave your own network; whether local indexing and redaction policies are configurable.
• Least privilege. Read-only by default, writes confined to a sandboxed workspace, explicit confirmation for destructive commands.
• Supply-chain security. Third-party dependencies introduced by generated code must still go through your existing provenance checks and license compliance scans; being “generated by an agent” is not an exemption.
• Audit trail. Retain prompts, tool-call traces, and the final diff; attribute changes to a specific session so they remain traceable and reversible.
• Observability. Token consumption, turn distribution, failure rate, and timeout rate should be collected and alerted on as routine metrics.

If you require a fully closed loop inside your own environment, you can host the evaluation and execution sandboxes on Baidu AI Cloud’s container platform, archive task traces and diffs in object storage, and track cost and quality metrics through its logging service and monitoring alerts, connecting model capabilities as needed. Evaluation, governance, and auditing then share one infrastructure stack instead of each tool building its own data channel.

8. Avoiding lock-in: treat the agent as a replaceable component

Capability shifts every six months, and any workflow hard-coded to one tool becomes debt within a few months. The approach is to define your own abstraction interface and let each tool be an adapter:

from typing import Protocol
class Budget:
    max_turns: int
    wall_clock_s: int
    token_limit: int
class RunResult:
    changed_files: list[str]
    turns: int
    tokens: int
    cost: float
    trace_path: str
class AgentRunner(Protocol):
    def run(self, task: str, repo: str, budget: Budget) -> RunResult: ...

Three conventions go with it: a single task spec format (as in 4.2), a single trace format, and a single verification entry point. When you swap tools, you swap only the adapter implementation; the evaluation set and the gate rules stay untouched.

9. A four-phase rollout

1. Weeks 1–2. Connect one or two candidate options, use them only for completion and single-file tasks, collect baseline data, and keep them out of the mainline flow.
2. Weeks 3–6. Build a 50–100 item in-house regression set, focus on unattended completion rate and human rework time, and produce a reproducible report.
3. Month 2. Integrate with pipeline gates: only let high-confidence tasks open merge requests automatically, and route the rest to human triage.
4. Every quarter. Re-run the same regression set. Models and harnesses both evolve, so historical scores go stale through capability drift and must be re-measured periodically.

10. Conclusion

Benchmark scores are an entry ticket, not a decision basis; subscription price is noise, not cost. The goal worth optimizing is: the lowest total cost per effective delivery on your own task distribution, with output that is reviewable, auditable, and reversible.

Build the evaluation set first, then discuss selection. Invert that order and any conclusion is just one quarter of luck.