AI Coding Agents Under Test: A Four-Dimensional Evaluation Framework and Selection Methodology

1. The Marginal Return on Benchmark Chasing Is Approaching Zero

Over the past year, leaderboards for AI coding tools have been refreshed almost monthly: one client takes the top spot, another overtakes it a week later, and a new combination breaks through soon after. For developers, the direct consequence of this information noise is that selection decisions get hijacked by benchmark scores — and benchmark scores are precisely the metric that saturates fastest and distorts most easily.

There are three reasons:

1. The leaders have converged. In aggregate evaluations covering real defect repair, terminal operations, and large-codebase comprehension, the score gap among the top three can compress to under 1 point. When measurement error itself (task sampling, run-to-run randomness, timeout retries) is already at that magnitude, a 1-point difference does not constitute a statistically meaningful advantage.
2. Scores do not correlate with R&D returns. Benchmark tasks are usually isolated snapshots of repositories with carefully trimmed context. Real engineering work is full of cross-service dependencies, legacy code, implicit conventions, and historical baggage. A high offline leaderboard score does not mean the tool works well on your own repository.
3. A single metric hides the cost structure. At comparable capability levels, token consumption can vary severalfold across models and integration paths, and per-unit pricing can differ by more than 30x. A configuration that scores “1 point higher at 30x the cost” will almost certainly be rejected when it has to scale across an enterprise.

The evaluation focus therefore has to shift from post-hoc benchmark scores to a joint four-dimensional assessment of capability / token consumption / cost / wall-clock time.

2. The Evaluation Framework: Three Benchmark Classes and Four Dimensions

2.1 What Each Benchmark Class Tests

For conclusions to be credible, the benchmark set must cover the core working chain of a coding agent, not just “writing code”:

Benchmark class	What it tests	Real-world scenario
Hard defect repair	Locating root causes in an existing repo, cross-file changes, making previously failing tests pass	Production bug fixes, regression patches
Terminal command execution	Running shell commands in a sandbox, building, installing dependencies, self-correcting from errors	CI troubleshooting, environment setup, script operations
Large-codebase Q&A	Cross-module retrieval, understanding call chains and architectural constraints, producing accurate explanations	Onboarding new engineers, architecture review, dependency impact analysis

These three classes stress-test the agent’s code modification ability, environment interaction ability, and context comprehension ability respectively. Missing any one of them introduces structural bias into the evaluation.

2.2 The Four Quantitative Dimensions

capability_score = f(bugfix_pass_rate, terminal_task_completion, qa_accuracy)
tokens           = input + output + cache_hits (cache hits are typically billed at a discount)
cost_per_task    = Σ(tokens_by_type × unit_price) + retry_amplification_factor
wall_time        = P50 latency + P95 tail latency (not just the mean)

The key point is that the fourth dimension must use percentiles rather than the mean. A configuration with a P50 of 6 minutes and a P95 above 40 minutes feels very different from one that holds steady around 10 minutes — and in a team making hundreds of calls a day, that difference in experience is far larger than the average suggests.

3. Finding One: The Capability Ceiling Is Set by the Underlying Model, Not the Client

One especially convincing observation from testing: after upgrading the same client from a lower-tier model to a flagship model, the aggregate score jumped by roughly 11 points (from a 49 level to a 60 level); meanwhile, different clients paired with the same model tier tended to land on the same score line.

The engineering implications are direct:

• Choose the model first, then the client. The client’s value lies in context construction, tool orchestration, permission control, and observability — it is an “amplifier.” The model is the “base.” Once the base moves from 49 to 60, no amplifier can compensate for starting at 49.
• Don’t pay a premium for a client brand. When multiple clients run the same model, the selection criteria should become engineering metrics: support for custom context injection, clear token and cost reporting, the ability to connect within a private network, and completeness of logging and auditing.
• Build your own adapter layer. Don’t hardcode one tool’s invocation style into business code. Abstract “model + client” into replaceable implementations:

# Use a protocol to isolate tool differences; model/client can be swapped at any time
class CodingAgent(Protocol):
    def run(self, task: Task, ctx: RepoContext) -> Result: ...
ROUTES = {
    "hard_refactor": ("flagship_model", "cli_client_a"),
    "bugfix":        ("mid_model",      "cli_client_b"),
    "explain":       ("lite_model",     "sdk_client"),
}

4. Finding Two: Token Consumption Varies Severalfold, and Cache Hit Rate Is the Hidden Dividing Line

At comparable capability levels, per-task token consumption can differ by more than 3x across combinations (for example, on the order of 4.8M vs. 1.5M). Two distinct quantities must be separated here:

• Effective billed volume: only the input portion that misses the cache is billed at the standard rate; the cached portion usually receives a substantial discount.
• Raw consumption: the total volume of context the model actually processes.

In the visualizations from testing, the large orange block that dominates is mostly cache hits — which explains why “consumption looks enormous, but the bill isn’t dramatic.” Conversely, if a combination has a low cache hit rate (common with third-party relay paths, frequently changing context prefixes, or dynamically assembled system prompts), its bill inflates several times over.

Three hard rules for improving cache hit rate:

1. Keep the prefix stable. Put the system prompt, repository conventions, and long-lived rules at the very front and keep them byte-for-byte identical; put user requests, timestamps, and random IDs at the end. Inserting mutable content in the middle invalidates the entire prefix.
2. Avoid unnecessary re-injection of large context. Stuffing the entire repository into every request wastes tokens and dilutes attention. Prefer retrieval augmentation: inject only the files, symbol definitions, and call chains relevant to the current task.
3. Watch out for excessive round trips. An agent that “thinks for 20 steps before acting” typically consumes several times the tokens of a “converges in 3 steps” approach for a single task.

# Low-cost context compression: trim by relevance instead of injecting everything
def build_context(repo, task, budget_tokens=40_000):
    hits = retrieve(repo.index, task.query, top_k=12)      # symbol-level retrieval
    files = expand_dependencies(hits, depth=1)             # one-hop dependency expansion
    picked, used = [], 0
    for f in rank_by_relevance(files, task):
        cost = estimate_tokens(f.content)
        if used + cost > budget_tokens:
            break
        picked.append(f)
        used += cost
    return Context(files=picked, stats={"used": used, "budget": budget_tokens})

5. Finding Three: A 30x Cost Spread — You Are Choosing a Point on the Price/Performance Curve

The cost distribution in testing was extremely steep: the cheapest combination cost about $0.07 per task, while the most expensive cost about $2.26 — a 32x difference. More interesting is the performance/cost efficiency ratio:

• One combination reached 50 points for about $0.35;
• Another reached 60 points for about $2.21;
• The former achieved 83% of the performance at roughly 1/6 of the cost.

This does not mean cheaper is always better. It means that the most expensive combination often does not sit at the knee of the price/performance curve, but on its flat section. Enterprise procurement should avoid a default “always use the strongest model” policy and instead tier by task:

Task tier	Typical tasks	Recommended strategy
L1 high-frequency, lightweight	Autocomplete, single-function rewrites, comments, log triage	Lightweight model + strict budget cap
L2 standard development	Defect repair, unit test generation, script writing	Mid-tier model + cache reuse + automated verification
L3 complex refactoring	Cross-module changes, architecture-level migrations	Flagship model + human review gate

The accompanying guardrails matter more than the routing policy itself: set a hard token and dollar cap per task, and automatically downgrade or escalate to a human when exceeded, so that a single runaway agent session cannot consume a month’s budget.

class BudgetGuard:
    def __init__(self, max_tokens: int, max_cost: float):
        self.max_tokens, self.max_cost = max_tokens, max_cost
        self.tokens = self.cost = 0.0
    def charge(self, usage) -> bool:
        self.tokens += usage.total
        self.cost   += usage.billed_cost        # cache discount already applied
        if self.tokens > self.max_tokens or self.cost > self.max_cost:
            return False                        # trigger downgrade/termination
        return True

6. Finding Four: Latency Tails Hurt Productivity More Than Average Latency

On the speed dimension, native direct-connect combinations with the shortest path performed best: about 5.8 minutes per task while also scoring 60 points, balancing efficiency and accuracy. Meanwhile, a combination at the same 50-point level took 41.5 minutes — more than 7x longer, most likely due to a convoluted path, many relay hops, and higher interface latency and queuing.

Why latency is a first-class citizen rather than just a UX concern:

• It rewrites the interaction model. A 6-minute wait is “submit–wait–review”; a 40-minute wait forces developers to open multiple tasks in parallel and come back to finish them, sharply raising cognitive load and the cost of lost context.
• It amplifies retry cost. Timeout retries are the norm on slow paths, and a single failure can cost an extra 40 minutes plus a full re-bill.
• It lowers task throughput. In CI or batch refactoring scenarios, per-task latency roughly determines pipeline duration linearly.

Two things should be done in engineering practice: measure P50/P95 instead of the mean, and explicitly govern slow requests —

POLICY = CallPolicy(
    timeout_sec=900,          # hard per-task timeout
    retries=1,                # retry once only, to avoid cost explosion
    fallback="mid_model",     # downgrade on timeout/failure
    stream=True,              # stream responses to surface reasoning direction early
)

7. Moving the Evaluation Into Your Own Repository: A Reusable Evaluation Loop

External leaderboards are only useful for initial screening; the final decision must be based on your own task set. A practical approach is to draw 50–150 cases from the team’s real engineering records over the past three months and build an internal benchmark:

from dataclasses import dataclass
import time, statistics
@dataclass
class Case:
    task_id: str
    repo_ref: str            # pinned repo snapshot (including commit hash)
    verify_cmd: str          # objective acceptance command, e.g. run specified tests
    kind: str                # bugfix | terminal | qa
    expected_hint: str       # optional; used only for manual review
@dataclass
class Run:
    case_id: str
    passed: bool
    retries: int
    tokens_in: int
    tokens_cached: int
    cost_usd: float
    wall_sec: float
def evaluate(agent, cases, policy) -> dict:
    runs = []
    for c in cases:
        guard = BudgetGuard(max_tokens=policy.max_tokens,
                            max_cost=policy.max_cost)
        t0 = time.time()
        res = agent.run(c, policy=policy, guard=guard)   # executed in sandbox
        ok = sandbox.verify(c.repo_ref, c.verify_cmd)
        runs.append(Run(c.task_id, ok, res.retries, res.tokens_in,
                        res.tokens_cached, guard.cost, time.time() - t0))
    return report(runs)
def report(runs) -> dict:
    return {
        "pass_at_1":     sum(r.passed for r in runs) / len(runs),
        "p50_sec":       statistics.median(r.wall_sec for r in runs),
        "p95_sec":       sorted(r.wall_sec for r in runs)[int(len(runs)*0.95)-1],
        "cost_per_pass": sum(r.cost_usd for r in runs) /
                         max(sum(r.passed for r in runs), 1),
        "cache_hit_rate":sum(r.tokens_cached for r in runs) /
                         max(sum(r.tokens_in for r in runs), 1),
    }

“Cost per pass” is the metric that most deserves a place in the procurement report. It constrains capability and spend simultaneously, and it directly eliminates “high cost for 1 point” options.

8. Four Practical Recommendations for Engineering Adoption

1. Prioritize context engineering over switching models. Accurate injection of the repository map, symbol index, type definitions, and test/build configuration often moves pass rate more than a model upgrade does, at a fraction of the cost.
2. An objective acceptance gate is mandatory. An agent’s output counts as delivered only after being verified by tests, type checks, static analysis, and the build pipeline. An agent without an acceptance gate is an uncontrolled source of code.
3. Sandboxing and least privilege are prerequisites, not options. Agents need to execute commands, which means they can read environment variables, write files, and make network requests. Run them in isolated containers, mount only the directories they need, forbid access to production credentials, and require manual confirmation for high-risk operations (deletion, force push, deployment, payment-related endpoints).
4. Treat untrusted content as untrusted input. Dependency descriptions, third-party documentation, issue content, and scraped web pages can all contain prompt-injection instructions that induce an agent to leak source code or credentials. Tag content from external sources as isolated and deny it any additional privileges in the tool-call chain.

9. Cost Governance Checklist for Teams and Enterprises

• Unify access and unify billing. Consolidate model calls behind a single gateway that produces bills broken down by project, by person, and by model, avoiding a “cost black box.”
• Tiered routing plus budget guardrails. Route tasks to different model tiers by difficulty, set token and dollar caps per session, and downgrade or escalate to a human automatically when caps are exceeded.
• Put caching strategy under review. Monitor and alert on cache hit rate as an operations metric on par with pass rate; an abnormal drop in hit rate usually means the prompt or the context assembly approach has changed.
• Build up observability. Call logs, latency distributions, categorized failure reasons, and the share of human-edited lines should be collected uniformly into logging and alerting systems, forming a data asset that supports continuous improvement. On the cloud, you can combine object storage to keep evaluation snapshots, a container platform to host isolated sandboxes, and a message queue to smooth batch task peaks, achieving on-demand scaling and controllable cost.
• Compliance and data security. Define which code may leave the internal network and which must go through private-network access and dedicated deployment; repositories containing personal information or business-sensitive data should be prohibited from leaving the network or being sent externally by default, with a complete audit trail retained.

10. Closing: From “Which One to Use” to “How to Use It”

The most valuable signal from this head-to-head test is not who came first, but that the center of competition has migrated from benchmark scores to cost, speed, and ecosystem. When the leading capabilities converge to within 1 point, the variables that determine team R&D efficiency are no longer “which tool did we pick,” but:

• whether the underlying model tier matches task difficulty;
• whether context construction is precise and whether caching is effectively reused;
• whether there is an objective acceptance gate and a security sandbox;
• whether latency tails and retry costs are explicitly governed.

Build the evaluation loop and the cost guardrails first, then talk about tool selection. The former is a transferable engineering asset; the latter is just a procurement conclusion valid for one moment — and the leaderboards change every month.