Evaluating Coding Agents on Real Repositories: What Scores Miss and How to Ship
Higher coding-agent scores do not automatically mean fewer people on the team. This article explains how to evaluate agents on real repository tasks, separate retrieval, planning, and tool-use failures, measure cost per useful patch, and roll out with retrieval, budgets, observability, and human merge gates.
Coding agents can already read a repository, apply patches, and run tests. A higher score still does not mean a team can staff fewer people. Selection should start with three questions: whether tasks come from real repositories rather than toy problems; whether failures can be attributed to retrieval, planning, or tool use; and whether compute and wait time per useful patch are acceptable. The rest of this article covers evaluation, how to think about architecture, cost, and a practical rollout path, without tying the discussion to a specific model or vendor API.
Split capabilities instead of chasing a single score
A composite coding score mixes language understanding, patch quality, and tool use. Engineering decisions benefit more from separate observations.
Real-repository defect repair measures whether, given an issue description and a codebase, the model can find relevant files, make a minimal change, turn failing tests green, and leave unrelated behavior intact. The hard part is not writing a new function; it is repository conventions, implicit dependencies, and historical patch style. Pin repository snapshots, dependency versions, and test commands so a pass today is not an environment-change false positive tomorrow.
Terminal and build work measures whether multi-step commands can recover. The model must read errors, change configuration, and rerun rather than emit a script that only looks correct. Score command sequences, exit codes, and whether the agent touched paths it should not modify on the evaluation host.
Natural language to repository skeleton measures completeness of a from-scratch setup: directory layout, module boundaries, a runnable entry point, and basic tests together. It tests planning; it does not replace defect repair. For long-running tasks, also check whether the agent can continue from existing artifacts after interruption instead of regenerating from scratch.
Leading one slice does not mean leading overall. Strength on repair can coexist with instability on cross-module design, ambiguous requirements, or security-sensitive edits. A selection report should at least list pass rate, average patch size in lines, regression failure rate, human takeover count, and end-to-end latency.
Build a small, reproducible evaluation set
Public leaderboards are useful for trends, not purchasing decisions. Teams should shadow-evaluate on their own repositories.
Sampling: take closed defects from the last one or two iterations, covering logic bugs, interface compatibility, and configuration or build failures. At least about ten items per class, or variance will hide differences. Drop items that need external accounts, production data, or unreproducible environments.
Gold standard: keep the original fix commit as reference, but do not require byte-for-byte identical text. A better bar is hidden tests pass, public tests pass, static checks pass, and the diff stays inside allowed directories.
Controls: same prompt, same tool permissions, same timeout; change only the model or decoding parameters. Temperature, maximum steps, and whether rollback is allowed must be in the experiment log or the run is not reproducible.
Slicing: group by language, repository size, and whether cross-file edits are required. A model may be stable on single-file patches and miss call sites on cross-service contracts. Averages hide that risk.
A schematic evaluation record need not bind to any platform; the fields should stay stable:
task_id: repo-issue-1042snapshot: commit pinnedallowed_paths: src/, tests/timeout_min: 45checks:- unit_tests- lintoutcome: pass | fail | timeout | unsafe_edithuman_takeover: true | falsetokens_in_out: recorded
Unsafe edits include changing dependency resolution beyond lockfiles without explanation, writing secret samples, or deleting tests to go green. Those should fail outright, not count as a “creative pass.”
A long context window is not the same as reading the whole repo
A larger window only means more text can fit in one request; it does not mean attention lands in the right place. Very long inputs incur three costs: prefill compute, cache occupancy, and interference from irrelevant code.
In practice, retrieve first, then generate. Narrow context by symbols, call graphs, and recently changed files. Putting the issue, related tests, and a few neighboring files in the prompt is more stable than pasting the whole tree. When retrieval fails, degrade to listing candidate files and stopping rather than editing the wrong files.
Sparse attention is a common long-sequence idea: not every position interacts with the full history at the same density; compute focuses on more relevant spans to cut compute and memory. Whether it helps depends on implementation and task mix; the name does not imply the same job is always faster or cheaper. For code, verify that latency and cost stay acceptable as the repo grows, and whether cross-file references are more likely to be missed.
Mixture of experts is another common structure: total parameters can be large while each inference activates only some experts, with routing sending different inputs to different subnetworks. That trades capacity against per-call compute, but routing quality, expert load balance, and server-side batching all affect real latency and unit price. Compare activated scale, context length, and output length together, not total parameter count alone.
Asynchronous, multi-task training or inference scheduling mainly raises accelerator utilization: while one batch waits on data or tools, another keeps computing. Iteration counts or “speedup factors” without controls and metric definitions are not capability claims. In production, watch SLAs: queue time, how rate limits fail, and whether long jobs get killed mid-run.
Put price, cache, and capacity on one ledger
Coding-assistant cost is usually driven by input, output, and cache hits together. As context grows, repeatedly sending the same repository prefix can make cache cut repeated prefill cost and latency. If cache pricing is close to or above some general-purpose tiers, “it used to be cheap so we overuse it” no longer holds.
Price by useful patch, not by call count. One task may include retrieval, multiple generations, and rollback after test failure. Record:
- Input and output volume, and cache hit ratio
- Wall-clock time from task receipt to tests green
- Minutes of human review
- Share of rejected or redone work
Capacity is cost too. Popular models queue or degrade at peak. Pilots should assume rate limits: tasks can pause, retry with smaller context, and a person can still take over the critical path. Do not write “always online” into the process.
Open weights, a commercially usable license, and a hosted API are three different paths. Self-hosting can keep data in-domain but you own inference clusters, monitoring, and upgrades. Hosted APIs reduce operations but require redaction and audit of sensitive code and prompts. Whether a license allows commercial use and whether attribution is required must follow the license text, not the word “open source.”
Long-horizon autonomy needs bounds, not unattended operation
Hours of autonomous repo edits fit tasks with clear dependencies, strong tests, and easy rollback: scaffolding modules, fixing a batch of similar defects in existing style, or generating a runnable demo. They do not fit payment, permissions, data migrations, or production configuration.
A minimal control plane includes:
- Permissions: write only to a working copy and a designated branch; secrets as short-lived credentials, unread by default.
- Step and time budgets: stop on timeout, keep logs and patches so a person can continue.
- Tool allowlist: read files, run tests, format; no downloading unknown scripts from the public internet unless the dependency source is pinned.
- Acceptance gates: tests, static checks, license and secret scans must pass before a reviewable change is produced.
- Human sign-off: merge rights stay with people; the model does not push to protected branches.
Observability matters more for long jobs than for short Q&A. Each step should answer: which paths were read, which commands ran, what tests returned, and whether the next plan still matches the original goal. Without that trail, eight hours of output cannot be reviewed, and “model skill” cannot be separated from “the environment happened to work.”
Building a runnable app from scratch can demonstrate capability, but acceptance must define scope: whether the UI opens, whether core paths have tests, whether dependencies can be locked, and whether default weak passwords or debug backdoors are present. A demo pass is not production-ready.
Embed the model in everyday engineering; do not replace the process
A reasonable pilot order is: read-only review, then generate patches without auto-merge, then multi-step edits on low-risk repositories. Compare each stage against a human baseline on the same task set.
Version prompts and repository norms. Put style, error-handling conventions, and test commands in a short, retrievable document rather than repeating them in chat. Patch descriptions should state motivation, blast radius, and how to roll back so a reviewer can decide in a few minutes.
Review where models fail: missed call sites, wrong edge cases, swallowed exceptions, weakened tests, overly broad dependency upgrades. Use a checklist, not intuition.
When combining with cloud development environments, generic capabilities are enough: code and artifacts in object storage or an artifact registry, task state on a queue, logs and metrics in logging and monitoring, alerts only on timeout, out-of-scope paths, or widespread test failure. A pilot need not bind to one vendor’s proprietary interface. If you use general cloud resources such as Baidu AI Cloud, treat them as compute, storage, and observability; evaluation conclusions still come from your own repository tasks.
Rollout checklist
- The eval set comes from real defects, the environment is reproducible, and it includes regression and safety-failure cases.
- Report sliced capabilities, human takeover rate, and cost per useful patch together.
- Trim context with retrieval; treat long-sequence and mixture-of-experts designs as cost and capacity hypotheses and verify them empirically.
- Long tasks have permissions, budgets, traces, and a human merge gate.
- Data-exfiltration scope, license obligations, and peak rate limits are written into the runbook.
A score can suggest “worth a trial.” It cannot replace this loop. When pass rate, rework, and cost are all acceptable on your repositories, a coding agent has moved from a demo to an engineering capability.
