Evaluating Bug Fixing in Real-World Code Repositories

    Repairing defects in production-style repositories requires much more than completing an isolated function. A reliable evaluation must measure whether an agent can understand repository structure, trace call chains, run tests, modify multiple files, recover from failures, and produce a mergeable patch. This article presents a practical framework covering evaluation design, model architecture, long-running agent control, observability, and cost analysis.

    Fixing defects in a real code repository is fundamentally different from completing a single function. It requires reading the directory structure, tracing call chains, running tests, changing multiple files, and recovering from failed attempts. When evaluating this capability, teams should separate the model, the toolchain, and the evaluation protocol:

    • The model provides reasoning and code generation.
    • Tools provide retrieval, execution, and observability.
    • The protocol determines whether the score is reproducible.

    The sections below focus on what to measure, where architectural efficiency comes from, how to stabilize long-running tasks, and how to calculate cost. The goal is to help teams deploy these systems responsibly rather than chase a temporary leaderboard position.

    1. Evaluate Mergeable Patches, Not Persuasive Explanations

    A common industry approach is to turn previously fixed defects from real repositories into tasks. The model receives a problem description, failing tests, or logs, then must submit a patch in an isolated environment. The primary metrics are whether the tests pass and whether existing behavior remains intact.

    This approach has strong external validity, but it also has clear limitations.

    First, the repository snapshot, dependency versions, and test stability can directly change the score. The same patch may fail in a network-restricted environment or on a time-sensitive test. Such failures should not automatically be interpreted as evidence that the model cannot write the code.

    Second, a single-pass success rate hides variance. Complex defects often require several rounds of experimentation. Reporting only the best result from one run conflates a solution that happened to work with one that is stable and reproducible. A more reliable report should fix the random seed and tool permissions, then include the mean, relevant percentiles, and a distribution of failure types.

    Third, a one-point score difference often has little engineering significance. If the sample contains only a few hundred tasks, the confidence interval may be larger than the gap between rankings. For procurement or model selection, public benchmarks should be treated as smoke tests. Teams should also build a targeted evaluation set from their own repositories, including historical defects, mixed-language projects, private build scripts, and internal coding standards.

    A minimal evaluation loop should include:

    • Task card: Symptoms, reproduction steps, allowed paths, and prohibited actions, such as weakening tests to make them pass.
    • Environment: A container image, locked dependencies, and either read-only network access or an explicit allowlist.
    • Acceptance criteria: The original failing test passes, existing regression tests do not all fail, and static analysis does not introduce new high-severity findings.
    • Telemetry: Tool-call count, context tokens, wall-clock time, and the number of human interventions.

    A task description can remain simple as long as its boundaries are explicit:

    1. Goal: Fix the counter undercounting during concurrent writes.
    2. Allowed: Modify implementation files under src/counter and related source files outside the unit tests.
    3. Prohibited: Do not remove or weaken existing assertions; do not introduce undeclared external services.
    4. Completion criteria: The reproduction script exits with code 0, and the full unit test suite passes.

    2. Parameter Count Does Not Equal Compute Used by Every Inference

    Architectures such as Mixture of Experts can have a large total parameter count while activating only a subset of experts for each forward pass. External communication should report both total parameters and active parameters. Otherwise, “larger” may be incorrectly interpreted as “more expensive on every request.”

    The active parameter count affects latency and throughput. The total parameter count affects memory residency, loading time, and distributed partitioning.

    For coding workloads, the value of expert specialization depends on routing stability. Similar patching tasks should tend to reach similar experts rather than oscillating widely between them. Routing instability reduces cache hit rates and increases tail latency. Useful engineering metrics include expert load balance, routing entropy, and per-token latency under long-context workloads.

    Long source files create another bottleneck. If attention performs dense computation over every historical position, both compute and memory requirements grow rapidly as the sequence becomes longer. Sparse or dynamic attention concentrates computation on positions that are more relevant to the current prediction, such as the current function, the error stack, or the most recently edited diff.

    This is not free improvement in code understanding. It exchanges computation per sequence length for a selection mechanism. If the selector chooses poorly, the model may miss distant invariants or configuration files. The result can be a patch that is locally correct but fails during integration.

    Claims about long context should therefore be supported by at least two measurements:

    • On a fixed defect set, gradually increase the context window and measure where the pass-rate curve begins to plateau.
    • Near that saturation point, compare dense attention with sparse strategies in latency, memory usage, and token consumption.

    If the pass rate has already plateaued, expanding the context window further may simply increase the cost of irrelevant files. In practice, retrieval is often more effective: retrieve snippets by symbol, call relationship, or error text, then provide them to the model. Vector retrieval can improve recall, but code identifiers, overloads, and repeated patterns in generated code make pure semantic nearest-neighbor search insufficient. Retrieval should be combined with paths and language-service features such as definition jumps and reference lookup.

    3. Long-Running Autonomy Depends on State, Not One Continuous Run

    For agents that operate on tasks lasting several hours, the main bottleneck is usually process control rather than single-step completion. A greenfield build or large refactoring task will inevitably encounter dependency-installation failures, flaky tests, and ambiguous requirements. Without checkpoints, a context overflow or process crash can force the entire task to restart.

    A practical control plane should include at least the following:

    1. Planning and budgets. Break the goal into verifiable subgoals, and set tool-call and wall-clock limits for each one. When a limit is exceeded, stop and write a failure summary instead of retrying indefinitely.
    2. Workspace snapshots. Commit intermediate states whenever a group of tests passes, so failures can be rolled back to the previous green state.
    3. Observability. Preserve command output, diffs, and important logs, but truncate excessively long output so irrelevant compiler warnings do not consume the context window.
    4. Permissions. Give the executor least-privilege access by default: it should read and write only within the task directory, have no access to production secrets, and be unable to send arbitrary packages to the public internet. External dependencies should be provided through prebuilt images.
    5. Human gates. When a change affects data migration, a public interface, or compatibility, the automated process should stop at a reviewable patch instead of merging directly.

    “ No human intervention in the middle” should not mean eliminating review. A more practical objective is for the agent to close the loop on routine compilation, unit testing, and formatting, while humans retain responsibility for design trade-offs and security-sensitive changes.

    Long-task evaluations should report completion rate, rollback count, human intervention points, and final defect density rather than only how long the agent ran.

    Asynchronous and parallel sampling of multiple candidate solutions, followed by selecting the best one, is common in both training and inference. It can improve exploration efficiency, but it also increases the risk of test contamination. If candidates share the same cache or external state, their results are no longer independent. Parallel experiments must isolate workspaces and ports.

    4. Cost Modeling: Price, Token Reduction, and Retries Belong in One Table

    Unit price, context length, and retry count jointly determine the cost of a repair. Looking only at a lower per-call price or a smaller active parameter count is not enough. A useful rough estimate is:

    1. Cost ≈ (input tokens + output tokens) × unit price × number of attempts + tool-side machine time

    The number of attempts often dominates variance. Sparse attention and better retrieval may reduce irrelevant context enough to offset a higher unit price. However, routing errors that trigger repeated retries can increase total cost instead.

    Before production rollout, replay real tickets for a week and measure the median and P95 cost of successful patches. This is more reliable than estimating costs from demonstration tasks.

    Caching must also be included in the cost model. Stable prefixes such as system prompts, repository summaries, and coding standards are suitable for prefix caching. Frequently changing diffs are not good candidates for long-lived caches. Logs should distinguish cached input from newly computed input; otherwise, billing fluctuations will be difficult to explain.

    When inference and execution environments run in the cloud, general-purpose infrastructure is usually sufficient: container platforms can run isolated evaluations, object storage can retain repository snapshots and execution traces, observability systems can track failure types and tail latency, and message queues can smooth batch workloads.

    The infrastructure choice should not be tied to the promotional parameters of a particular model generation. Platforms such as Baidu AI Cloud can provide support for images, storage, and observability, while the model itself should be evaluated independently under the protocol described above.

    Before introducing these capabilities into daily development, use four gates rather than relying on a single aggregate score:

    • Targeted benchmark: Sample defects from the repository’s most recent three months and report pass rate along with the number of lines humans had to modify.
    • Security: Patches must not expand privileges or write secrets into the repository. Dependency changes must be reviewable.
    • Cost: The P95 cost and latency per defect should be lower than the opportunity cost of a human’s initial investigation before traffic is increased.
    • Rollback: Agent-generated commits and human commits should follow the same code-review process and retain one-click rollback capability.

    Models will continue to change quickly, and leaderboard rankings will shift. The durable methodology is to define “can fix bugs” with reproducible repository tasks, define “is efficient” using active compute, retrieval quality, and retry count, and define “can run for a long time” using checkpoints, permissions, and review.

    A dashboard built around these three dimensions can show whether an upgrade is genuinely more capable—or merely better at passing benchmark tasks.