Why Repository-Level AI Agents Deliver Only 14% of End-to-End Tasks

1. The real boundary: from “can modify code” to “can deliver a feature”

Today’s code agents already have a solid set of basic capabilities: reading files, retrieving symbols, editing code blocks, and running build or test scripts. On function-level repair tasks, their performance often approaches, and sometimes exceeds, that of junior developers. But when the task changes from “fix this function” to “implement this product requirement,” the difficulty rises sharply.

The problem is not a single weak capability. The task shape itself changes fundamentally. When a developer receives a requirement, it is usually neither a file list nor a function signature. It is a natural-language description carrying business context: multi-condition filtering, staged rollout, export rate limiting, state-machine extension, and so on. To bring such a requirement into the system, the agent must make four sequential judgments: understand explicit and implicit constraints, trace dependencies between modules, locate the places that actually need modification, and design an acceptance plan that can prove the change is correct.

These four judgments form an analysis chain with strong dependencies. An error at any step can be amplified by later steps. This is also why evaluation must scale the task unit from “modify a function” to “deliver a feature.”

2. Why large software repositories are hard: compounded difficulty across four dimensions

When an agent is placed inside a system with an average of roughly two million lines of source code, the difficulty does not come from code volume alone. It comes from multiple factors stacking together.

1. Sparsity of relevant code

The implementation code related to a requirement often accounts for less than 0.01% of the entire repository. The most critical constraints may also be hidden in configuration files, migration scripts, annotation processors, or generated code artifacts. These locations often have little lexical overlap with the requirement description, so purely semantic retrieval can easily miss them.

2. The scale of implicit contracts

Large systems accumulate many conventions that are not written in documentation: field order must remain immutable, event payloads must stay backward compatible, idempotency keys must be calculated in a specific way, and cache and persistence layers must remain consistent. These conventions are scattered across test assertions and caller code, forming invisible red lines for any modification.

3. Cascading changes across modules

A feature that looks local often requires coordinated changes across interface definitions, implementation layers, serialization structures, permission checks, frontend contracts, and data migration scripts. If any one part is missed, the build may still pass, but runtime behavior can already be wrong.

4. The verifiability gap

An agent needs to “prove the change is correct,” but that proof depends on whether the repository contains tests that cover the affected behavior. In modules with insufficient test coverage, the agent can neither prove correctness nor obtain useful correction signals. It is forced to rely on pattern matching and guesswork.

3. What repository-level benchmarks should test: separating perception from implementation

A class of repository-level agent benchmarks is taking shape in the industry: using real, long-evolving repositories as the environment, defining tasks as feature proposals, and scoring results through automated acceptance. There are three methodological design choices worth learning from.

Make acceptance criteria objective

“The code looks reasonable” should not be a passing condition. Instead, acceptance should be based on hard thresholds: the build succeeds, unit tests pass, contract tests detect no breaking changes, and the regression suite is fully green. This removes the subjectivity of human review.

Score perception and implementation separately

Perception capability measures whether the agent can find the correct modification locations and all related points. It is often evaluated by the hit rate on key files and key symbols. Implementation capability measures whether the changes satisfy acceptance criteria once the correct locations have been identified.

Separating the two scores helps locate the bottleneck. If the perception score is high but the implementation score is low, the problem lies in generation quality. If the reverse is true, retrieval and context construction should be improved first.

Manage context budget

The agent’s context window is a limited resource. Benchmarks should clarify the tradeoff between the total amount of retrieved context and the final pass rate. Stuffing too much irrelevant code dilutes useful signal, while providing too little context makes implicit constraints hard to discover.

In a typical task specification, the requirement, entry points, invariants, and acceptance conditions can be written explicitly so that both the agent and the verification pipeline can consume them:

{
  "task_id": "FR-2031",
  "requirement": "Order export supports combined filtering by time range and order status",
  "entry_points": ["services/export/handler", "api/orders/export"],
  "invariants": [
    "Total rows per export must not exceed the existing limit",
    "Export field order must remain backward compatible",
    "Async tasks must be idempotent"
  ],
  "acceptance": {
    "build": true,
    "unit_test": "not lower than the baseline before the change",
    "api_contract": "no breaking changes",
    "regression": "all core-path regression cases pass"
  }
}

4. What the measured results show: delivery rate stays below 15%

In public large-scale repository-level evaluations, researchers systematically tested multiple agent configurations, covering different model combinations, retrieval strategies, context organization methods, and whether self-verification loops were introduced. The results showed a consistent pattern: even the best-performing configuration achieved only about a 14% end-to-end full delivery rate, meaning the vast majority of tasks failed to pass all automated acceptance checks.

Failures were not evenly distributed. They clustered into several identifiable patterns:

• Localization bias: The agent modified an adjacent but wrong module. The implementation itself may have been decent, but it did not affect the behavior being accepted.
• Constraint omission: The change broke an undocumented contract, causing contract tests or compatibility checks to fail.
• Cross-module inconsistency: The implementation layer was modified, but the interface definition and callers were not synchronized. The build passed, but execution failed.
• Missing verification: The agent claimed the task was complete without running the build or tests, allowing errors to remain hidden.
• Context drift: During multi-round task progression, the agent gradually drifted away from the initial requirement, and repeated patching introduced new defects.

This distribution leads to an important conclusion: moving the delivery rate from the low teens to a usable level depends not on continuing to enlarge models, but on closing two engineering gaps: context construction and verification loops.

5. Reusable engineering methods

1. Build a repository knowledge layer instead of relying on brute-force retrieval

Build an index over the repository and connect symbol definitions, reference relationships, call chains, and configuration items into a queryable code graph. Compared with pure text-similarity matching, structured indexes can follow dependency edges and bring hidden related code into context.

def collect_context(task, repo_graph, budget):
    seeds = resolve_symbols(task.requirement)          # Extract candidate entities from the requirement
    related = set()
    for s in seeds:
        related |= repo_graph.callers(s, depth=3)      # Upstream callers
        related |= repo_graph.callees(s, depth=2)      # Downstream implementation
        related |= repo_graph.tests_covering(s)        # Tests covering the symbol
        related |= repo_graph.config_refs(s)           # Related configuration and feature flags
    return rank_by(task, related)[:budget]             # Truncate by task relevance

The value of this flow is that it elevates “tests covering the symbol” and “configuration referencing the symbol” to first-class retrieval targets. These files are often exactly where implicit contracts are concentrated.

2. Turn requirements into executable task specifications

Natural-language requirements need to be translated into structured specifications: entry points, an invariant list, acceptance conditions, and forbidden areas. This step is equivalent to manually completing requirement clarification, compressing ambiguous space into a decidable scope.

The benefit is not only higher accuracy. More importantly, failures become attributable: an acceptance report can explicitly state which invariant was violated.

3. Build closed-loop verification and feed failure evidence back

The design principle of the verification loop is to feed only failure evidence back to the model, not the entire repository. This provides a correction signal while controlling context expansion.

def verify_cycle(workspace, spec, max_rounds=5):
    report = None
    for _ in range(max_rounds):
        results = [
            run("build", cwd=workspace),
            run("static_check", cwd=workspace),
            run("unit_and_regression_test", cwd=workspace),
        ]
        report = aggregate(results)
        if report.all_green and contract_unchanged(workspace, spec):
            return report
        patch_context(spec, report)   # Feed back only failed cases and error stacks
    return report

Isolating this process in a sandboxed environment is especially important: intermediate states produced by the agent may conflict with other repository workflows. A container platform with isolated instances and snapshot capabilities can help ensure that execution is reproducible and can be rolled back.

4. Clarify the role of agents in the development workflow

At the current capability level, it is unrealistic to place repository-level agents directly into a “receive requirement and deliver immediately” role. A more practical positioning is three roles: a requirement impact-analysis assistant, an executor of mechanical cross-module changes, and an automated front-end for regression verification.

Using end-to-end delivery rate as the only metric can hide real value. Even in failed cases, the impact analysis produced during the perception stage can often be reused directly by human developers.

6. Supporting cloud capabilities for enterprise deployment

To run this workflow stably inside an enterprise, what is needed is not a single product but a combination of general-purpose cloud capabilities:

• Object storage can store code snapshots, retrieval indexes, and complete execution traces for each task, supporting later attribution and effect comparison.
• Message queues can handle asynchronous scheduling and peak shaving for long-running tasks, preventing large numbers of parallel tasks from overwhelming build clusters.
• A container platform can provide isolated build and test environments, ensuring that execution results are reproducible and can be rolled back.
• A logging service can record the agent’s retrieval hits, tool calls, and failure reasons, forming searchable debugging clues.
• Monitoring and alerting can track drift in metrics such as delivery rate, average number of repair rounds, and resource consumption per task, providing a basis for continuous tuning.

It is also recommended that enterprises build their own small-scale evaluation set: select 50 to 100 historical requirements from real repositories, retain the original changes as ground truth, and score regressions using the same automated acceptance method as public benchmarks. This way, the effect of every retrieval-strategy or prompt adjustment can be quantified instead of being based only on isolated impressions.

7. Conclusion

The core fact revealed by repository-level evaluation is that the agent’s weakness is not coding, but perception and self-verification. The roughly 14% end-to-end delivery rate is not the final limit of capability. It is the ceiling of the current context-construction method and verification mechanism.

Teams that first make code-graph indexing, specification-driven task definitions, and closed-loop verification solid will be the first to move this number into an engineering-usable range. These three items are also fully controllable by engineering teams.