AI Agent Feature Delivery in Million-Line Codebases: Why Full Completion Stays Under 20%

1. Why “Correcting One Line of Code” Is Not “Delivering a Feature”

Product requirements rarely arrive as a file list plus function signatures. They are usually natural-language statements mixing business motivation, implicit constraints, boundary conditions, and acceptance criteria. A developer’s real work often happens before typing code:

• Understand what the system does and where the requirement falls within business boundaries.
• Determine which modules and interfaces actually depend on the requirement.
• Estimate how the change may spread through call relationships and which downstream components may break.
• Define what “correct” means: which tests, metrics, and regression items will prove it.

A recent agent evaluation on ultra-large code bases turned this chain into comparable tasks. The agents worked on real software systems averaging about 2.4 million lines of source code. They started from 5,000-word feature proposals and had to complete module localization, solution planning, and final code delivery. Even with mainstream capabilities such as code retrieval, task planning, multi-round editing, and test execution, the full-delivery rate was only about 14%. Across 28 agent configurations, almost none could reliably turn long-document requirements into runnable, test-passing feature changes.

The value of this result is not to rank products. It reveals a structural fact: agents’ code-understanding capability is already fairly mature, but requirement-implementation capability remains weak. The gap is not model intelligence; it is the engineering pipeline.

2. How the Evaluation Turns “Can It Deliver?” Into Measurable Problems

To make delivery capability comparable, the evaluation had to answer three questions: where the tasks come from, how real the environment is, and how metrics are defined.

2.1 Task construction: real feature proposals instead of synthetic patches

The evaluation did not use local patches such as “add a null check to a function.” It used feature proposal documents from real software evolution. These documents typically have the following characteristics:

• Length: about 5,000 words, with constraints spread across motivation, solution, compatibility requirements, performance expectations, and acceptance checklists.
• Redundancy and implicit constraints: many constraints are not in a formal requirements section; they are hidden in discussions, exception notes, or rejected alternatives.
• Not directly executable: the document may not specify which files to change, and may not even identify the service boundaries involved.

This matches everyday software development and is also the input format agents currently handle least well.

2.2 Environment scale: constraints from million-line code bases

A codebase averaging 2.4 million lines introduces hard constraints:

• Context cannot load everything: stuffing the entire repository into the context window is not viable; retrieval and summarization are required.
• Naming and structure carry historical baggage: similar features may have three implementations; old and new interfaces coexist; deprecated paths are not cleaned up.
• Cross-language and cross-layer call chains: a business change may touch service interface layers, domain layers, data-access layers, configuration layers, and even frontend type contracts.

2.3 Metric dimensions: beyond delivery rate, measure how clean the change is

An evaluation that only checks whether tests pass will significantly overestimate agent capability. Engineering usually needs four orthogonal metrics:

Metric	Definition	What it reflects
Delivery completeness	Passed acceptance items / Total requirement items	Coverage of long-document constraints
Localization hit rate	Files correctly identified in the first round / Files that actually needed changes	Codebase awareness quality
Change precision	1 − (Irrelevant changed lines / Total changed lines)	Penalizes large, blind changes that happen to pass
Regression safety	Original tests still passing / Total original tests	Destructiveness to existing behavior

Illustrative calculation:

def score_delivery(delivery, requirement_items, original_tests):
    completed = sum(1 for i in requirement_items if delivery.verifies(i))
    touched_noise = count_irrelevant_changes(delivery.diff, expected_scope)
    regressions  = [t for t in original_tests if not delivery.passes(t)]
    return {
        "completeness": completed / len(requirement_items),          # delivery completeness
        "localization": delivery.hit_files / len(delivery.target_files),
        "precision":    1.0 - touched_noise / max(1, total_changed_lines),
        "regression":   (len(original_tests) - len(regressions)) / len(original_tests),
    }

2.4 Configuration dimensions: 28 combinations show the bottleneck is not a single point

The 28 agent configurations were essentially a Cartesian product over several orthogonal dimensions:

• Retrieval strategy: keyword retrieval, semantic vector retrieval, dependency-graph traversal, or hybrid retrieval.
• Planning method: one-shot generation, phased planning, or iterative revision.
• Edit granularity: whole-file rewrite, function-level replacement, or patch-style editing.
• Verification strategy: no verification, unit-test-driven verification, or a compile-plus-test loop.

The differences between configurations were significant, but none broke through 20%. This indicates that the problem cannot be solved by swapping in a better retriever or a stronger model alone. The entire pipeline lacks engineering constraints.

3. Six Failure Modes Revealed by the Evaluation

Failure 1: Long requirements suffer compression distortion

Constraints in a 5,000-word document follow a long-tail distribution. On the first read, agents usually capture the main feature description accurately, but tail constraints are easy to lose during summarization. These include backward-compatibility requirements, permission models, performance expectations under concurrency, and gray-release or rollback requirements.

Once constraints are lost, later code changes are built on a wrong premise. This is hard to detect in testing because the requirement itself was never converted into tests.

Failure 2: Retrieval is shallow and finds “similar code” instead of “related code”

Semantic retrieval is good at finding code that looks similar, but it often finds the wrong target. Typical errors include:

• Retrieving a similar implementation from another module and editing that module, while missing the real target because of different naming conventions.
• Breaking cross-language call chains: changing a Python-side signature while the Go gateway and frontend type definitions that call it are ignored.

Failure 3: Dependency tracking only goes one hop

Agent dependency analysis is often “breadth-first, one level.” Real impact is a transitive closure:

Requirement: add a “partial refund” status to orders
Direct changes: status enum, refund interface
One-hop propagation: state-machine validation, refund amount calculation
Two-hop propagation: reconciliation jobs, report aggregation semantics, message notification templates
Three-hop propagation: frontend status-label mapping, external partner callback protocol, historical-data migration

Changing only the first two layers may allow the system to start and unit tests to pass, but reconciliation jobs may silently calculate the wrong amount in production. This “locally correct, globally wrong” behavior is the main hidden failure behind the 14% delivery rate.

Failure 4: “Looks done” replaces “proved done”

Agents without a verification loop may claim the task is complete immediately after editing. They may neither add tests nor run compilation. In mature engineering practice, verification often costs more than implementation: a feature may only be a few dozen lines of code, but it may need more than ten tests to cover boundaries, exceptions, and regressions.

Failure 5: Context budget mismatch

Putting many “possibly relevant” files into a limited context window introduces noise and dilutes the weight of key constraints. Agents may repeatedly read the same irrelevant code and exhaust reasoning turns without locating the place that actually needs modification.

Failure 6: Change boundaries get out of control

When facing ambiguous requirements, some configurations tend to “refactor along the way”: renaming variables, splitting functions, or reordering imports. These changes may not harm functionality, but they significantly reduce change precision, make code review harder to focus, and make regression localization more difficult.

4. Capability Layering: Break Delivery Into Four Layers

Improving agents requires clarifying the responsibility and acceptance artifact of each layer.

Perception layer: codebase understanding and impact analysis
Required capabilities: symbol indexing, call graphs and dependency graphs, cross-language boundary identification, association of configuration and protocol definition files, and deprecated-code detection. The acceptance artifact is a confidence-ranked candidate change list, not a pile of similar code.

Planning layer: requirement structuring and task decomposition
Required capabilities: converting long documents into acceptance-ready items, identifying implicit constraints, marking acceptance criteria, and producing an ordered execution plan with rollback milestones. The acceptance artifact is a set of task cards.

Implementation layer: cross-module coordinated changes
Required capabilities: interface contract synchronization, data-model and serialization compatibility, atomic multi-file changes, and minimal-change constraints. The acceptance artifact is a diff set that passes compilation and static checks.

Verification layer: self-proving correctness
Required capabilities: automatic test writing, compilation and test execution, failure attribution, regression comparison, and coverage/metric validation. The acceptance artifact is a reproducible verification report.

If any layer is missing, overall delivery drops to the level of that layer. This is the shortest-plank effect behind the 14% result.

5. Six Practical Engineering Improvement Paths

5.1 Pre-index the codebase and organize context by layers

Do not decide what to read only at retrieval time. Build layered indexes in advance:

• Symbol level: definitions, references, and signatures.
• Structural level: module and package dependencies.
• Behavioral level: tests and execution traces.
• Documentation level: design documents, change records, and interface protocols.

When assembling context, use a layered budget instead of unlimited filling:

CONTEXT_BUDGET = {
    "requirement_constraints": 0.20,   # Original requirement constraints, including tail sections
    "target_symbols":          0.25,   # Precisely located target symbols and signatures
    "dependency_closure":      0.30,   # Key call sites in the transitive closure, truncated by distance
    "contracts_and_tests":     0.20,   # Interface definitions and existing test patterns
    "reserve":                 0.05,
}

Key principle: requirement constraints must not be replaced by summaries. Long documents should be split by section and numbered item by item. Any compression must preserve the item number and a traceable reference to the original constraint text.

5.2 Structure requirements: turn a 5,000-word document into task cards

Before coding, require a structured acceptance checklist:

feature: partial_refund
items:
  - id: R1
    statement: Support multiple partial refunds for an order
    acceptance: [test_multi_partial_refund_succeeds]
    implied_constraints: [total refunded amount must not exceed the actual paid amount of the order]
  - id: R2
    statement: Refund status must remain compatible with existing reconciliation jobs
    acceptance: [test_reconciliation_total_matches, test_legacy_record_readable]
    implied_constraints: [historical records have no refund_parts field, so the field must be nullable or backward-compatible]
  - id: R3
    statement: P99 response time of a single refund API must not degrade by more than 10%
    acceptance: [benchmark_refund_latency_regression_guard]

The direct effect of this table is: any item not covered by a corresponding test is not considered delivered. It turns “I think it is done” into a decidable Boolean value.

5.3 Contracts first and tests first

Change contracts first: interface definitions, data models, and protocol files. Then change the implementation. Finally, add tests. If the order is reversed, downstream modules may not fail at compile time but may break at runtime.

For high-impact changes, generate failing tests from the acceptance checklist first and let the agent iterate only toward making those tests green. This naturally suppresses unrelated refactoring, because refactoring alone cannot make new tests pass.

5.4 Make impact analysis explicit and enforce the transitive closure

Before making changes, require the agent to output an explicit dependency closure and explain the reason for each layer:

L0 Direct changes: refund/service.py, refund/model.py
L1 One-hop propagation: state-machine validation, amount accumulation logic
L2 Two-hop propagation: reconciliation calculation semantics, report aggregation, notification templates
L3 Three-hop propagation: frontend status mapping, external callback protocol, historical-data migration strategy
This iteration: handle L0-L2; mark L3 as pending confirmation and report it explicitly

The value of this step is to turn “I did not think of it” into a known to-do item and avoid silent omission.

5.5 Separate agent roles: retriever, implementer, and reviewer

A single agent handling retrieval, implementation, and review may develop confirmation bias: it tends to believe the location it found is correct. Splitting roles helps:

• Retrieval agent: outputs candidate change locations and confidence only; does not write code.
• Implementation agent: changes code within the given scope; must not add files outside that scope.
• Review agent: independently checks whether all acceptance items are covered, whether unrelated changes were introduced, and whether any impact paths remain unhandled.

Cross-review usually improves change precision and regression safety more than swapping the underlying model.

5.6 Bring the evaluation loop into the pipeline

Fix this kind of benchmark set in the development pipeline as a continuous regression gate for agents:

Submit candidate change
  → compile and static checks
  → new tests plus full regression of existing tests
  → score completeness, precision, and regression rate
  → if below threshold, reject with failure attribution
  → store results and historical trends as observable metrics

Evaluation should not be a one-time horizontal comparison. It should be a vertical regression check after each capability upgrade.

6. Infrastructure Support

Running these processes requires a stable, reproducible, and observable engineering environment rather than a single “intelligence” feature. From a general cloud-product perspective, the key components include:

• Container platform: provides an isolated and reproducible execution sandbox for each task, ensuring consistent codebase snapshots, dependency versions, and toolchains, and avoiding environment drift that makes conclusions unreliable.
• Object storage: stores codebase snapshots, requirement documents, evaluation artifacts, and historical baseline sets, supporting task-level traceability.
• Message queue: handles task distribution and result collection, supporting asynchronous parallel evaluation of large numbers of configurations.
• Logging service: retains complete agent execution traces, including which files were retrieved, why a location was selected, and why a task failed. This is the primary basis for attribution analysis.
• Monitoring and alerting: tracks trends in delivery completeness, regression safety, and average turns, so capability regressions can be detected quickly.

Baidu AI Cloud can provide the corresponding container, storage, messaging, logging, and monitoring components for building such evaluation and development pipeline environments. Specific choices should be evaluated based on codebase size, concurrent evaluation volume, and compliance requirements. A single “AI coding tool” output should not be the only criterion for the quality gate.

7. Conclusion: From “Completing Code” to “Owning Delivery”

A 14% delivery rate is not a rejection of agent capability. It confirms the realism of the evaluation difficulty. The bottleneck has moved from “can it write this code?” to “can it understand a million-line system, enumerate the impact surface, and prove its own correctness?”

For teams, three practical paths matter:

1. Structure requirements and acceptance conditions first, then hand them to agents. Improving input quality is often more effective than upgrading the model.
2. Make impact analysis and transitive closure explicit, turning silent omissions into known to-do items.
3. Build quantifiable delivery evaluation gates, replacing “it looks done” with completeness, precision, and regression rate.

When agents are placed in sufficiently realistic engineering constraints, such as large codebases, long-document requirements, and strict regression checks, their progress will show up in delivery quality rather than demo effects.