Benchmarking AI Agent Delivery on a Million-Line Codebase

Introduction: From “Using Tools” to “Delivering Features”

Today’s large-model AI agents can already read and write files, execute commands, and call test frameworks with reasonable competence. In well-bounded, self-contained tasks, they often perform quite well. But the moment the scenario shifts to a real software system that has evolved over many years, the situation becomes much more complex.

A product manager’s new request usually does not arrive as a clean list of files or function signatures. Instead, developers receive a natural-language description. The first job is not to write code, but to understand system constraints, trace dependencies between modules, decide where a change belongs, determine how to make it, and finally prove that the change is correct.

This “understand → locate → modify → verify” chain is often overlooked in AI agent evaluations, yet it is one of the hardest parts of real engineering work.

Some research teams have focused directly on this gap: they place AI agents inside software systems averaging around 2.4 million lines of source code, give them a requirements document of about 5,000 words, and evaluate the full process from “feature proposal” to “module delivery.” The results are striking. Across multiple agent configurations, the proportion of agents that can complete the entire task and deliver a fully working feature remains low.

This is not simply a matter of one model being “not good enough.” The task itself is inherently difficult. This article examines, from a technical perspective, why it is difficult, where the difficulty lies, how to evaluate it scientifically, and what developers and enterprises can learn from it.

1. Why Large Codebases Are the Real Test for AI Agents

For a small function or an isolated script, a model can often grasp the code at a glance. But an engineering system that has evolved over several years contains complexity across several different dimensions:

• The codebase exceeds the context window. A million-line codebase cannot be loaded all at once. The model can only see fragments and must rely on retrieval, navigation, and local reading to assemble a global understanding. This naturally introduces the risk of incomplete information.
• Key knowledge lives outside the code. Many critical constraints are not written directly in the code. They are embedded in architectural conventions, naming practices, historical decisions, missing comments, and unwritten rules. Which modules should not be modified casually, and why a particular interface was designed that way, often only becomes clear to people familiar with the system.
• Dependencies form a web. A single change can affect many other parts of the system. Callers, callees, shared data structures, and implicit timing assumptions create long-range dependencies across modules.
• Understanding a fragment is not the same as understanding the system. Explaining a piece of code does not mean understanding its position, responsibility boundaries, and impact within the whole system.

Therefore, placing an AI agent inside a real codebase is no longer just a test of whether it “knows the syntax.” It tests whether the agent can think like an engineer who is familiar with the system.

2. Two Core Capabilities: Perception and Implementation

This kind of evaluation abstracts complex delivery into two measurable capabilities. The logic is clear.

1. Perception: Judging Where a Requirement Belongs in the System

• Requirement understanding: Extract the real intent from about 5,000 words of natural language, and identify implicit constraints and boundary conditions.
• Code localization: Find the genuinely relevant code locations among many modules, not merely locations that look superficially similar.
• Dependency tracing: Clarify which callers and data flows will be affected by the change.

2. Implementation: Completing and Proving Changes Across Modules

• Change landing: Make the right modifications in the right places, while remaining consistent with existing style and interface conventions.
• Consistency maintenance: After modifying one place, update all affected locations.
• Result verification: Run tests, inspect behavior, and prove that the feature works as required without introducing regressions.

An interesting insight is that weaknesses in perception are amplified downstream. If the agent goes wrong during localization and understanding, later implementation efforts can make the deviation worse. This explains why many failures are not because the agent “wrote the wrong code,” but because it “changed the wrong place.”

3. Why End-to-End Delivery Rates Remain Low

Once the task structure is clear, it becomes easier to understand why the success rate drops so sharply. The main bottlenecks include:

• Local optima caused by limited context: The model often makes decisions based on incomplete information, misses key constraints, and produces changes that seem locally reasonable but are globally wrong.
• Long-range dependencies are hard to close: Dependencies that span multiple modules and cover large code distances exceed what a single reasoning pass can handle. This leads to situations where one part is changed but another is forgotten.
• Interface and API hallucination: The model may invent functions, parameters, or configuration options that do not exist. They may look plausible but fail to compile or pass tests.
• Bias in requirement understanding: Natural language is ambiguous, so models tend to choose the “most common implementation” while ignoring project-specific conventions.
• Missing verification: If the model only generates code without actually running tests, it cannot prove correctness. Delivery remains at the level of “looks correct.”

Together, these factors make “from requirement to deliverable feature” a much longer chain than it first appears.

4. Designing Rigorous Evaluations for End-to-End Delivery

To measure this kind of end-to-end capability, the evaluation methodology itself must be rigorous. Useful design principles include:

1. Build tasks from real codebases. Tasks should come from real, long-evolving engineering projects rather than artificial toy examples. Only then can systemic difficulties be exposed.
2. Use the requirements document as the only input. Provide only natural-language requirements, without a predefined file list or function signatures. This forces the agent to perform the full analysis and localization process.
3. Use executable acceptance criteria as the judging standard. Rely on the pass/fail results of unit tests and integration tests as an objective signal of delivery success, rather than subjective scoring.
4. Cover multiple configuration combinations. Evaluate different models, retrieval strategies, and tool combinations separately to determine which stage the capability gap originates from.

A schematic evaluation flow can be expressed in pseudocode:

def evaluate_agent(repo, requirement_doc, agent_config):
    # 1. Provide only the requirements document and the natural-language description
    task = build_task(repo, requirement_doc)
    # 2. The agent executes the full chain: understand -> locate -> modify -> verify
    result = agent_config.run(
        repo_snapshot=task.repo_snapshot,
        requirement=task.requirement,
        tools=["search", "read_file", "edit_file", "run_tests"]
    )
    # 3. Use real tests as the objective acceptance signal
    passed = run_hidden_tests(task.repo_with_changes, task.acceptance_tests)
    # 4. Record intermediate performance across perception and implementation for attribution
    return {
        "end_to_end_success": passed,          # Whether delivery succeeded end to end
        "localization_accuracy": result.locate_score,  # Perception: localization accuracy
        "change_consistency": result.consistency_score # Implementation: change consistency
    }

The key to this design is to treat delivery as a result that must be proved, not as a seemingly polished generation.

5. Practical Lessons for Developers and Enterprises

A low delivery rate is itself a valuable signal. In engineering practice, it suggests a more pragmatic strategy:

• Provide context properly: During requirement review, fill in constraints, impact scope, and acceptance criteria as much as possible to reduce the agent’s need to guess when information is missing.
• Use retrieval instead of overloading context: For very large codebases, code search and semantic indexing can provide the model with precise relevant fragments. This is more effective than stuffing in more irrelevant content.
• Keep humans in the loop: Let agents handle localization and first drafts, while people familiar with the system review key decisions and final merges. This creates human-AI collaboration.
• Make tests the only judge: Use continuous integration and automated testing as the source of truth for verification. Agent output should count only if it passes real tests. This helps expose hallucinations and prevents regressions.
• Deliver in small steps and verify quickly: Break large requirements into independently verifiable small changes. This shortens the feedback loop and reduces the risk of losing control over cross-module changes.

Conclusion

Placing an AI agent into a real system of about 2.4 million lines of code and challenging it with a roughly 5,000-word requirements document essentially brings “the daily work of engineers” into the evaluation arena.

The core insight is that agents being able to call tools and write code is only part of the capability puzzle. Whether they can understand the constraints of a system, and whether they can make and prove correct changes amid cross-module complexity, is the dividing line between “usable” and “deliverable.”

For developers, this is both a warning and an opportunity. Rather than waiting for a fully automated “black box” delivery system, it is better to build an engineering system based on precise context, objective verification, and human-AI collaboration. The value of the evaluation is not to produce a nice-looking score, but to clearly expose capability gaps and point toward the correct path for putting AI agents into real production environments.