Delivering Large-Scale Software Requirements with AI Agents: Capability Boundaries and Engineering Practices

1. The Problem: The Gap Between Calling Tools and Delivering Features

Today, AI agents can use tools proficiently, read and write files, run test commands, and complete fairly complex code generation tasks within a single file. However, when the target becomes a long-lived, large-scale software system, the problem changes fundamentally. Product managers usually submit new requirements as feature proposals written in natural language. Developers, whether human or agent, receive neither a file list nor a function signature. The executor must first understand the business constraints, trace dependencies across modules, and then decide where to change code, how to change it, and how to prove the change is correct. The quality of these analysis steps directly determines whether delivery succeeds.

A research team proposed a benchmark that places agents in real software systems averaging about 2.4 million lines of source code. It evaluates the full process from feature proposal to module delivery and focuses on two capabilities. The first is perception: the ability to map natural-language requirements accurately onto system structure. The second is implementation: the ability to make cross-module changes that keep system behavior correct. Results show that even current strong agents achieve a full delivery rate of only about 14%. This figure exposes a gap that is easy to overlook. An agent’s tool-use ability and its system-level delivery ability are far apart.

2. Why Large-Scale Requirement Delivery Is So Difficult

2.1 No Structured Bridge Between Requirements and Code

In small projects, requirements and code files roughly correspond one to one, so an agent can locate change points quickly through keyword matching. In large systems, a single feature proposal may require coordinated changes across the configuration, service, data access, and interface layers. Requirement text does not name a function in a specific module. It describes behavior changes visible to users. The agent must map semantics to structure on its own, and errors at this step amplify through later stages.

2.2 Search Space Explosion in Dependency Tracing

Large systems have complex module dependencies, including many indirect calls, dynamic bindings, and configuration-driven behavior. If an agent relies only on local text search to locate change points, it easily misses implicit dependencies. This leads to incomplete changes or regression defects. One major reason for the low delivery rate in the benchmark is that agents failed to cover all affected code paths during dependency analysis.

2.3 Missing Verification Loop

After making changes, human developers run relevant tests, check logs, and compare behavior before and after the change. If an agent lacks a systematic verification strategy and declares a task complete based only on code that looks correct, delivery quality cannot be assured. In large systems, showing that a change is correct requires evidence. That evidence comes from targeted test execution and result analysis.

3. Breaking Down the Agent Delivery Workflow

A complete system-level requirement delivery workflow can be divided into the following stages. Each stage has its own technical challenges.

Stage 1: Requirement parsing and constraint extraction. The agent must extract explicit constraints, such as input and output formats and boundary conditions, and implicit constraints, such as performance and compatibility requirements. In practice, structured prompts can make the agent produce a constraint checklist, which is then cross-checked against the later change plan.

Stage 2: System structure perception. The agent must build an overall understanding of the codebase, including module boundaries, entry points, configuration items, and data flow. Common methods include retrieval-based code localization, call-relationship analysis, and inferring module boundaries from build scripts and configuration files.

Stage 3: Change planning. After identifying candidate change points, the agent should produce an explicit change plan. The plan lists each file to modify, the reason for each change, and the expected impact scope. A change plan makes the reasoning reviewable and reduces the risk of omissions.

Stage 4: Code modification and self-consistency checks. When making changes, the agent must keep files consistent with one another. For example, after an interface signature changes, all callers must be updated. After the changes, a consistency self-check can ask the agent to reread the affected files and verify reference relationships.

Stage 5: Verification and evidence collection. The agent should run the test cases related to the change, analyze failures, and iterate on fixes. The key to this stage is targeting. Instead of blindly running the full test suite, the agent should select tests based on the change’s impact scope.

4. Collaboration Patterns for Multi-Agent Systems

A single agent working on a large system can lose key information during long contexts and complex reasoning. Multi-agent setups reduce this problem by dividing the work. Common collaboration patterns include:

• Separating planner and executor: One agent handles requirement parsing and change planning, while another performs the code changes. The planner outputs a structured change plan, and the executor implements it item by item. This reduces the reasoning load during execution.
• Separating retriever and modifier: A dedicated retrieval agent locates relevant files and dependencies in the codebase, organizes the results into a context summary, and passes it to the modifying agent. This improves localization accuracy.
• Separating modifier and reviewer: A review agent independently checks the completeness and consistency of the changes, simulating a code review. It catches missed callers and inconsistent interface usage.
• Verifier role: A dedicated verification agent designs the test strategy, runs tests, and analyzes results, forming a “modify, verify, fix” loop.

The benchmark compared several combinations. Configurations with an explicit planning step and an independent verification step achieved a noticeably higher full delivery rate than a single agent executing directly. This suggests that the design of the workflow structure matters more than simply adding more agents.

5. Engineering Recommendations

5.1 Build Infrastructure for Change Impact Analysis

Before introducing agents, teams should have basic code indexing and dependency analysis capabilities, such as module dependency graphs and interface call-relationship tables. This infrastructure can serve as retrieval tools for agents and significantly reduce the difficulty of structure perception.

5.2 Require Explicit Intermediate Artifacts

Require agents to produce structured intermediate artifacts at key points: a constraint checklist, a change plan, an impact scope list, and a verification report. These artifacts serve as quality checkpoints and as handles for human intervention.

Example change plan (structured output):
- Requirement summary: Add validation logic for new configuration items in a service
- Files to modify:
  1. Configuration parsing module: add validation branch
  2. Service initialization module: call the validation function
  3. Error handling module: add error code mapping
- Impact scope: configuration loading path, service startup flow
- Verification strategy: run configuration parsing unit tests and service startup integration tests

5.3 Align Verification Strategy with Change Scope

Verification should select test sets based on the change’s impact scope instead of always running the full test suite. Agents can first generate a mapping from changes to tests, then execute the tests by priority. For paths that cannot be covered, mark them clearly as “unverified risk points” so that humans can review them.

5.4 Defining the Boundary Between Humans and Agents

At the current stage of the technology, agents are best suited to analysis support, draft generation, and verification execution. Final confirmation of high-risk changes still requires human oversight. A sound approach is to design delivery as a collaboration in which agents produce the work and humans review it. This improves efficiency while controlling risk.

6. Conclusion

Delivering requirements with agents in large software systems is fundamentally a closed-loop engineering problem spanning perception, planning, implementation, and verification. The 14% delivery rate in the benchmark does not mean agents lack value. It identifies the gap between current capabilities and the demands of system-level delivery: mapping requirements to structure, completeness of dependency tracing, and rigor of the verification loop. With a sound multi-agent division of labor, explicit management of intermediate artifacts, and verification strategies aligned with change scope, teams can meaningfully improve the usability of agent-based delivery with current capabilities. As code understanding and planning continue to improve, this number still has considerable room to rise.