AI Agents in Large-Scale Software Engineering: Delivery Challenges and Practical Solutions

In recent years, large language models and AI Agents have made impressive progress in software engineering. They can call external tools, modify files, run unit tests, and fix certain defects, suggesting that they may eventually become effective “digital employees.” However, the picture changes significantly when the scope expands from a single file or a small project to a long-lived software system containing millions of lines of code.

A recent academic benchmark exposed a difficult reality: when agents were given a complex requirements document of approximately 5,000 words, the best delivery completion rate among 28 tested agent architectures was only 14%. This result highlights the gap between the apparent capabilities of AI coding tools and their ability to deliver reliable changes in large-scale engineering environments.

1. The Gap Between Expectations and Reality

In everyday development, a product manager typically provides a natural-language requirements document rather than a ready-made list of files or function signatures. Engineers must perform several layers of reasoning:

1. Understand how the business requirements interact with the system’s existing constraints.
2. Trace dependencies across relevant modules.
3. Determine where changes are needed and how they should be implemented.
4. Prove that the implementation is correct.

When processing this type of unstructured input, AI Agents commonly reveal three major weaknesses.

1.1 Weak Awareness of Requirements and Constraints

Large systems contain many implicit constraints, such as global concurrency limits and module-specific error-handling mechanisms. Current coding agents can easily lose track of important details when processing requirements longer than 5,000 words.

An agent may identify the visible functional goal while overlooking underlying security, reliability, or performance constraints. The resulting code may be syntactically valid but incompatible with the existing architecture.

1.2 Difficulty Tracking Cross-Module Dependencies

Large software systems often have highly interconnected dependency structures. A seemingly simple feature may require coordinated changes across the API gateway, business logic layer, data access layer, and message queue configuration.

Agents generally perform reasonably well when modifying code within a single file. However, they still lack a comprehensive representation of the codebase as a whole. Unlike an experienced engineer, an agent may struggle to follow a call chain upstream and downstream and identify every related module that must be updated.

1.3 An Incomplete Verification Loop

Proving that a change is correct is a central part of software engineering. An agent may be able to run an individual unit test, but correctness in a large system often requires integration tests or end-to-end validation.

Without deep control over complex CI/CD pipelines and test environments, agent-generated code can remain in a dangerous state: syntactically correct, yet capable of breaking system behavior.

2. What Benchmarks Reveal About Agent Capability

Recent academic research has introduced benchmarks designed to evaluate an agent’s ability to evolve large software systems. These benchmarks assess the complete process, from understanding a feature proposal to modifying code and validating the result.

Multiple mainstream agent configurations were tested against open-source software systems containing an average of approximately 2.4 million lines of code. The agents were expected to independently complete requirement analysis, code modification, and test validation.

The results showed a sharp decline in performance for cross-module changes. When a task involved only the internal logic of a single function, some advanced agents achieved success rates above 50%. However, when a task required cross-service calls, database schema changes, and synchronized configuration updates, the delivery rate dropped to approximately 14%.

This indicates that an agent’s engineering awareness is still far behind its code-generation ability. Today’s agents are closer to highly capable typists than to system engineers with a global architectural perspective.

3. Practical Strategies for Improving Delivery Rates

The 14% result is discouraging, but it does not mean that teams have no way forward. Senior developers can build a more effective human-agent collaboration model that compensates for the agent’s limited global awareness.

3.1 Build a Precise Context-Delivery Pipeline

An agent should not be expected to search through 2.4 million lines of code unaided. Developers can combine static analysis tools with requirement analysis to extract the relevant module topology, core interface definitions, and key data structures. These artifacts can then be provided as targeted context before the agent starts implementation.

# Pseudocode example: building a context-delivery pipeline
context_input = {
    "requirement_doc": load_doc("feature_req.md"),
    "api_dependencies": extract_api_graph(target_module),
    "db_schemas": fetch_related_tables(tables=["user", "order"]),
    "coding_conventions": get_repo_guidelines()
}
agent.execute(context_input)

The objective is not to provide the entire repository. Instead, the system should supply the smallest set of high-value context needed for the requested change.

3.2 Combine Hybrid Retrieval with Code Graphs

Text-only vector retrieval, or RAG, can introduce substantial noise in complex engineering tasks. A stronger approach is to combine vector retrieval with AST-based code graph queries.

For example, when an agent needs to modify a class, the system can use the code graph to identify all known call sites and related dependencies. Those contexts can then be injected into the agent’s working set. This significantly reduces the risk of making isolated changes without understanding their cross-module impact.

3.3 Decompose Work into Small, Verifiable Loops

A 5,000-word requirement should be divided into multiple small tasks that can be independently verified. Each task should follow a short design-code-test loop.

An agent framework can orchestrate these tasks and ensure that every output is checked by automated test scripts. If validation fails, the error logs should be returned to the agent immediately for another attempt. This prevents defects from accumulating and multiplying across a longer dependency chain.

3.4 Establish a Sandboxed Automated Validation Loop

In a cloud development environment, agents should work inside independent, containerized sandboxes. The sandbox should include the required microservice dependencies, test database, and mock services.

After each code change, the agent must trigger the integration test suite inside the sandbox through an API. Only when the test results meet a predefined threshold should the change be allowed to merge into the main branch.

This mandatory validation loop is the final safeguard against subtle defects introduced by autonomous code changes.

4. Conclusion

AI Agents still have a long way to go before they can evolve from code assistants into independent delivery engineers for large software systems. The 14% delivery rate reflects the current limitations of their ability to understand complex engineering environments, while also pointing toward the direction of future development.

By integrating code graphs, improving context engineering, and strengthening automated testing loops, development teams can reduce the agents’ blind spots and improve delivery reliability. The goal is not to remove engineers from the process, but to create a robust human-agent engineering system that maximizes AI-assisted productivity without compromising system stability.