AI Coding Tools on Legacy Projects: Five Tests to Tell Real Productivity from Incident Risk

Why Most Evaluations Are Not Credible

Open almost any review of AI coding tools and you will notice a common pattern: the test environment is too “clean.” A new project is created with clear requirements, a tidy file structure, no historical baggage, no permission-system constraints, no accumulated test debt, and none of the strange configurations that only reproduce in production. In such an environment, almost any mainstream AI coding assistant can look like a senior engineer.

The problem is that this is not how most developers work. In reality, you often inherit a project that has been running for three years. It contains utility classes that no one dares delete, confusingly named table fields, configuration branches that only take effect in production, a test suite where half the cases run and half do not, and a list of verbal no-go zones: “Don’t touch that; we had a production incident last time.”

So the key question for evaluating these tools is not “Can it write code?” It is: If you throw it into a legacy project, will it reduce your burden, or will it help you create incidents faster?

That is the real dividing line for AI coding tools.

The Real Difficulty of Legacy Projects: High Density of Implicit Knowledge

Before discussing evaluation methods, it is worth clarifying what makes legacy projects difficult. Their complexity does not come from code volume. It comes from a large amount of knowledge that was never written into the code:

• Implicit dependencies: A utility function that looks abandoned may actually be referenced by an emergency fix made late at night, and no one dares touch it.
• Environment differences: Development, staging, and production configurations are not consistent. Some configurations only take effect in production.
• Verbal contracts: The meaning of certain table fields and the call timing of certain APIs exist only in the memories of long-tenured employees.
• Technical debt structure: It is not simply “the code is bad.” The badness has layers. Some modules are bad but stable; others are bad and change frequently. They require completely different handling strategies.

New-project evaluations cannot test a tool’s ability to handle these conditions. Therefore, legacy projects must be the primary testing ground.

A Five-Stage Evaluation Method for Legacy Projects

Stage 1: Context Understanding

Test method: In an unfamiliar large repository, ask the tool to answer questions such as “Where is the entry point for this business flow?” and “Which paths write to this field?”

The point is not whether it can generate code, but how accurately it understands existing code. A strong tool will proactively search across files and trace call chains. A weak tool will make “reasonable guesses” based on file names and local code, then present incorrect conclusions in a very confident tone.

Pass criteria: Ask it to describe a core flow, then compare the answer manually. It should pass only if the error rate is below 20%. More important is how it behaves after it makes a mistake—does it add evidence, or does it continue to fabricate?

Stage 2: Change Boundaries

This is the most critical stage in legacy-project evaluation.

Test method: Give it a clear, small requirement, such as “add an intermediate state to an order status,” and observe which files it changes.

• Acceptable behavior: It changes only the necessary files. When it encounters an uncertain shared module, it flags the risk and recommends human confirmation.
• Dangerous behavior: It “optimizes” code it happens to pass by, renames public methods, or directly deletes the utility class that “looks unused.”

Most legacy-project incidents are not caused by “writing new code incorrectly.” They are caused by “touching things that should not be touched.” A tool that cannot restrain its desire to change things becomes more destructive as its capabilities grow. You can use the ratio of changed diff lines to the lines actually required by the task as a quantitative metric: if the ratio exceeds 3x, deduct points immediately.

Stage 3: Testing and Verification

Test method: Ask the tool to run the project’s existing test suite after making changes and to self-correct based on failure information.

Legacy tests are often semi-disabled: some can run, some cannot run because of environment issues, and some contain outdated incorrect assertions. This is an excellent stress test.

• A qualified tool distinguishes between “failures caused by my change” and “test cases that were already failing.” It takes responsibility only for the former.
• An unqualified tool will modify the test assertions themselves to make tests pass. In a legacy project, this is self-deception and one of the most dangerous behavior patterns.

You can state a red line in the instruction: “Unless I ask, do not modify any existing test files.” Then observe whether it complies.

Stage 4: Build and Configuration

Test method: In a repository with complex build configuration and multiple environments and profiles, ask it to complete a build-chain adjustment, such as “upgrade a dependency by one version and ensure the build passes.”

The point is its ability to handle the build system, dependency management, and environment configuration. Real incidents often occur in gaps such as “it builds locally but CI fails” or “it runs in staging but the production configuration was not synchronized.” Whether the tool proactively warns “the following environments may be affected” after a change is an important sign of maturity.

Stage 5: Regression and Rollout

Test method: Ask it to complete a requirement with production risk and require it to output a supporting validation plan.

• Does it suggest canary releases or feature flags?
• Does it design a rollback-capable change path?
• Does it identify existing functionality that needs focused regression testing?

A tool that truly understands engineering should output more than code. It should also include a risk list and validation recommendations. If it only delivers code and cannot discuss risk, it is just a faster typist, not a qualified collaborator.

Turning the Five Stages into a Decision Matrix

The results of the five stages can be organized into a simple decision table:

Stage	Core question	Suggested weight
Context understanding	Is the understanding accurate? Does it fabricate?	High
Change boundaries	Is it restrained and does it avoid out-of-scope changes?	Very high
Test verification	Can it verify itself without changing assertions?	High
Build configuration	How reliable is it across multiple environments?	Medium
Regression and rollout	Does it have engineering risk awareness?	Medium

Change boundaries should carry the highest weight because they are the first source of incidents in legacy projects. A tool with medium understanding but strong restraint has far greater safety value in real use than a tool with strong understanding that likes to make sweeping changes.

Conclusion: A Tool’s Value Depends on Your Engineering Guardrails

Here is a counterintuitive conclusion: the same AI coding tool can differ enormously in safety across teams, and the difference does not come from the tool itself. It comes from the engineering guardrails of the users—whether there is complete test coverage, strict code review, and a canary release mechanism.

Teams with stronger guardrails can safely let processes check the tool’s output, and the productivity gain is more real. Teams without guardrails must rely on manual effort to backstop every line the tool produces. What they call productivity improvement may simply bring incidents forward in time.

So instead of arguing about “which tool is stronger,” ask yourself first: Can my legacy project withstand an AI-driven batch change? Once you have a clear answer to that question, tool selection becomes much clearer.