Evaluating AI Coding Agents on Real-World Software Engineering Tasks

1. Why Coding Evaluation Must Reflect Real Engineering

Traditional coding benchmarks often ask models to complete a function, fix a few syntax errors, or write a short program with clearly defined inputs and outputs. These tasks are easy to organize and score automatically, but they remain far removed from everyday software development.

In a real repository, a single requirement may span multiple directories, modules, and configuration files. Developers must understand the requirement, trace call chains, identify existing abstractions, assess compatibility, and ensure that changes do not break existing functionality. Even code that compiles may implement only part of the required behavior, miss edge cases, handle errors poorly, or violate architectural constraints.

Evaluation for real-world software engineering should therefore go beyond asking whether a model can write code. It should measure whether the model can:

1. Quickly understand the structure and responsibility boundaries of an unfamiliar codebase.
2. Turn natural-language requirements into an actionable implementation plan.
3. Make changes across files while keeping interfaces, types, and configuration consistent.
4. Proactively run tests and iterate based on failures.
5. Control the scope of changes and avoid unrelated refactoring.
6. Address compatibility, security, performance, and maintainability requirements.

This is closer to comprehensive acceptance testing of a software engineering agent than to a static review of a single generated answer.

2. Designing the Task Set

Use Original Tasks to Reduce Data Contamination

Public bug reports, historical commits, and community discussions can easily enter model training data. If an evaluation simply repackages an existing fix, a model may reproduce a memorized answer, inflating its score.

A more reliable approach is to select a repository, design new requirements, and have evaluators independently implement a reference solution. Original tasks should not be artificial problems without practical value. They should introduce reasonable changes within the repository’s existing architecture and domain, such as:

• Adding input validation and error handling to an existing interface.
• Introducing a new data transformation rule.
• Extending configuration while preserving default behavior.
• Fixing inconsistent state across modules.
• Handling edge cases in concurrency, caching, or resource cleanup.
• Adding observability information without changing core execution results.

Before releasing a task, evaluators should search the repository’s history for highly similar commits, patches, or discussions. Requirements may need substantive reworking, but renaming variables does not make an existing task original.

Maintain Diversity Across Repositories and Technology Stacks

If every task uses the same language, framework, or directory structure, a leaderboard may primarily measure familiarity with a particular pattern. A strong task set should cover different repository sizes, architectures, and engineering constraints, including:

• Interpreted and compiled codebases.
• Monolithic and multi-module architectures.
• Command processing, network services, data processing, and foundational components.
• Statically and dynamically typed code.
• Synchronous, asynchronous, and concurrent logic.
• Unit, integration, and end-to-end testing.

Tasks should also span multiple difficulty levels. Simple tasks can assess local code navigation and basic implementation. Intermediate tasks can emphasize cross-file changes. Difficult tasks can involve state management, compatibility constraints, or subtle edge cases.

If every task is too difficult, the benchmark will struggle to distinguish models with moderate capabilities. If every task is too easy, leading models will quickly reach saturation.

Specify Behavior Without Revealing the Implementation

Task descriptions should define inputs, outputs, error conditions, compatibility requirements, and prohibited behavior clearly enough to support reliable evaluation. Ambiguous requirements turn score differences into differences in how models guess the intended behavior.

However, descriptions should not tell the model exactly which file to edit, which internal function to call, or which implementation structure to use. Otherwise, the benchmark becomes an exercise in following a prescribed recipe.

A useful specification focuses on externally observable behavior:

When configuration is missing, preserve the existing default behavior. When a configuration value is invalid, return a clear error. The new logic must not change execution results for previously valid configurations.

This is testable while leaving room for code search, design, and implementation decisions.

3. Building Reproducible, Isolated Execution Environments

Repository-level evaluations typically let models read code, modify files, and run tests. Each run must therefore execute in an isolated environment to prevent interference between tasks and reduce security risks.

A complete execution environment should include at least:

• A fixed repository snapshot.
• Locked dependency versions.
• Clearly defined build and test entry points.
• CPU, memory, disk, and execution-time limits.
• External network access disabled or tightly restricted by default.
• Write access limited to the task workspace.
• Complete records of commands, outputs, file diffs, and resource consumption.

Environment consistency is essential. If the same patch produces different results under different dependency versions, model comparisons are not fair. Repository snapshots, runtime images, and test scripts should have immutable identifiers recorded alongside evaluation results.

The evaluation must also distinguish environment failures from model failures. Dependency download errors, flaky baseline tests, and insufficient execution-node resources should not automatically count as unsuccessful model attempts.

Before formal evaluation, run the baseline tests repeatedly against the unmodified repository to identify and exclude unstable tasks.

4. Evaluating the Full Problem-Solving Loop

Real coding capability appears not only in the final patch but also in the process used to reach it. A typical workflow includes:

1. Searching directories, symbols, and call relationships.
2. Reading relevant implementations and existing tests.
3. Developing a modification plan.
4. Editing one or more files.
5. Running targeted tests or static checks.
6. Revising the implementation based on errors.
7. Running the full regression suite.
8. Summarizing changes and remaining risks.

The evaluation framework should record these actions, but more steps should not automatically imply stronger capability. A model may repeatedly read the same file, run irrelevant commands, or become trapped in an unsuccessful loop.

Process data is primarily diagnostic. Behavioral correctness should remain the core basis for the final result.

An evaluation run can be represented by the following record:

Task identifier
Repository snapshot identifier
Environment version
Model configuration summary
Action trace
Final file diff
Public test results
Hidden test results
Execution time and resource consumption
Failure stage and error category

These records support reproduction and help distinguish misunderstood requirements, code navigation failures, implementation errors, test gaming, and infrastructure failures.

5. Testing and Acceptance Criteria

Baseline Regression Tests

Baseline regression tests verify that existing functionality remains intact. A model has not fully completed a task if its implementation satisfies the new requirement but breaks existing tests.

However, existing tests usually cover historical behavior rather than new requirements. Task-specific acceptance tests are therefore necessary.

Hidden Acceptance Tests

Hidden tests verify the core behaviors and edge cases described in the task while reducing opportunities to hard-code answers around visible test files. They should cover:

• Typical success paths.
• Null or empty values, extreme values, and invalid inputs.
• Error propagation and resource cleanup.
• Compatibility with existing interfaces and configurations.
• Interactions across modules.
• Repeated calls, concurrency, or state recovery.
• Side effects that must not occur.

Tests must not require the internal structure of a particular reference patch. An implementation should be accepted if it satisfies the published requirements, even when it uses a different algorithm or file organization.

Acceptance should validate the behavioral contract, not demand a copy of the reference solution.

Preventing Test Gaming

Common forms of test gaming include modifying tests, skipping assertions, hard-coding example outputs, suppressing exceptions, or returning predefined values for specific filenames.

Evaluation systems should include integrity checks such as:

• Preventing changes to test directories and critical configuration.
• Flagging large-scale deletion of unrelated files for review.
• Rejecting changes that make validation conditions always evaluate to true.
• Checking whether generated code depends on hidden environment details.
• Reviewing unusually small or unusually large patches.

A successful test-process exit code is not sufficient evidence that a task is complete. The final judgment should combine patch inspection, hidden tests, and behavioral checks.

6. Defining the Core Metrics

Pass@1

Pass@1 is the proportion of tasks completed successfully in a single independent attempt. It is the most direct primary metric. If there are N tasks and S pass on their first run:

Pass@1 = S / N

This reflects a common enterprise concern: how reliably the model completes a task on its first attempt, rather than whether extensive sampling can eventually produce a correct answer.

Repeated-Sampling Metrics

For models with stochastic behavior, multiple independent runs of the same task can reveal result variance. Repeated sampling helps distinguish consistent capability from occasional success.

However, results are comparable across evaluations only when the number of samples, inference budget, context limit, tool permissions, and stopping conditions are consistent.

Increasing the number of attempts or the compute budget may substantially improve results without improving engineering efficiency per unit of cost.

Success Rates by Task Category

An aggregate success rate can hide important capability differences. Results should also be broken down by:

• Task difficulty.
• Language and repository size.
• Local changes versus cross-module changes.
• Feature development, bug fixing, and engineering optimization.
• Tasks that require test-driven iteration versus those that do not.

These breakdowns help teams assess whether a model fits their workloads. A model with a strong overall score may still perform inconsistently in large, multi-module repositories—the very setting an organization cares about most.

Efficiency and Quality Metrics

Beyond correctness, useful measurements include:

• Average completion time.
• Input and output volume.
• Number of tool calls.
• Number of test executions.
• Number of modified files.
• Patch size in lines.
• Proportion of unrelated changes.
• Resource consumption.
• Variability across runs.

These metrics should not be reduced to a single weighted score without careful consideration. Together, they provide a capability profile.

In production, a modest increase in success rate may not justify substantially higher costs or less predictable results. A more stable approach can be the better choice.

7. Failure Categories Are More Useful Than a Score Alone

Failure classification helps guide model improvement and engineering decisions.

Requirement Understanding Failures

The model misunderstands the intended behavior or implements only part of the requirement. The code may run successfully while hidden tests fail on essential semantics.

Code Navigation Failures

The model fails to find the actual execution path and changes a similar but inactive module. Duplicate function names, compatibility layers, and indirect calls make this risk particularly significant in large repositories.

Correct New Behavior with Regression Failures

The new feature passes its targeted tests but breaks existing interfaces, default configuration, or error behavior. This indicates insufficient assessment of the change’s broader impact.

Edge-Case Failures

The main path works, but empty inputs, repeated calls, concurrent execution, or resource shortages cause errors. Such failures suggest that the model can produce a typical implementation without developing a complete understanding of its behavior.

Engineering Quality Failures

Code may pass tests while introducing obvious duplication, unrelated refactoring, unsafe permissions, sensitive-data writes, or unmaintainable implementations.

For high-risk applications, human review or static rules should make such problems explicit grounds for rejection.

8. Using Evaluation Results in the Enterprise

Public benchmark scores are a starting point, not a replacement for internal validation. Organizations differ in repository size, quality standards, dependency environments, and security requirements.

A more reliable approach is to build an internal task set using representative scenarios drawn from real development requirements with sensitive information removed.

Internal validation can proceed through three stages:

1. Offline evaluation: Use fixed repository snapshots and hidden tests to compare completion rates and consistency.
2. Shadow runs: Let the model work on real requirements, but provide its results only as references for engineers rather than merging them into production branches.
3. Controlled adoption: Allow the model to submit candidate patches, subject to automated checks, code review, and release gates.

When using general-purpose compute, container platforms, object storage, logging, or monitoring and alerting services from providers such as Baidu or Baidu AI Cloud, keep evaluation data, execution environments, and metric definitions independent of infrastructure changes. Infrastructure variation should not distort judgments about model capability.

9. Benchmark Maintenance and Version Governance

A software engineering task set is not a one-time asset. As tasks become public, training data changes, and repositories evolve, existing tests may lose their ability to distinguish model capabilities.

A clear versioning process should:

• Freeze tasks, repository snapshots, and environments for each release.
• Preserve historical versions when adding tasks to prevent misleading score comparisons.
• Record the reasons for changes to disputed tests.
• Regularly check whether tasks have leaked or entered training data.
• Add more complex tasks when leading models approach saturation.
• Retain a stable subset of anchor tasks to track long-term trends.

Score changes across benchmark versions do not automatically demonstrate capability improvements. A new version may introduce harder tasks or correct earlier environment problems.

Evaluation reports should state the benchmark version, run configuration, inference budget, execution date, and rules for handling failures.

10. Conclusion

Evaluating real-world software engineering is not about creating more programming exercises. It is about reproducing the full workflow developers follow when working in unfamiliar repositories.

A credible evaluation requires original and diverse tasks, fixed and isolated environments, behavior-focused hidden tests, safeguards against test gaming, and reproducible metrics and version management.

For engineering teams, the most useful signal is not an isolated leaderboard position. It is the model’s consistency in their own code environment, its handling of edge cases, its regression risk, and its cost per task.

Combining automated evaluation, process diagnostics, and human review provides a stronger basis for deciding whether an AI coding agent is ready for complex software engineering work.