SWE-bench Practical Guide: Choose a Version, Run the Evaluation, and Debug Failures
This guide explains what SWE-bench measures, how the Verified and Pro versions differ, how to run preparation, inference, and verification, which metrics to report, and how to debug common patch failures before using scores for decisions.
Software engineering tasks are far more complex than simply writing code. A model must read repository context, understand an issue description, find the relevant files, produce a patch, and make the tests pass. SWE-bench is a benchmark built around that process. It draws real issues from open-source repositories and asks the model to generate a fix patch that can be applied.
What SWE-bench actually measures
Each sample contains four items: a snapshot of the repository at a specific commit, the issue text, the relevant test cases, and a human-written gold patch. During evaluation, the model receives only the issue and the repository structure, then outputs a diff patch in a unified format. The evaluator applies the patch in an isolated environment and runs the tests specified for that sample.
Success depends on two kinds of tests:
- FAIL_TO_PASS: tests that fail before the patch is applied and must pass afterward. These tests show that the issue was actually fixed.
- PASS_TO_PASS: tests that already passed before the patch is applied. The patch must not break this existing behavior.
A sample is marked resolved only when both kinds of tests are satisfied. Note that the tests run are those specified for the sample, not the repository’s full test suite. This checks whether behavior is correct, which is closer to real development than comparing code similarity alone.
Choosing between Verified and Pro
According to official SWE-bench release materials, the Verified version contains 500 human-reviewed samples. Reviewers removed samples with vague descriptions, unstable tests, or dependencies on external resources, so results are more reproducible.
The Pro version targets longer, more complex tasks. Its samples come from a wider range of sources and cover more repositories and programming languages. For the exact size, filtering method, and release date, follow the official documentation. The table below compares them on dimensions that can be checked:
| Dimension | Verified | Pro |
|---|---|---|
| Sample filtering | Human review; vague, unstable, or externally dependent samples are removed | Follow the official documentation |
| Task profile | Mostly routine fixes for a single issue | Longer, more complex engineering tasks |
| Main use | Quickly check basic repair ability | Assess performance on complex engineering scenarios |
| Result stability | Relatively reproducible | Must be reproduced using the official run method |
Selection advice: to compare basic model ability quickly, start with Verified. To judge whether a model can handle cross-file, long-context tasks, then look at Pro results. Whichever version you use, state it clearly in the report.
Evaluation workflow and metrics
Evaluation has three stages: preparation, inference, and verification.
Preparation:
- Pin the repository commit and record the commit hash.
- Create an independent environment for each sample. Containers are recommended so dependencies do not contaminate one another.
- Install dependencies and run a test baseline. Confirm that FAIL_TO_PASS tests really fail and PASS_TO_PASS tests really pass.
Inference:
- The input includes the issue text, the repository structure, and any retrieval context that is required.
- Require the model to output a standard diff.
- Record the model version, temperature, random seed, context length, and number of samples.
Verification:
- Apply the patch in a clean environment. If it cannot be applied, mark the sample as failed and record the reason.
- Run the sample’s specified FAIL_TO_PASS and PASS_TO_PASS tests, with timeout and memory limits.
- Aggregate the metrics.
The example commands below use placeholders that must be replaced from the sample configuration:
git clone <repo_url> repo && cd repogit checkout <base_commit># Install dependencies according to the sample configurationgit apply --check "$PATCH_FILE" && git apply "$PATCH_FILE"# Run the tests specified for the sample<test_cmd> <F2P_tests> <P2P_tests>
$PATCH_FILE must be quoted. Model output is untrusted data and must not be concatenated directly into a shell command.
Report these metrics separately:
| Metric | Meaning |
|---|---|
| Patch apply rate | Share of samples whose patches apply cleanly |
| F2P pass rate | Share of previously failing tests that now pass |
| P2P retention rate | Share of previously passing tests that remain unbroken |
| Resolved rate | Share of samples that satisfy both test groups |
When sampling more than once, save every run. Report both the mean and the range of variation, rather than only the best run.
Common failures and how to debug them
- Truncated context: a key file never entered the input. Check the input token count and confirm that the target file is included. A practical fix is to retrieve and locate files first, then generate the patch in stages.
- Misunderstood directory structure: the wrong path was edited, or an import path is incorrect. Compare the paths in the diff headers with the real paths in the repository.
- Invalid diff format: the patch cannot be applied. Run
git apply --checkto locate the problem, and include a standard format example in the prompt. - Missed edge cases: the main path is fixed, but null values, exceptions, or concurrent branches are not handled. Inspect the inputs of failing F2P or P2P tests. Ask the model to list edge cases, and add static checks.
- Over-editing: unrelated files were changed and P2P tests broke. The prompt should explicitly require changing only the necessary files and preserving the existing style.
A reusable prompt fragment:
Output only one standard unified diff that can be applied with git apply.Change only the files required to resolve this issue, and keep the existing code style.If several solutions exist, choose the one with the smallest change that does not affect existing behavior.
A checklist for putting this into practice
- Select the issue types your team sees most often and build a private evaluation set.
- For every sample, record the repository, commit, test command, and timeout settings.
- Check test stability: include a sample only if the same patch produces consistent results across repeated runs.
- Redact issues and code that contain sensitive information, and control access.
- Periodically review passing samples by hand to confirm patch quality, not just the pass rate.
- Use public benchmarks for side-by-side comparison, and private evaluations to judge performance on your team’s real scenarios.
- Pilot on a small set of real projects and compare evaluation results with human review.
A score is only a reference. Its practical value depends on how well the model matches the team’s stack and coding standards. Results can guide decisions only after you have a local evaluation process and regularly reproduce public benchmarks.
