Evaluating LLM Code Review Capabilities: A Practical Benchmarking Framework

1. Why a Quantifiable Review Baseline Is Needed

When introducing large language models into the code review process, the most common question is not “Can we use it?” but “How well does it actually work, and to what degree?” A few demo conversations cannot answer this question: a model performing well on simple examples does not mean it can handle cross-file, cross-module changes in a real repository; a single偶然 correct suggestion does not mean its false positive rate is within an acceptable range.

Therefore, engineering teams need a reproducible, comparable, and iterable evaluation baseline. It must satisfy at least three conditions:

1. The test set is close to real engineering: it includes real defect patterns, real context scales, and real noise;
2. Scoring dimensions are interpretable: scores can be broken down into specific capability items, rather than being a black-box number;
3. Aggregation is stable: results are comparable across multiple runs and different sample batches.

This article does not evaluate or recommend any specific product. Instead, it explains from a methodological perspective how to design such an evaluation framework and discusses the most common pitfalls encountered in practice.

2. Overall Design of the Evaluation Framework

2.1 Layered Structure

A complete evaluation framework is typically divided into four layers:

• Sample layer: a collection of defect samples, normal samples, and boundary samples;
• Execution layer: responsible for driving the model, collecting outputs, and recording latency and invocation costs;
• Scoring layer: maps outputs into structured judgment results;
• Aggregation layer: applies weighted scoring by dimension and outputs a composite score with a breakdown report.

The benefit of layering is that adjustments at any layer (e.g., changing scoring rules or expanding samples) do not contaminate the implementation of other layers, making long-term maintenance easier.

2.2 Sample Construction Principles

Sample quality directly determines the validity of evaluation conclusions. In practice, the following principles are recommended:

• Balanced defect types: null pointer dereferences, resource leaks, concurrency races, boundary conditions, missing exception handling, logging and sensitive information leakage, etc., should each account for a certain proportion to avoid one category dominating the total score;
• Reasonable positive-to-negative sample ratio: testing only defect samples overestimates the model’s “detection capability.” A certain proportion of defect-free changes must be mixed in to measure false positives;
• Realistic context: samples should preserve real file structures, call chains, and dependency declarations, rather than isolated function snippets;
• Difficulty stratification: can be divided into three tiers—“explicit defects” (locally visible), “implicit defects” (requiring cross-function reasoning), and “design-level issues” (requiring understanding of architectural intent).

A common mistake is that all samples come from manually constructed small snippets. Models generally score high on such samples, but the correlation with production performance is weak.

2.3 Defining Scoring Dimensions

It is recommended to cover at least the following five dimensions, each with clear judgment criteria:

Dimension	Description	Judgment Method
Defect localization	Whether the specific location of the problem can be identified	Location hit rate
Problem description	Whether the explanation of the defect’s cause is accurate	Human or rule-based judgment
Fix suggestion	Whether the proposed modification is usable	Compilation/test verification
False positive control	Whether the model stays silent on defect-free changes	False positive rate
Output formatting	Whether the structure is stable and easy to parse programmatically	Format compliance rate

Among these, “fix suggestion” is best verified in a concrete way: apply the model-suggested patch to the codebase, run existing unit tests, and count it as valid if they pass. This is more objective than human scoring and more suitable for scale.

3. From Individual Scores to Composite Scoring

3.1 Weighted Aggregation

Scores from each dimension need to be normalized to the same scale before weighting. A usable form is as follows:

WEIGHTS = {
    "localization": 0.25,
    "description": 0.20,
    "fix": 0.30,
    "false_positive": 0.15,
    "format": 0.10,
}
def aggregate(scores: dict) -> float:
    total = 0.0
    for k, w in WEIGHTS.items():
        total += normalize(scores[k]) * w
    return round(total, 2)

Weights are not fixed: if review results ultimately enter a human review stage, the weight for false positive control should be increased appropriately; if used in an automated fix pipeline, the weight for fix suggestions should dominate.

3.2 Comparability of Scores

In practice, score differences between different models in similar ranges are often very small—for example, values like 2.68, 2.68, and 2.97. Such results need to be interpreted carefully:

• The same score does not mean the same capability; it may simply be the result of weighted dimensions canceling each other out;
• Small leads need to be judged with confidence intervals; when the sample size in a single batch is insufficient, the difference may not be significant;
• Priority should be given to the per-dimension radar chart rather than only the composite score.

It is recommended to include sample size, per-dimension scores, and fluctuation ranges in the report to avoid making selection decisions based on a single number.

3.3 Repeatability and Randomness Control

LLM outputs are stochastic, so evaluations must control variables:

• Fix decoding parameters (temperature, sampling strategy) and note them in the report;
• Repeat each sample multiple times, take the mean, and record the variance;
• Apply structured parsing to outputs; parsing failures should be counted as format non-compliance rather than discarded samples.

4. Common Issues in Engineering Deployment

4.1 Context Window and Truncation

Changes in real repositories often involve large contexts. If simply truncated during evaluation, the model’s cross-file reasoning capability will be systematically underestimated. It is recommended to explicitly annotate “required context” in samples and separately track the proportion of failures caused by truncation.

4.2 Actionability of Review Comments

Model suggestions often contain generalized statements like “validation should be added.” When scoring, a distinction should be made between “directionally correct but not executable” and “directly actionable”—the former should not receive the same score as the latter.

4.3 Pipeline Integration

The evaluation framework is best decoupled from the continuous integration process: evaluation tasks are offline batch processing, producing reports for human selection; online review is real-time or near-real-time invocation. Both share the same scoring rules, but samples and thresholds are managed separately.

5. Summary

Evaluating LLM code review capabilities is essentially an engineering measurement problem, not a one-off comparative experiment. To make conclusions solid, you need realistic samples, interpretable dimensions, stable aggregation methods, and systematic control of interference factors such as randomness and truncation. Only on this basis does a composite score have reference value for guiding tool selection and process improvement; and per-dimension data often reveals a model’s true weaknesses better than the total score.