How to Choose a Code Review LLM: A Practical Evaluation and Selection Framework

    Public leaderboards rarely tell you which code review LLM actually fits your codebase. This article argues that model selection should be a repeatable engineering process rather than a one-time leaderboard lookup, and presents a framework called Coding Plan. It walks through building a reproducible task set from real merge requests, running evaluations under consistent real-world conditions, scoring quality with multi-dimensional metrics, estimating cost to decide a routing strategy, and setting up regression gates for continuous calibration. It also covers common pitfalls. The goal is not the strongest model, but the one with the best balance of quality and cost for your review workflow.

    1. Why You Shouldn’t Pick a Code LLM from Leaderboards Alone

    Capability-comparison articles are updated almost monthly, and the models they cover span light, standard, and flagship tiers. The question readers ask most often is “So which one should I choose?” If the answer is just a restatement of a ranking, its value is limited, for three reasons.

    First, the leaderboard doesn’t match your workload distribution. Public benchmarks are usually carefully curated, with a balanced spread of difficulty and clearly bounded problems. Real enterprise codebases are full of business constraints, legacy baggage, and private dependencies, along with a large volume of low-profile but high-frequency changes like adding logs, adjusting configs, and aligning interfaces.

    Second, evaluation criteria differ substantially. Benchmarks vary in several ways: whether multiple sampling is allowed, whether test feedback is used, which prompt template is applied, and how the tool-call budget is set. Comparing scores side by side without accounting for these premises leads to wrong conclusions.

    Third, results age quickly. Models iterate on a weekly cadence, so a leaderboard reflects a snapshot at a single point in time. Engineering selection cycles are often measured in quarters, which makes reusing old data risky.

    A safer approach is to treat selection as ongoing work and build a reproducible evaluation and decision mechanism. This article calls that mechanism the Code Review LLM Evaluation and Selection Practice, or Coding Plan for short. Note that this article cites no specific ranking scores; it discusses only the method itself.

    2. The Overall Coding Plan Framework

    Coding Plan is organized around the code review scenario and forms a closed loop of five stages:

    1. Task modeling: define the scope of the code change, its context boundaries, and the acceptance criteria;
    2. Task set construction: distill reproducible evaluation samples from real merge requests;
    3. Evaluation execution: run the models in an isolated environment and score them automatically;
    4. Metric system: balance quality, efficiency, stability, security, and cost;
    5. Selection and regression: set a routing strategy per scenario and re-verify automatically whenever a model is upgraded.

    At its core, the framework decomposes code review capability into four observable abilities:

    • Context understanding: can it read callers, interface contracts, and surrounding modules;
    • Problem localization: can it distinguish real defects from surface-level style issues;
    • Actionable suggestions: are the fixes specific enough to apply directly;
    • False-positive control: can it resist the urge to raise comments for the sake of commenting.

    Traditional review and LLM-based review each have weaknesses:

    Dimension Traditional rule-based tools LLM review
    Consistency High; rules are deterministic Relatively lower; needs fixed sampling parameters
    Coverage Limited to known rule patterns Can surface logic defects, edge-case gaps, and readability issues
    Context understanding Weak; mostly single-file or local analysis Stronger; can correlate call chains across files
    False-positive control Clear rules; false positives are predictable Needs prompt constraints and post-processing filters
    Expansion cost New rules must be written by hand New review dimensions mostly by adjusting prompts and the task set

    So the selection goal is not to find the “strongest model,” but the one with the best combined quality and cost for your codebase and review workflow. This mirrors the separation of concerns in software engineering: like a coding plan, first define the scope and dependencies, then execute and verify. Applied to a code review LLM, it means first defining the review scope and context dependencies, then calling the model to perform the review and verifying its conclusions.

    3. Step 1: Build a Reproducible Task Set

    1. Where samples come from. Start by collecting samples from your team’s historical merge requests, especially:

    • changes that triggered a defect rollback after release;
    • changes that a human reviewer flagged with substantive issues;
    • changes that took several rounds of discussion to reach agreement.

    These samples best represent the risks your team actually cares about. You can also add samples with deliberately injected defects to test recall.

    2. Sample structure. Each sample should include at least the following fields:

    • a unique identifier;
    • the change type (bug fix, new feature, refactor, config change, etc.);
    • a snapshot of the code before the change;
    • the diff;
    • frozen context, i.e., the set of files the model may view that will not change;
    • a human-annotated list of issues with severity levels;
    • the acceptance method.

    3. Size and layering. To start, prepare 100–200 samples per change type. For difficulty, roughly distribute them 3:5:2 across simple, medium, and complex so results are statistically meaningful. For example, if the detection rate for the same defect differs by more than 20 percentage points across two task sets, the conclusion isn’t strong enough to support a decision.

    4. Contamination control and reproducibility.

    • During data cleaning, uniformly strip comments, identifier-like fields, and project code names from the documentation, so models answer by understanding the code rather than guessing where it came from;
    • Keep an internal holdout set that is never published and is rotated periodically, as the basis for final acceptance;
    • Record the code version for each sample and pin dependency versions and the runtime environment so results can be reproduced;
    • When internal code is involved, use desensitized or synthetic equivalent samples.

    Risk note: Before sending internal code to an external model service, you must complete compliance approval, desensitize sensitive information, and encrypt data in transit. When core code or highly sensitive industry data is involved, evaluate options such as private deployment or inference in a dedicated environment.

    4. Step 2: Evaluate Models Under “Real Working Conditions”

    1. Standardize the context supplied. For the same change, feeding the model different context can produce very different results. Standardize these rules:

    • which files to provide (the changed files, direct callers, relevant interface definitions, necessary tests);
    • how they are organized (by directory structure or by call relationships);
    • the upper bound on the context budget.

    2. Standardize the prompt and output format. Ask the model to output in a structured form, for example:

    1. {
    2. "findings": [
    3. {
    4. "file": "src/order/service.py",
    5. "line": 42,
    6. "severity": "major",
    7. "category": "boundary handling",
    8. "message": "Quantity of 0 is not validated, which may cause a division-by-zero error",
    9. "suggestion": "if quantity <= 0: raise ValueError(...)"
    10. }
    11. ]
    12. }

    Structured output lets the evaluation script parse and compare results directly, and makes it easy to plug into downstream workflows.

    3. Run evaluations in containerized isolation. Run each sample in its own container with limits on runtime and resource usage. All models face exactly the same input, which removes interference from environment differences.

    4. Re-test and repeated sampling. Generative models are stochastic; the same model can reach different conclusions for the same input. Pin the sampling parameters, evaluate each sample 3–5 times, use the mean pass rate to measure effectiveness, and use the variance to measure stability.

    A sketch of the evaluation framework:

    1. import os
    2. import statistics
    3. import subprocess
    4. import tempfile
    5. from dataclasses import dataclass
    6. # Pass JSON through standard input so prompt content is not executed as command-line arguments
    7. HARNESS = r'''
    8. import json, sys
    9. payload = json.load(sys.stdin) # {prompt, files, budget}
    10. findings = call_model(**payload) # only this adapter layer changes for different models
    11. json.dump(findings, sys.stdout, ensure_ascii=False)
    12. '''
    13. @dataclass
    14. class Sample:
    15. sample_id: str
    16. instruction: str # change description (desensitized)
    17. diff: str # code diff
    18. context: dict # frozen context: file path -> content
    19. gold: list # human-annotated issues (with file/line/category/severity)
    20. def run_once(sample: Sample, timeout: int = 60) -> list:
    21. """Run one review in an isolated process; timeout or error is treated as no output"""
    22. with tempfile.TemporaryDirectory() as workdir:
    23. try:
    24. proc = subprocess.run(
    25. [os.sys.executable, "-c", HARNESS],
    26. input=json.dumps({
    27. "instruction": sample.instruction,
    28. "diff": sample.diff,
    29. "files": sample.context,
    30. }),
    31. capture_output=True, text=True, timeout=timeout,
    32. cwd=workdir,
    33. )
    34. return json.loads(proc.stdout)["findings"]
    35. except (subprocess.TimeoutExpired, json.JSONDecodeError, KeyError):
    36. return [] # invalid output is also a failure, so missed detections are not hidden
    37. def evaluate(samples, judge, repeat: int = 3) -> dict:
    38. per_sample = []
    39. for s in samples:
    40. runs = [judge(s, run_once(s)) for _ in range(repeat)] # multiple samples
    41. per_sample.append({
    42. "recall": statistics.mean(r["recall"] for r in runs),
    43. "precision": statistics.mean(r["precision"] for r in runs),
    44. })
    45. n = len(per_sample)
    46. return {
    47. "recall": sum(p["recall"] for p in per_sample) / n,
    48. "precision": sum(p["precision"] for p in per_sample) / n,
    49. }

    Infrastructure such as the evaluation sandbox, container runtime, and logging services can be provisioned on demand through cloud platforms (for example, Baidu AI Cloud, which offers general-purpose capabilities such as containers, object storage, and logging and monitoring), while permission and key-management capabilities help protect code and credentials.

    5. Step 3: Measure Review Quality with Multiple Metrics

    1. Detection rate (recall). Number of issues found by the model ÷ total number of human-annotated issues. It reflects coverage and is the most central code review metric.

    2. Acceptance rate (precision). Number of comments developers judge as “must fix” ÷ all comments the model produced. Too many false positives quickly erode developer trust and lead to results being ignored out of habit.

    3. Critical-issue miss rate. Among annotations with severity “high,” the proportion not found by any single sample. These issues are the costliest and should be tracked separately.

    4. Critical-issue severity accuracy. The proportion of issues where the model’s assigned level exactly matches the human annotation. Accurate severity assignment helps developers prioritize quickly.

    5. False-positive distribution. Count false positives by issue category to identify which directions the prompt needs to constrain most.

    6. Critical-issue prioritization rate. The number of non-critical issues ranked ahead of a critical issue in a single review. It measures whether the model places the risks that most need attention where they are most visible.

    7. Stability. The proportion of samples in which the same issue is detected across repeated runs. Poor stability directly amplifies miss risk.

    8. Efficiency and cost.

    • Average output length per item and average review time (P95);
    • Cost per review, i.e., input and output usage each multiplied by their respective unit price and summed;
    • It’s worth watching input-side cost too. In review tasks, context is often far longer than output, so how you organize context can affect the bill more than switching models.

    These metrics can be computed automatically by a parser and a judge:

    • Parser: validates output format and filters out results that can’t be parsed;
    • Judge: compares results using line-and-row matching and semantic equivalence (issues of the same kind count as a hit if the location and fix direction agree);
    • Semantic equivalence can be assessed with text similarity or “model-as-judge,” but it must first be calibrated against human annotations and periodically spot-checked.

    Review quality is determined jointly by “detection rate, false-positive rate, and response speed.” Looking only at detection rate tends to select models that “raise many comments, most of them useless”; looking only at speed risks missing critical defects. In practice, first set weights for each metric (for example, detection rate 0.4, acceptance rate 0.3, critical-issue miss rate 0.2, latency 0.1), then aggregate them with weights and rank. The team should set the weights itself based on its quality requirements and process tolerance.

    6. Step 4: Estimate Cost and Decide the Invocation Strategy

    The cost of a single review can be approximated as:

    1. C ≈ Pin × Cin + Pout × Cout + Σ Pr × Cr
    • Pin, Pout: input and output unit prices;
    • Cin, Cout: average input and output usage;
    • Pr, Cr: probability and corresponding cost of the r-th retry.

    Two things are worth noting:

    • Review output is usually far shorter than input, so the impact of the output unit price is often smaller than that of the input unit price and the context length;
    • For models with unstable output format, invalid outputs trigger retries, and the hidden cost can be much higher than expected.

    A cost estimate example:

    Model tier Input price (relative) Output price (relative) Avg input (tokens) Avg output (tokens) Invalid-output retry rate Cost per run (relative)
    A 1 4 8000 600 5% 1.084
    B 3 12 8000 600 5% 3.252
    C 5 20 8000 600 2% 5.200

    The unit prices and usage figures in this table are illustrative values used only to demonstrate the calculation method; they do not represent the pricing of any actual product. Actual selection should follow the billing rules published by the service provider.

    With similar task volumes, a 5× difference in unit price leads to roughly a 5× difference in cost per run. Unit price isn’t the only factor: even at a higher price, a model with fewer invalid outputs may be more economical.

    Combined with evaluation results, a common strategy is tiered usage:

    • Routine changes: use a light model to give quick feedback;
    • Core modules or high-risk changes: have a flagship model re-review, or use “dual-model cross-review”;
    • Models with unstable output format: increase the retry count and tighten output constraints, reflecting the extra cost in the metrics.

    Which tier to use and whether to retry should be decided by evaluation data. The same model can perform very differently across change types, so set the strategy per change type rather than globally.

    7. Step 5: Set Up Regression Gates and Calibrate Continuously

    Models iterate frequently, and today’s conclusion may be obsolete next month. Connect the evaluation pipeline to continuous integration as a “regression gate” for model upgrades. Each time you switch model versions or adjust the prompt, the system automatically runs the full task set and outputs a comparison report.

    A minimal working gate looks like this. It only aggregates metrics and produces a verdict; the judging details can be adjusted to your situation.

    1. Evaluation input: model=candidate model, prompt_version=v3, dataset=review_set_2026Q4
    2. ↓
    3. Execute: containerized sandbox · repeat each sample 3 times · structured parsing
    4. ↓
    5. Compute: detection rate / acceptance rate / critical-issue miss rate / P95 latency / cost per run
    6. ↓
    7. Gate rules (example; thresholds set by the team)
    8. ├─ detection rate ≥ baseline × 0.95
    9. ├─ critical miss rate ≤ baseline × 1.10
    10. ├─ acceptance rate ≥ baseline × 0.90
    11. ├─ P95 latency ≤ baseline × 1.20
    12. └─ cost per run ≤ baseline × 1.10
    13. ↓
    14. All pass → move to canary (try on a small set of merge requests)
    15. Any fails → stop the upgrade, output the gap details

    Three points matter in practice:

    1. Rules first, data second. Set the thresholds before evaluation begins to avoid cherry-picking favorable data afterward.
    2. Keep a canary stage. Passing the gate doesn’t mean full rollout. First try it on some repositories, and collect whether developers accept or ignore the comments as supplementary evidence.
    3. Keep updating the task set. Periodically feed newly surfaced human-review issues and production defects back in as evaluation samples, and rotate the holdout set to prevent the task set and the model from “co-overfitting.”

    8. Common Pitfalls and Improvement Suggestions

    Pitfall 1: Copying leaderboard scores. Leaderboards are only a first-pass reference; the final conclusion must come from a task set that matches your own workload distribution.

    Pitfall 2: Looking only at the miss rate. A low miss rate may just mean the model raised everything it could, mixed with a large number of useless comments. The miss rate must be read together with acceptance rate and critical-issue prioritization rate.

    Pitfall 3: Ignoring context supply. Many judgments of “the model isn’t good enough” are actually caused by insufficient context or missing call chains. Fix the context strategy in evaluation first, align the conditions, and only then compare models.

    Pitfall 4: Weak output-format constraints. Unstructured output is both hard to score automatically and hard to plug into the development workflow. Structured constraints are both a prerequisite for evaluation and a requirement for engineering adoption.

    Pitfall 5: Treating review as one-time work. Both the codebase and the models evolve, so a selection conclusion without a regression gate quickly becomes obsolete.

    9. Conclusion

    Choosing a code review LLM is fundamentally a data-driven engineering problem, and it can borrow the separation-of-concerns idea from software engineering:

    • Keep separate the modeling of review scope and context dependencies, the construction of evaluation tasks, the execution of the model, and the verification and gating of review conclusions;
    • Let each stage evolve and be evaluated independently, then connect them into a closed loop with data.

    This approach requires little investment. A small set of real samples plus an automated evaluation pipeline can usually be built in one or two weeks, yet it can continuously answer “Is this model / this version suitable for our code review workflow?” Rather than chasing the ranking in each new capability report, it’s better to master a method that keeps recalibrating itself as models iterate.