Can AI Write Static Checkers? Benchmark Pass Rates Below 50%

1. First, define the problem: why static checkers are hard

Static analysis does not run the program. Instead, it parses source code into machine-reasonable intermediate structures, such as abstract syntax trees, symbol tables, and control-flow graphs. It then layers data-flow analysis and pointer analysis on top of these structures to look for known defect patterns. A checker is a component that fixes the decision logic for one type of defect: some focus on null-pointer dereference, some on unreleased resources, and some on out-of-bounds access or integer overflow.

Writing checkers has long been expert work because the difficulty stacks up across multiple levels:

• Semantic complexity: pointer aliasing, macros and template/generic instantiation, dynamic dispatch, and reflection calls make seemingly simple questions—such as whether a variable on this path could be null—hard to decide statically.
• Theoretical limits: problems such as memory safety and out-of-bounds access are undecidable. Any tool can only trade off among analysis precision, inter-procedural depth, and runtime cost; it cannot be optimal in all dimensions at once.
• False-positive governance: in real engineering, the value of a rule does not depend only on how many problems it finds. It also depends on whether developers are willing to keep it enabled for a long time. Once the false-positive rate crosses a threshold, teams may disable the rule entirely.
• Domain-knowledge threshold: to catch real defects, one must understand the cause and semantics of the defect, not just its surface code shape.

A useful rule of thumb is: checker engineering value ≈ number of true defects detected ÷ (1 + false-positive governance cost). If detection is increased without controlling false positives, the net benefit may be negative.

2. Why patch-generation benchmarks cannot measure checker-writing ability

The mainstream way to evaluate code-writing models has been: give the model a defect report and a corresponding repository, let it generate a patch, and then use human-written test cases to judge whether the patch is correct. This process works well for “fixing a single defect,” but its incentive direction is different from “writing a checker.”

The key differences can be summarized in five dimensions:

Dimension	Patch-generation task	Checker-generation task
Judgment signal	Whether a small set of test cases passes	Detection rate and false-positive rate over a large set of labeled samples
Generalization requirement	Only the current defect needs to be fixed	Must cover multiple forms of the same class of defects
False-positive constraint	Almost absent	A core metric that determines whether the rule can be deployed
Output form	A local code change	A reusable, configurable, and maintainable analysis component
Correctness validation	Run the tests	Construct positive and negative examples and perform cross-project regression

If these two tasks are placed in the same evaluation framework, the strategy most likely chosen by a model is not “implement a general analysis algorithm,” but rather to hardcode a shortcut for known samples. For example:

# A pseudo-checker that can pass sample cases but has no generalization
def check_null_deref(node, source_file):
    if "PaymentService" in source_file:          # only effective for the test repository
        if node.name == "orderId" and node.used_without_guard():
            report(node, "possible null dereference")

This code can turn all test cases for that defect green, but it knows nothing about renamed variables, other modules, or other call chains. Measured by patch-generation standards, this kind of solution would be considered “excellent.” Measured by checker standards, its practical value is close to zero. This is the systematic distortion caused by mismatched evaluation criteria.

3. How to design a credible checker benchmark

To make an evaluation truly distinguish “implemented an algorithm” from “hardcoded an answer,” the benchmark needs five elements:

1. Task specification: define a unified input form—source directory, language version, and optional intermediate representation—and a unified output contract—diagnostic location, severity, evidence path, and fix suggestion. Without this, different implementations cannot be compared horizontally.
2. Defect-pattern layering: divide patterns into public patterns that can be used for development iteration, held-out patterns that are revealed only during evaluation to measure generalization, and adversarial samples generated by transformations such as renaming, control-flow reordering, and cross-file splitting.
3. Multi-dimensional metrics: detection rate, precision or false-positive density per unit of code, harmonic mean, cross-project migration decay, runtime and memory overhead, and maintainability of the generated code.
4. Automated judgment: maintain a labeled sample repository as the gold standard, together with mutation-generated negative samples, so that judgment does not depend on subjective human scoring.
5. Anti-cheating constraints: randomize naming and package structure, restrict direct access to evaluation samples, and limit retries and runtime to prevent reward hacking.

The metrics can be defined as follows, and the pass condition can be set accordingly:

Recall    = TP / (TP + FN)          # How many true defects were caught
Precision = TP / (TP + FP)          # How many reported findings are true
F1        = 2 * P * R / (P + R)
Generalization retention = Recall(held-out patterns) / Recall(public patterns)
Pass-condition example: Recall >= 0.70 and false positives per 1,000 lines <= 1
             and generalization retention >= 0.80 and single-run cost <= N x baseline

The key point is that the pass condition must constrain detection capability, false-positive level, and generalization at the same time. If only one dimension is evaluated, the model will inevitably overfit toward that direction.

4. What the empirical results show

In the study, 21 combinations of models and agent frameworks were covered. Each combination was run three times. The average pass rate was about 32.30%, and the best-performing combination reached 45.33%. These two numbers give a clearer signal than they may first appear to:

• Capability exists but is insufficient. Models can produce syntactically correct and runnable checker skeletons, and even implement partial control-flow traversal. However, they are not yet at the level of being enabled on a real codebase.
• The gap is concentrated on generalization. Models can handle defect patterns that appeared during training to some extent, but when switched to held-out patterns or equivalently transformed samples, detection drops significantly. This suggests that the models learned surface features rather than defect semantics.
• False positives are the main failure point. Many implementations prefer to over-report, and under the evaluation’s false-positive constraints, they lose qualification directly.

The common failure modes can be summarized in six categories:

1. Only performs syntax-level pattern matching and lacks intra-procedural data-flow analysis.
2. Overfits to samples and cannot migrate to new cases.
3. Has uncontrolled false positives and lacks context conditions and exception handling.
4. Ignores cross-file and cross-procedural call chains and fails to trace the sources of return values.
5. Cannot construct counterexamples to validate the rules it writes.
6. Generates rules that are hardcoded, have no switches, and do not support suppression comments, making them unsuitable for engineering maintenance.

This confirms the judgment in Section 2: under mismatched evaluation criteria, models learn to “satisfy the tests,” while under correctly matched evaluation criteria, what is exposed is their real weakness in analysis methods and engineering constraints.

5. Engineering practice: how AI should participate in rule development

Since “fully autonomous generation of production-ready checkers” is not currently viable, a more pragmatic approach is to place AI at the appropriate position in a human-AI collaboration pipeline. AI is responsible for candidate implementations and test construction, while automated regression and human review protect the quality gate:

1. Spec first: domain experts should define the defect pattern clearly using natural language and positive/negative samples. This input is non-negotiable.
2. Candidate generation: the model outputs a checker implementation and is also required to generate accompanying test cases and minimal reproduction examples.
3. Automated regression: run the checker in batch on historical code corpora, calculate detection rate and false-positive density, and compare it with baseline rules.
4. Gradual rollout: first enable alert-only mode without blocking, observe for one or two iteration cycles, then block only newly added code, and finally enable it fully.
5. False-positive governance: provide suppression markers, path whitelists, and exception registration mechanisms, turning false positives from “defects” into “manageable data.”
6. Rules as code: checkers should be placed under version control and code review, with clear owners, applicable language versions, and invalidation conditions.

A minimal evaluation-driven development framework is shown below:

CASES = load_labeled_cases()   # Each case: file, expected diagnostic locations, and whether a report is expected
def evaluate(checker, cases):
    tp = fp = fn = 0
    for c in cases:
        got = set(checker.run(c.file))
        exp = set(c.expected_diag)
        tp += len(got & exp)
        fp += len(got - exp)
        fn += len(exp - got)
    recall    = tp / (tp + fn) if tp + fn else 0.0
    precision = tp / (tp + fp) if tp + fp else 0.0
    f1        = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
    return {"recall": recall, "precision": precision, "f1": f1, "false_positives": fp}

Supporting engineering capabilities can be implemented using common cloud platform categories: code hosting and artifact management for storing rule and sample repositories; container platforms for fixing analysis environments and dependency versions so that multi-round evaluation results remain comparable; pipeline services for connecting generation, evaluation, and gradual rollout into an automated flow; logging services and monitoring alerts for tracking online false-positive rates and changes in rule hit rates; and object storage for archiving historical evaluation results and difference reports, making it easier to trace the performance drift of the same rule across different corpora.

6. Conclusion and recommendations

At the current stage, a relatively clear judgment can be made: AI is already capable of handling the “from specification to first implementation” portion of checker development, including traversing control flow, matching typical defect patterns, and generating test cases. However, there is still a substantial gap between this and a checker that can be kept online for a long time. The main gaps are inter-procedural semantic analysis, generalization ability, and false-positive control, rather than whether the code can run.

Three actionable recommendations for R&D teams:

• Change the evaluation criteria. Do not use patch pass rate to measure rule-generation capability. Use an evaluation set with held-out samples, false-positive constraints, and generalization-decay measurement. Otherwise, the improvement direction itself is wrong.
• Treat the false-positive rate as a first-class metric. Fix false-positive gates and gradual rollout into the workflow to prevent rules with high detection but high noise from entering daily blocking.
• Make AI a copilot, not the lead author. Humans should be responsible for defect semantics and specifications, while AI handles candidate implementations, test construction, and batch regression. Quality gates should be enforced by automation and review.

Under this division of labor, AI’s ability to generate checkers will steadily improve as evaluation criteria mature. If mismatched benchmarks continue to be used, the resulting “high scores” will only hide the real capability boundary.