Can AI Write Static Checkers on Its Own? Evaluations Reveal a Real Gap

Static analysis checkers are a special and critical engineering asset in a software quality system. They do not actually run the program. Instead, they parse code structure, build intermediate representations, and run data-flow or taint analysis to recognize potential defect patterns. Null-pointer dereferences, unclosed resources, memory leaks, out-of-bounds accesses, and concurrency races often propagate across functions, files, and even modules. A checker must both catch real defects and keep false positives low. That is why writing checkers has long been treated as expert work. The barrier is not a few lines of matching logic. It is a combined grasp of defect semantics, analysis precision, and engineering maintainability.

As large language models and agent technology have advanced, a natural question has appeared: can an agent that reads and writes code, calls tools, and runs tests automatically generate a set of static checkers? Existing evaluation research is not optimistic. In experiments that cover multiple model and tool combinations and repeat several rounds, the average pass rate is about 30 percent, and the best combination still falls short of 50 percent. In other words, getting AI to write a rule that runs is not hard. What is hard is writing a checker that stably finds real defects on a hidden code set, keeps false positives under control, and remains maintainable and reusable. This article analyzes the problem from four angles: the nature of the task, evaluation design, failure modes, and engineering practice.

A Checker Is Not a Patch Generator

Many code-generation evaluations follow the same flow: given a defect description and a repository, generate a fix, then validate it with tests. That flow fits defect repair. It does not fit checker generation, because the goals differ. A patch is a one-time repair of a known defect. A checker is a reusable diagnostic capability for a class of defect patterns. If an agent writes an extremely narrow rule aimed only at the sample defect, the sample may pass while new variants of the same problem go undetected. If the rule is too broad, it floods normal code with false positives and the tool becomes unusable.

An analogy helps. Fixing a defect is like clearing a blockage. Building a checker is like designing a monitoring system that warns of similar blockages. The latter requires understanding causes, defining observable signals, establishing decision conditions, and controlling false alarms. Static checkers also face inherent program-analysis problems. Alias relationships, path conditions, interprocedural calls, exceptional control flow, and reflection or other dynamic behavior all affect precision. Checker generation therefore requires at least four sub-capabilities: abstracting the defect pattern, operating on program representations, choosing an analysis algorithm, and suppressing false positives while explaining evidence. None of these can be missing.

What Checker Evaluation Needs: Hidden Samples and Multiple Metrics

If evaluation only asks whether the sample defect disappears, agents can take a shortcut and produce a disguised checker that works only on the sample. A more reasonable task is: given a defect-category description, a small set of positive and negative examples, and limited code access, generate an executable checker; then measure generalization on a hidden code set. Metrics cannot stop at detection rate. They should also cover precision, recall, false-positive rate, stability, runtime cost, and maintainability.

A true positive means the checker reported a real defect. A false positive means normal code was flagged. A false negative means a real defect was missed. Precision asks how many reports are real. Recall asks how many real defects were found. The false-positive rate asks what share of normal samples were incorrectly flagged. Evaluations should also measure consistency across repeated runs, so the same input does not drift, and should assess rule complexity and explanation quality, so the result is not an unmaintainable black box. The set should include negative cases: code that resembles the defect pattern but is already correctly guarded, such as paths that are null-checked, released, or locked. Without negative cases, a checker easily degrades into “alarm on a keyword.”

Why Strong Models Still Struggle to Pass

Experimental results of about a 32 percent average pass rate and about 45 percent for the best combination show that current agents remain at the stage of “useful as assistance, unreliable on their own.” Passing here usually does not mean the code compiles. It means the generated checker reaches an acceptable true-positive rate and false-positive control on hidden samples. Strong models still fail mainly not because they cannot write syntax, but because the task demands both program-analysis intuition and awareness of engineering constraints.

First, models overfit the samples. From a few positive examples they may latch onto surface features—a function name, a call order, or a comment—rather than the defect’s data-flow conditions. Second, they tend to write overly broad rules. To avoid misses, a rule may report every suspicious call and drive the false-positive rate up. Third, interprocedural analysis is weak. Real defects often travel through parameters, return values, global state, or container objects, which single-function pattern matching cannot cover. Fourth, they lack an explicit precision tradeoff. A checker must balance misses against false alarms, and models often have no explicit precision strategy. Fifth, engineering details are missing, such as report deduplication, path evidence, suppression, exception handling, and regression tests.

Failure Modes: From Surface Rules to Missing Evidence

Common failures fall into a few types. Sample-memory rules match only specific identifiers or code shapes in the examples and break when a variable is renamed. Keyword-trigger rules alarm on a class of calls without checking guards or release paths. Single-function rules match only inside one function and do not track parameter sources or return-value propagation. Uncontrolled false-positive rules are so broad that they create noise across a large codebase. Unexplainable reports give a location but no path evidence, so developers cannot judge whether the finding is real.

These modes point to the same requirement: a checker must rest on verifiable program facts. For a null-pointer check, it should at least answer: where might the pointer be null? Is the dereference dominated by a null check that proves it is non-null? Is there an interprocedural path that returns null? Without those facts, the rule is only text matching. For an unclosed resource, it must track allocation, release, exceptional paths, and ownership transfer. The value of a checker is not that it can report, but that it reports accurately, explains clearly, and can be regression-tested.

A Practical Loop: Generate, Validate, Regress

Agents should not generate a checker once and hand it over. The pipeline should be closed: read the specification, generate candidate rules, run them on positive and negative samples, analyze false positives and misses, revise the rules, and regress. The following product-neutral pseudocode is only illustrative:

rule NullPointerDereferenceCheck:
  input: function-level IR, data-flow fact maybeNull, guard set guards
  for each dereference site d:
    if maybeNull(d.pointer) and no guard g that dominates d can prove d.pointer is non-null:
      report(d.location, "possible null-pointer dereference", evidence=d.propagationPath)

Evaluation samples should include both positive and negative cases:

samples:
  positive: defect category = unclosed resource, expected report location = exceptional path after allocation
  negative: same resource pattern but correctly closed, expected no report
  negative: dereference after a null check, expected no report

In the loop, the agent’s output should not be only rule text. It should also include rule assumptions, covered defect patterns, known boundaries, a test-result summary, and false-positive analysis. Validation uses hidden samples to compute precision, recall, and the false-positive rate. Regression keeps failing samples so later edits do not destroy existing capability. For high-risk rules, add human review and differential testing: mutate the same defect pattern into variants and check whether the rule stays stable, and run it on normal codebases to observe noise.

Put AI in the Assistant Seat, Not the Replacement Seat

For engineering teams, the realistic path is to treat the agent as a checker-development assistant, not a replacement for static-analysis experts. First, build a defect-pattern library and labeled set, turning real historical issues into positive and negative samples. Second, manage rule risk in layers: low-risk, well-defined patterns can be drafted by AI and enter an observation period after automated regression; high-risk, interprocedural rules need human review. Third, put checkers into continuous integration, with false-positive thresholds and incremental scanning, so a full-corpus alarm flood does not bury developers. Fourth, run reproducible evaluation jobs on a container platform, store samples and results in object storage, record failure causes in a logging service, and track rule stability with monitoring and alerts. These generic cloud service categories can support an auditable evaluation pipeline, but sample quality and metric constraints remain the core.

One misconception should be avoided: reading the average pass rate as “AI is useless.” A pass rate of roughly 30 to 40 percent shows that independent delivery is insufficient, but it also shows that, inside a structured process, AI can draft rules, expand test cases, attribute false positives, and generate documentation. Real productivity comes from collaboration: people define defect semantics and acceptance criteria, and AI speeds up candidate generation and iterative validation.

Conclusion

Can AI write a static code-checking tool by itself? A more accurate answer is that it can produce runnable rule prototypes, but it still falls well short of independently delivering reliable, reusable, low-false-positive checkers. Checker generation is not ordinary code generation. It is a combined test of program analysis, defect modeling, and an engineering validation system. The steadier strategy for a team is hidden-sample evaluation, multi-metric acceptance, and a generate–validate–regress loop, so AI amplifies expert efficiency inside a controlled process rather than leaving tool quality to a one-shot generation.