Can AI Agents Write Static Analysis Checkers? Benchmark Shows Only 32.30% Average Pass Rate

Introduction: A Specialist Area That Has Always Needed More Experts

Almost every class of high-severity software defect has caused real incidents in production systems. For example, a widely used multimedia processing tool once undercounted a single byte of space while parsing a PNG image. As a result, the program wrote a string terminator one byte beyond the intended boundary, creating a memory-corruption risk.

These defects often arise not because developers are careless, but because the defect patterns themselves are highly subtle:

• A null pointer may be dereferenced only on one execution path out of thousands.
• An out-of-bounds write may only become visible with specific image dimensions or particular memory layouts.

What truly grows with software scale is the need to detect this kind of defect before code is committed. That work falls primarily to static code analysis and its core component: the checker.

Concept	Plain-language explanation
Static code analysis	A technique that does not run the program, but infers defects by analyzing source-code structure and logic
Checker	A component in a static-analysis tool that identifies one class of defect, such as a null-pointer checker, buffer-overflow checker, or path-traversal checker

For a long time, writing checkers has been the work of a small group of experts. They must understand compiler construction and program analysis, master the root causes and boundary conditions of the target defect, and repeatedly tune the checker until its false-positive rate becomes acceptable to engineering teams.

Can AI agents that can already read and write code, run commands, and call tools complete this work independently? Researchers from several universities and research institutes have provided a useful reference point.

1. Why a New Benchmark Is Needed: Patch-Style Evaluation Cannot Measure Checkers

Most existing benchmarks for evaluating AI coding ability follow the same process:

1. Provide an issue and its corresponding repository from a code-hosting platform.
2. Ask the model to generate a patch.
3. Use human-written test cases to determine whether the patch is correct.

This process works well for evaluating whether an AI can “fix a bug,” but it was not designed for “writing a checker.” The difference shows up in three dimensions.

1. Different task goals: one-time repair vs. continuous detection

Patch-style benchmarks reward fixing the immediate issue. Writing a checker requires a reusable and generalizable detection mechanism: it must catch the known defect instance, identify similar defects in previously unseen code, and avoid excessive false positives on normal code.

A simple analogy makes the difference clear:

Task type	Plumber analogy	Requirement for an AI agent
Generating a patch	Using a pipe wrench to remove the current blockage	Accurately locate the problem and perform a local fix
Writing a checker	Designing a continuous leak-detection and alerting system	Understand the cause, cover similar cases, and control false positives

2. Different evaluation criteria: test pass vs. precise detection

A patch can be judged with unit tests, which provide a clear pass/fail result. The quality of a checker, however, cannot be reduced to a simple pass/fail decision, because a checker is fundamentally a probabilistic inference tool. It must trade off between two competing metrics:

• To catch more defects, conditions must be relaxed, which increases false positives.
• To report more accurately, conditions must be tightened, which increases false negatives.

In engineering practice, teams usually evaluate both of the following metrics:

Recall = number of true defects correctly reported / total number of true defects
False-positive rate = number of incorrectly reported locations / total number of locations reported by the checker

An evaluation that only asks whether a known defect was caught cannot show whether this trade-off was handled well.

3. Different data-leakage risks: memorized answers vs. learned methods

Defect records from public code-hosting platforms often overlap heavily with model training data. A model may generate a seemingly reasonable patch simply because it has seen a similar final fix, without truly understanding the defect pattern. A checker task requires the model to produce a new piece of analysis logic, which makes it a more realistic test of capability.

2. Why Writing a Checker Is Difficult

To understand why AI agents score poorly on this task, it helps to examine the technical structure of a checker. Using a typical compile-time static-analysis tool as an example, a checking rule usually goes through the following steps.

Step 1: Turn source code into an analyzable structure

A compiler parses source code into an abstract syntax tree, or converts it into an intermediate representation. On that basis, it builds a control-flow graph (CFG) to describe the program’s possible execution paths.

The example below shows how two if branches create a new control-flow path at their merge point:

int f(int *p, int flag) {
    if (flag) {
        p = NULL;
    } else {
        p = malloc(sizeof(int));
    }
    return *p;        // NULL dereference when flag is true
}

Each individual line looks fine. The defect exists only in the combined state after the branches merge.

Step 2: Infer variable states across multiple paths

Path-sensitive analysis tracks possible variable values along each execution path. In the example above, there are two paths reaching return *p: one where p is null and one where p points to newly allocated memory. The checker must merge these states and decide whether a possible null dereference should be reported.

Step 3: Track data flow across procedures

Real-world defects rarely stay inside a single function. Defective data often flows through function parameters, struct fields, global variables, and callback functions. The checker must build an interprocedural call graph and connect externally controllable data sources with dangerous operation points. This is the basic idea behind taint analysis.

Step 4: Define a reporting policy and integrate with the analysis framework

Finally, the checker must be implemented against the interfaces of a specific analysis framework: registering rules, traversing program structures, maintaining state, and emitting diagnostic information. This step involves many framework-specific engineering details.

The following is a simplified rule-logic example in pseudocode:

on_dereference(expr):
    taint = source_of(expr)              # Does the expression come from external input?
    if taint and not null_checked(expr):
        report("Possible null-pointer dereference", at=expr)
    # Do not report if expr has an explicit null check or is already known to be non-null

Although this is only a few lines long, a real implementation must answer many questions:

• Which function return values may be null?
• Which conditional checks count as valid null checks?
• How should state converge inside loops?
• How can duplicate reports for the same issue be avoided?

These decisions require solid program-analysis knowledge, and that is one of the main reasons checkers are difficult to write.

3. How to Design a Proper Evaluation: Basic Requirements for an Exam

Because patch-style evaluation has clear limitations, the researchers built a benchmark specifically for checker writing, referred to here as the “new benchmark.” From a methodology perspective, such a benchmark should meet at least four requirements.

1. Understandable task specifications
   Each evaluation item should correspond to a clear defect pattern, with semantic explanations and positive/negative examples, so the agent knows what it should catch and what it should not catch.

2. Real-world code corpus
   The benchmark should use defect scenarios from real projects rather than artificially assembled toy code, reducing the chance that a model can succeed through simple pattern matching.

3. Executable scoring
   The checker produced by an agent should actually run on real code. Its results should be compared automatically against expected labels, instead of relying on subjective human judgment.

4. Support for repeated independent runs
   Defect detection itself is uncertain. Only multiple runs can show whether results are stable, rather than reflecting one lucky attempt.

4. Core Results: Even the Best Combination Passed Less Than Half

The researchers evaluated 21 combinations of models and tools. Each combination was run independently for three rounds. The main results were as follows:

Metric	Value	Meaning
Average pass rate	32.30%	Overall performance is below a barely passing line
Best combination score	45.33%	Even the strongest configuration failed to complete more than half of the tasks

Three conclusions can be drawn from these results.

First, AI agents are still only partially capable of writing checkers.
Nearly 70% of evaluation items could not be completed. This shows that current agents are not yet sufficient to independently deliver production-ready checkers.

Second, model capability is not the only bottleneck.
Among the 21 combinations, performance differences came from both the models themselves and the surrounding toolchains and agent architectures. The way code is accessed, the retrieval strategy, and how well the analysis-framework APIs are wrapped can all significantly affect results. This leaves room for engineering optimization.

Third, the repeated-run design is necessary.
It reveals the variance of agents on this type of task. A high score in a single run may not be reproducible.

5. Failure-Mode Analysis: Where Agents Get Stuck

Combined with the typical difficulties of static analysis, the main reasons agents fail can be grouped as follows.

1. Imprecise understanding of defect semantics
Natural-language defect descriptions often contain many implicit conventions. For example, “null-pointer dereference” actually requires considering which return values may be null, which checks count as valid null checks, and whether error-handling branches already guarantee non-null values. Misunderstanding these details can make a rule too broad or too narrow.

2. Insufficient path and interprocedural analysis capability
Defects often span functions and modules. They require passing state across procedures and merging values when paths converge. Agents can usually perform local syntax matching, but they struggle to correctly implement this kind of global data-flow reasoning.

3. Lack of experience in controlling false positives
Beginner-written checkers often over-report, producing large numbers of false positives. Experienced engineers combine project coding standards and common defensive programming patterns, adding whitelists and suppression mechanisms. This kind of experience is hard to infer from syntax alone, and models often lack project-level context.

4. Complex engineering integration details
A checker must integrate with the analysis framework of a specific tool. This involves framework APIs, diagnostic output formats, cross-language support, and other details. Differences between framework versions can also cause interface changes that an agent may not know how to handle accurately.

5. Lack of self-verification
Agents often cannot independently judge whether their implementation is correct. They lack runnable validation cases and do not know whether the relevant metrics meet expectations. As a result, their output often remains at the level of “looks reasonable.”

6. Practical Lessons for Engineering Teams

The value of the new benchmark is not only the score itself, but also its clarification of the current capability boundary of AI agents. For teams that want to introduce AI into their code-quality systems, several practical lessons stand out.

1. Break checker tasks into verifiable subtasks
Instead of asking an agent to produce a complete tool in one step, proceed incrementally:

• First, ask the agent to output a semantic description of the defect pattern and positive/negative examples.
• Second, after human review confirms that the specification is correct, ask the agent to generate the rule implementation.
• Third, use the confirmed examples to regression-test the rule.

2. Build a team-specific defect-pattern library
Historical defect tickets, incident postmortems, and issues repeatedly raised in code reviews are high-quality material for both training and evaluation. Organizing them into triples of “defect description + defective code + fixed code” can help prompt agents and also serve as an internal benchmark.

3. Use the false-positive rate as the core acceptance metric
When introducing any checking tool, the first question a team must answer is whether it can tolerate the tool’s noise. For rules generated by agents, teams should evaluate the false-positive rate first, then discuss recall, and include suppression mechanisms and whitelist management in the rule design.

4. Keep experts in the loop
At the current stage, agents are better suited to act as draft writers and rule-sketch generators. They can handle tedious framework integration and initial implementation, while experts review semantic correctness, tune thresholds, and decide when a rule is ready for production.

7. Outlook: From “Can Write” to “Can Trust”

The conclusion of this research is not overly pessimistic. An average pass rate of 32.30% shows that agents can already complete a meaningful portion of the basic work involved in writing checkers, but there is still a clear gap before they can deliver independently. The path forward can be grouped into three directions:

• Provide more complete context
  Give agents project coding standards, historical defects, and analysis-framework documentation as structured knowledge.

• Introduce execution feedback loops
  Let agents run their own generated checkers on real code, then iterate based on recall and false-positive rates.

• Reduce task granularity
  Instead of asking an agent to “write a tool,” break the task down into smaller units: write one rule, add one test case, or fix one false positive. This better matches the current capability level of agents.

Static analysis has always been one of the most experience-dependent areas in software engineering. Involving AI in checker writing is essentially an attempt to turn that experience into automatable productivity. The “failing” score from this benchmark is not the endpoint. It is a yardstick that tells us how far we still are from truly usable automated static-analysis capability.