How to Build an Evaluation Framework for AI-Powered Code Vulnerability Discovery
Evaluating LLMs for code security auditing requires more than showcasing a single successful finding. This article presents a reproducible framework for measuring recall, precision, consistency, cost, and latency across realistic vulnerability samples. It also explains how to prevent benchmark leakage, standardize execution protocols, interpret results, and turn offline evaluation into practical engineering gates.
When using large language models for code security auditing, one of the easiest mistakes is to treat a single successful demonstration as an evaluation. Vulnerability discovery is a high-variance task: even with the same model, repository, and prompt, three runs may produce completely different findings. Looking only at the best run overestimates recall, while looking only at average execution time underestimates the cost of repeated exploration and queueing.
The purpose of an evaluation framework is not simply to rank models. It should answer three actionable questions:
- How many real defects can the model recover when they are not already familiar?
- Are the results reproducible?
- Is the cost per effective finding acceptable?
1. What to Measure
Vulnerability discovery should be evaluated with at least four categories of metrics rather than a single score.
Recall
Recall measures whether the model finds defects that should be found. The evaluation set should contain known vulnerabilities, and the success criteria must be defined in advance. A finding should count as a hit only when it identifies the correct file and an approximately correct code region, while also explaining the exploitable condition or incorrect assumption. Merely naming a vulnerability class or pointing to an unrelated file should not count as success.
Precision, or the false-positive burden, is equally important. Security teams are often constrained not by a complete lack of findings, but by an excessive number of alerts that cannot all be triaged. Track the number of candidates produced per run, the true-positive rate after human review, and the average number of analyst minutes required per true positive.
Consistency
Consistency measures stability. Run the same sample independently multiple times and record both:
- Hit-at-least-once: the sample is detected in at least one run.
- Hit-every-time: the sample is detected in every run.
The first metric is closer to the model’s capability ceiling. The second is closer to the capability required for an operational, on-call workflow. A large gap between the two indicates that the model depends heavily on random exploration or prompt wording. Such a model may be useful for generating leads, but it should not be used as a blocking release gate.
Cost and Latency
Cost and latency include input and output volume, tool calls, wall-clock time, and failed retries. Convert these measurements into:
- Cost per recovered vulnerability
- Recall under a fixed budget
Do not compare only total consumption. A model that is cheap but has extremely low recall, or one that has high recall but regularly exceeds the time limit, cannot be added to a production pipeline without further qualification.
2. Preventing Benchmark Memorization
A public benchmark that remains unchanged for a long time may be present in a model’s training data or retrieval sources. A more reliable approach is to use recently fixed, real-world vulnerabilities that have not yet become part of widely available general-purpose corpora. Preserve the code context from before the fix.
The sample set should cover multiple categories, including:
- Memory safety
- Injection
- Missing authorization
- Path traversal
- Deserialization
- Configuration errors
- Dependency vulnerabilities
It should also include multiple programming languages and multi-module repositories. Otherwise, the evaluation may only measure performance on one framework or repository style.
Each sample needs an immutable scoring rubric that defines:
- The root cause
- The relevant code location
- What wording qualifies as an equivalent hit
- Which neighboring issues do not qualify
The rubric must not be visible to the model. Remove subsequent fix commits, the original security advisory, and comments that clearly reveal the answer. This prevents the model from reading the conclusion directly from repository metadata or annotations.
The sample set does not need to be extremely large, but it must be large enough to distinguish accidental hits from stable capability. As a baseline, run each model independently at least three times on every sample. Fix and record the temperature and sampling strategy. If the intended use case requires low randomness, add a temperature-zero or near-zero control to determine whether recall collapses under deterministic settings.
3. Make the Execution Protocol Reproducible
Fair comparison depends on equivalent environments, not equivalent marketing claims. The following constraints should be fixed in the evaluation protocol.
Exploration Budget
Define the maximum number of turns, tool calls, and execution time per sample. An unlimited run confuses willingness to spend more steps with auditing ability.
Information Boundaries
By default, disable external network access. Prevent the model from reading the scoring rubric, fix patches, or vulnerability advisories. Allow only controlled code browsing, search, and local static analysis. This measures reasoning over the code rather than the ability to retrieve a published analysis.
Tool Parity
Give every model the same read-only tools, such as:
- Listing directories by path
- Searching for symbols
- Reading file segments
- Running lightweight static checks already available in the project
Do not provide one model with a private knowledge base or a specialized vulnerability plugin. Otherwise, the comparison measures the surrounding system rather than the model itself.
Prompt Consistency
The system prompt should describe only the role, output format, and stopping conditions. It should not embed a checklist for a particular vulnerability category. If a checklist-assisted auditing workflow is being evaluated, treat it as a separate experiment rather than mixing it with results from a bare model.
Failure Handling
Timeouts, refusals, and empty outputs should count as misses. Do not keep sampling until the model succeeds. Repeatedly retrying a high-variance model artificially inflates its observed upper bound.
A simple aggregation process is sufficient as long as the definitions remain consistent:
for case in cases:for run in range(3):pred = model.audit(repo_before_fix, budget=fixed)hit[case][run] = judge(pred, rubric[case]) # Human review or independent rulesrecall_any = count(any(hit[case])) / Nrecall_all = count(all(hit[case])) / Ncost_per_hit = total_tokens_or_time / max(1, count_hits)
Human review should use two-person cross-checking or a blind-first process. Reviewers should first examine the model’s conclusion and the code, and only then open the scoring rubric. This reduces anchoring. Discuss disputed samples separately and update the decision rules so that standards do not drift between evaluation rounds.
4. How to Interpret the Results
Look at stable hits before looking at hit-at-least-once results. Stable hits represent repeatable inspection paths. Hit-at-least-once results represent exploratory potential and are useful as a research signal, but should not independently determine a release gate.
If a model has a high upper bound but a low rate of hitting the same issue in all three runs, the likely improvement areas are constrained exploration strategies, a fixed inspection order, or self-consistency voting—not simply increasing the temperature.
Analyze slices instead of relying only on the overall score. Break results down by:
- Programming language
- Vulnerability category
- Repository size
- Whether cross-file reasoning is required
A cross-module authorization problem and an obvious single-file injection are not equally difficult. Two models with similar aggregate scores may behave very differently: one may be good at pattern matching, while the other can follow a call chain. Slice-level results determine where each model belongs—for example, in a fast pre-commit scan, a deep overnight audit, or a workflow focused only on regression checks for known dangerous patterns.
Cost curves are more informative than a single price point. Compare budgets at the same recall, or recall at the same budget. Add the costs of all three runs before dividing by the number of true positives; this reflects the real expense of achieving stable detection. Also record the analyst time caused by false positives. A low model price can be offset by an alert surge.
5. From Offline Evaluation to Engineering Gates
An evaluation set is not a one-time leaderboard. It is an asset that becomes stale. Periodically introduce new vulnerabilities, and reduce the weight of or retire samples that have been publicly available for too long and may have entered training data. Whenever the model version, system prompt, or toolset changes, run a regression against the same frozen samples and check whether recall, stable detection, and false-positive rates have changed together.
In production, a three-tier approach is practical:
- High stability and acceptable false-positive rate: Run these checks as part of pull-request automation. Human reviewers should still confirm whether the result blocks merging.
- Moderate recall and high variance: Run them on a scheduled basis. Include the code location and reasoning so that findings can be reviewed efficiently.
- Systematic blind spots: Do not expect the model to cover these vulnerability classes. Retain deterministic controls such as rules, type systems, and dependency scanners.
Human and model responsibilities should be explicit. The model proposes candidates and relevant code paths. Humans confirm reachability, business assumptions, and remediation priority.
Evaluation reports should publish the protocol and categories of failures while withholding exploit details that could be repeated directly. Otherwise, the evaluation material itself may become an attack guide.
6. Common Sources of Bias
Several practices can distort the results:
- Using one model to score another without independent sampling or spot checks can amplify shared biases.
- Selecting only repositories where the model has previously succeeded effectively leaks the answers.
- Allowing network access during execution measures retrieval rather than auditing.
- Reporting only the best run hides non-reproducibility.
- Comparing total consumption without comparing cost per true positive favors models that simply produce more output.
- Ignoring build failures and differences in tool permissions incorrectly attributes environment problems to model capability.
Once this protocol is standardized, the team gets more than a slogan such as “which model is strongest.” It gets an auditable capability map: which defects can be recovered reliably, which are found only occasionally, and which must remain the responsibility of deterministic tools and human reviewers.
That map can directly guide budgets, timeouts, gate thresholds, and review staffing. It also provides the only reliable baseline for comparing future model changes.
