LLM Evaluation for Cybersecurity: Measuring Recall and Run-to-Run Stability
Security teams increasingly evaluate large language models for vulnerability discovery, code auditing, threat intelligence analysis, and incident response. This article explains why a single leaderboard number is misleading: it breaks down a real-world evaluation of 32 recently disclosed vulnerabilities run three times per model under offline constraints, covering roughly 11.7 billion tokens, and shows that aggregate recall, first-run recall, and stable hit rate are not equivalent. It walks through the metrics security teams should track, the technical reasons high aggregate recall can coexist with unstable output, the full cost picture behind token consumption, the limits of such evaluations, and a practical framework for building an internal benchmark with gold-standard datasets, layered task routing, stability thresholds, and regression testing.
1. Background: Why Security LLMs Need Rigorous Evaluation
Large language models and agent-based systems are now widely explored for security work: finding vulnerabilities in code, analyzing threat intelligence, mapping attack surfaces, auditing configurations, and assisting triage during incident response. Compared with traditional rule engines, models can interpret both natural language and code semantics and carry some cross-context reasoning ability. Their weaknesses are just as clear: results may be inconsistent, costs are high, explainability is limited, and behavior shifts with prompt design, context length, tooling, and runtime policy.
So the questions the industry actually cares about are not “does a model look strong in one demo,” but three more engineering-oriented ones:
- Can it reliably find problems in real code and real vulnerability samples?
- Are results consistent across runs, and can it be trusted inside an automated security pipeline?
- Is the gain in recall worth the token spend and engineering effort?
One evaluation that drew wide attention used a deliberately demanding setup — high consumption, real samples, repeated runs, and an offline constraint — and consumed roughly 11.7 billion tokens in total. Its value is not a simple ranking. It exposes a contradiction that security model evaluation often glosses over: aggregate recall, first-hit rate, and stable hit rate are not the same thing.
2. Test Design: Why “Real Vulnerabilities + Repeated Runs” Is More Trustworthy
The baseline setup is broadly instructive. The evaluators selected 32 samples from recently disclosed real vulnerabilities and tried to reproduce realistic codebase conditions rather than using hand-crafted toy cases. Each model had to attempt each case three times, with up to 30 exploration rounds per case, and no internet access at any point.
This design addresses at least three problems.
2.1 Real Vulnerability Samples Reduce Memorization Contamination
If an evaluation uses vulnerabilities, old patches, or classic public cases that already appear extensively in the model’s training data, the model may be matching against memory rather than demonstrating analysis capability. Choosing recently disclosed vulnerabilities lowers the probability that the model is effectively “recalling answers” from its training corpus, and brings the test closer to a real unknown-threat scenario.
2.2 Offline Execution Tests Intrinsic Capability
No internet access means the model cannot rely on online vulnerability databases, search engines, or external patch content. It has to work from its own understanding of code, configuration, protocols, permission models, and common attack patterns. This does not exactly match many real security workflows, where internal knowledge bases or external intelligence may be available. But as a floor-level measure, offline evaluation is a better reflection of the model’s native reasoning ability.
2.3 Repeated Runs Expose Stability Differences
LLM output is stochastic: the same question can produce different paths and different conclusions across runs. Security work is especially sensitive to this. If a model finds the key vulnerability in only 1 out of 10 runs, it is better suited as an exploratory assistant than as an automated gate. Repeated runs let you separate:
- Aggregate recall — at least one hit counts, representing the upper bound of potential;
- First-run recall — hit on the first attempt, representing cold-start capability;
- Stable hit rate — hit in all three runs, representing the reliability floor;
- Exploration depth — how many rounds were needed to find it, representing reasoning efficiency.
3. Core Metrics: One Recall Number Is Not Enough
Many evaluations report only “was the vulnerability found.” Enterprise adoption needs finer-grained metrics. At minimum, track the following.
3.1 Aggregate Recall
Aggregate recall counts a vulnerability as found if any of the multiple runs succeeded. It reflects the model’s ceiling.
Aggregate recall = number of vulnerabilities hit at least once / total vulnerabilities
This metric is useful for asking whether the model is capable of finding complex vulnerabilities at all, but it tends to overstate practical usability.
3.2 First-Run Recall
First-run recall measures whether the first attempt succeeded. For automated scanning or low-latency pipelines, first-run success matters a great deal.
First-run recall = number of vulnerabilities hit on the first run / total vulnerabilities
If aggregate recall is high but first-run recall is low, the model likely needs multiple attempts to finish the job, and the pipeline needs an additional retry mechanism.
3.3 Stable Hit Rate
Stable hit rate measures the proportion of cases hit in all runs. It is closer to what production demands.
Stable hit rate = number of vulnerabilities hit in all three runs / total vulnerabilities
A low stable hit rate means insufficient output consistency. For critical security gates, low stability translates into significant manual re-review load.
3.4 Cost Efficiency
Recall has to be read alongside cost. Security models typically perform multi-round reasoning, code retrieval, context summarization, and tool calls, so token consumption can far exceed a normal question-answering session.
Cost per effective find = total cost / number of effective discoveries
“Effective discoveries” should be counted as stable hits or human-verifiable hits, not simply as any single hit.
3.5 Exploration Rounds and False-Positive Cost
If one model needs 30 exploration rounds to hit and another hits in 10, the latter may have better engineering efficiency even at similar recall. Security work also demands attention to false positives. A single incorrect alert can consume substantial developer time in investigation, offsetting the efficiency gains the model was supposed to deliver.
4. Key Results: Leading Aggregate Recall, Insufficient Stable Hits
Described neutrally, the evaluation surfaced a typical pattern. One model reached 28/32 on three-run aggregate recall, clearly ahead of the other participants, most of which landed in the 25/32 to 26/32 range. Yet that model’s single-run behavior was uneven: only 17 cases were found on the first run, and just 10 cases were hit in all three runs.
The implications:
- Ranking first on aggregate recall does not mean ranking first in engineering usability.
- The model may hold real potential while producing unstable output paths.
- Insufficient first-run hit rate significantly raises retry costs in automated pipelines.
- A low stable hit rate weakens its credibility as a critical security gate.
Meanwhile, another model’s production build reached 25/32 — only one case below the top score — at a noticeably lower cost, about 69.8% cheaper than the leading option. Results like this are valuable for procurement decisions: in scenarios with limited budget, large sample volumes, and a need for batch screening, a strong cost-performance model can be more attractive than maximum recall.
5. Why High Aggregate Recall Can Coexist with Poor Stability
From a technical standpoint, high aggregate recall with a low stable hit rate is not unusual. The usual causes are:
5.1 Divergent Reasoning Paths
A single vulnerability may be reachable through many routes — entry-point parameters, authentication logic, serialization flows, file handling, permission boundaries, configuration parsing, or dependency call chains. Each run may pick a different entry point. Some paths reach the vulnerability; others abandon the attempt early or incorrectly conclude the code is safe.
5.2 Context Window Limits
Real codebases are usually large enough that no model can absorb every file at once. Although the evaluation allowed up to 30 exploration rounds, the context injected per round is still bounded. When a key vulnerability spans multiple files, functions, or configuration layers, the model can miss it for lack of intermediate evidence.
5.3 Complex Vulnerabilities Require Deeper Reasoning
Simple issues are often caught by pattern matching: hard-coded secrets, obvious command concatenation, unsafe deserialization. Complex vulnerabilities require understanding business logic, permission models, call ordering, and error-handling paths. These tasks demand more from a model’s planning ability and fail on a single run more often.
5.4 Prompting Strategy Affects Results
Even with the same model, different task templates, system prompts, output constraints, and tool-calling approaches produce different outcomes. A model instructed to “provide an exploitable path” or “analyze file by file” may score higher than one asked a simple question.
5.5 Sample Type Distribution Shapes Conclusions
32 vulnerability samples are real, but the set is small. If one category dominates — injection, authorization, deserialization, or dependency issues — the results tilt toward the capabilities that category requires. Enterprises should not extrapolate the conclusion directly to every language, framework, and business context.
6. The Cost Angle: 11.7 Billion Tokens Is Only the Surface Cost
A cumulative consumption of roughly 11.7 billion tokens is itself evidence that the real cost of security model analysis should not be underestimated. Vendor selection cannot stop at per-call pricing; the full lifecycle cost matters.
6.1 Inference Cost
Multi-round exploration amplifies token consumption. If a vulnerability takes an average of 20 reasoning rounds, each carrying code context, prior analysis, and tool output, the cost can far exceed a single ordinary query.
6.2 Retry Cost
A low first-run hit rate forces the automation to rerun. Each attempt looks cheap, but multiplied by batch sample volume and retry counts, the total can exceed that of a higher-capability model.
6.3 Manual Review Cost
After the model reports a finding, security staff still have to confirm whether it is real, exploitable, and relevant to the current version. With a high false-positive rate, manual review becomes the primary bottleneck.
6.4 Cost of Missed Detections
In security work, a miss usually costs more than a false positive. An undetected high-severity vulnerability can lead to patching, incident response, compliance, and reputational costs later. Enterprises therefore have to trade off recall, stability, and their own risk tolerance.
6.5 Engineering Integration Cost
If a model cannot be embedded into code repositories, defect-tracking systems, CI pipelines, log-alerting systems, and the access-control model, even excellent recall is hard to convert into real value. Integration includes context preparation, permission isolation, structured results, audit trails, and version management.
7. Limits of the Evaluation: A Leaderboard Is Only a Starting Point
Evaluations like this are useful, but they carry clear limitations.
7.1 Limited Sample Size
32 vulnerability cases can show a trend, but they cannot cover a complex production environment. Basing a purchase decision on a single leaderboard carries risk.
7.2 Frequent Model Version Changes
Models iterate quickly, and today’s ranking can shift after the next release. Such conclusions have a shelf life and need periodic re-testing.
7.3 Limited Language and Framework Coverage
Security patterns differ widely across languages, frameworks, cloud-native environments, mobile components, and legacy systems. A model that performs well on generic codebases may not excel on a specific business codebase.
7.4 Evaluation Conditions Differ from Deployment
The offline, no-internet condition raises the bar for testing intrinsic model ability, whereas real enterprises can usually combine internal knowledge bases, historical tickets, patch repositories, and static analysis tools. Actual capability may be higher than the offline result — or lower, if the engineering wrapper is inadequate.
7.5 Recall Alone Is Not a Complete Picture
Production adoption also requires false-positive rate, explainability, quality of remediation advice, completeness of the evidence chain, access-control safety, and data compliance. Chasing recall only can produce a flood of “suspected issues” and increase the team’s workload instead.
8. Enterprise Recommendations: Build Your Own Security Model Evaluation
Enterprises should not copy external leaderboard conclusions directly. They should build an evaluation system around their own context.
8.1 Define the Business Scenario
Different scenarios place different demands on a model:
- Vulnerability discovery: recall, stable hits, and understanding of attack paths matter most;
- Code audit: cross-file tracing and remediation quality matter most;
- Threat analysis: semantic understanding and synthesis of multi-source logs matter most;
- Incident response: low latency, first-run hits, and reliable response guidance matter most.
8.2 Build an Internal Gold-Standard Dataset
Build the gold standard from internal historical vulnerabilities, already-fixed defects, authorized penetration test results, dependency security incidents, and manually audited samples. Each sample should include:
- Vulnerability description;
- Root-cause file or configuration;
- Trigger conditions;
- Scope of impact;
- Remediation approach;
- Whether it can be verified by automated tooling.
8.3 Layer the Tasks
Do not hand every task to a single model in one pass. Split the work:
- Triage: a low-cost model quickly flags suspicious files;
- Deep analysis: a higher-capability model performs cross-file tracing;
- Verification: confirm findings with static rules or dynamic testing;
- Review: a human security expert confirms risk and priority.
8.4 Multi-Model Collaboration and Routing
Model performance varies considerably by task. Build a routing strategy: route simple samples to low-cost models, complex samples to higher-capability models, and require multiple samples or human involvement for high-risk projects.
8.5 Set Stability Thresholds
In automated pipelines, “found it once” should not be sufficient. Set stability gates instead. For example:
- Routine scanning: at least two hits before an alert is raised;
- High-severity vulnerabilities: at least two hits across three runs, with a complete evidence chain attached;
- Critical gates: route to manual review when stable hits fall short.
8.6 Build a Cost-and-Effectiveness Dashboard
Record the key metrics for each run:
- Model version;
- Prompt template version;
- Token consumption;
- Exploration rounds;
- Whether the first run hit;
- Whether the result was a stable hit;
- Number of false positives;
- Time spent on human confirmation.
Only with accumulated longitudinal data can an enterprise judge whether a given model class genuinely fits its codebase baseline and team workflow.
8.7 Always Regression-Test After a Model Upgrade
After a security model upgrade, recall may improve while false-positive rate, output format, context length, and tool-calling behavior also change. Every version update should re-run the internal gold standard to avoid introducing new risk.
9. Reading “Number One in Recall” Rationally
“Best recall” spreads easily as a headline, but it is only one facet of capability assessment. For an enterprise, what matters is not who led a single evaluation, but who can reliably create security value inside real development workflows.
An ideal model would show the following characteristics:
- Reasonable recall on real vulnerabilities;
- Stable results across repeated runs;
- Clear evidence chains;
- Controlled token and engineering cost;
- Ability to work alongside existing security tooling;
- Ability to pass internal regression tests;
- Verifiable remediation advice.
If one model has high aggregate recall but a low stable hit rate, it can serve as an exploratory tool. If another has slightly lower recall but lower cost and better stability, it may suit batch triage better. The two are not simple substitutes — they can be combined.
10. Conclusion: The Value of Security LLMs Lies in Stability, Verifiability, and Integrability
This high-token-cost evaluation of security models delivers one clear lesson: security model evaluation cannot rest on a single run, and certainly not on “was the vulnerability found.” Aggregate recall represents potential. First-run recall represents cold-start capability. Stable hit rate represents engineering reliability. Cost efficiency represents feasibility at scale.
For enterprises, the sensible path is not to chase a single leaderboard but to establish an internal evaluation baseline and make selection decisions against business code, vulnerability samples, budget, and manual review capacity. A model delivers real security value only when it can reliably enter the development workflow, reduce human effort, provide verifiable evidence, and hold total cost in check.
