Large Language Model Vulnerability Detection Evaluation: From Recall Rate to Stable Hits
As large language models (LLMs) are increasingly used for code auditing, vulnerability discovery, and security operations, how their vulnerability detection capabilities are evaluated matters more than any single leaderboard ranking. This article examines why LLM vulnerability detection evaluation needs a dedicated approach. It covers how to build trustworthy datasets from real code, how to distinguish first-run, pooled, and stable recall, and why cost and reproducibility belong in the metrics. It also shows how enterprises can build an internal evaluation loop with isolated environments, fixed scoring rules, and regression tests, and use it alongside static analysis and human review. Readers will gain a practical framework for assessing LLMs not by who ranks first, but by whether a model can find high-value vulnerabilities reliably and affordably in production.
Discussions about using large language models for cybersecurity vulnerability detection have recently been intensifying across the industry. Many of them fixate on a single question: which model has the highest recall rate? If you genuinely intend to use these capabilities for code auditing, vulnerability discovery, or security operations, however, the evaluation methodology, metric design, and engineering feasibility usually matter more than any single ranking.
This article does not focus on specific products or vendors. Instead, it takes a general technical view of what LLM vulnerability detection evaluations should focus on, how to design them, how to interpret the results, and how organizations can build a reusable, engineering-grade evaluation system.
1. Why LLM Vulnerability Detection Evaluation Deserves Its Own Approach
Once large language models entered the security domain, common applications emerged quickly: identifying code defects, surfacing vulnerability leads, generating patch suggestions, analyzing attack surfaces, attributing anomalies in logs, and summarizing security reports. Compared with traditional static analysis tools, LLMs have a distinct advantage. They can understand context, reason with business semantics, and explain uncertain code patterns. Precisely because they introduce probabilistic generation, however, evaluating them cannot simply reuse the “pass or fail” logic applied to traditional software.
Traditional vulnerability scanners rely mainly on rules, signatures, or dataflow analysis, so their results are relatively deterministic. An LLM, by contrast, may produce different outputs for similar inputs. Given the same vulnerable case, the model might pinpoint the critical issue in one run, offer only generic advice in another, and miss the core defect altogether in a third. Evaluating LLM vulnerability detection therefore has to account for several dimensions at once:
- Can it find the issue at all?
- Does it find the issue consistently?
- How much does it cost to find?
- Can the finding be verified?
This is also why a single leaderboard result is not enough. A model may stand out in aggregate statistics. But if it hits the target only occasionally across repeated attempts, or performs well on simple cases yet proves unreliable in complex, real-world scenarios, its value in production needs to be reassessed.
2. Start with Real Vulnerability Cases: Why the Dataset Matters
The core of LLM vulnerability detection evaluation is not a multiple-choice quiz. It is whether the model can locate problems in a real code environment, understand their impact, and produce conclusions that can be verified. Dataset design therefore often determines how credible the results are.
Valuable evaluation data typically has the following characteristics:
Drawn from real codebases
Real-world vulnerabilities usually occur in complex engineering environments. They often span multiple files and involve business logic, permission models, data transformations, and configuration dependencies. A model may perform well on isolated function snippets. Once it enters a real repository, however, missing context and tangled dependencies can cause its performance to drop noticeably.Backed by clear ground truth
Cases should ideally include publicly available or internally confirmed root cause analyses, fix patches, affected versions, and security advisories. This gives scoring a solid basis instead of relying on subjective judgment.Designed to avoid training data contamination
LLMs may have already seen public vulnerability disclosures, patch discussions, or related technical articles. If evaluation cases are too public and too widely circulated, the model might not be finding the flaw through code understanding at all. It may simply be “recalling” the answer. To reduce this risk, you can:- choose cases disclosed relatively recently;
- use internally sanitized cases; or
- rewrite key clues in the case.
Covering a range of vulnerability types
Real security vulnerabilities vary widely. Examples include injection flaws, access control bypasses, sensitive information disclosure, deserialization risks, file upload flaws, authentication and authorization logic errors, and sensitive data written to logs. A dataset covering only a few types can easily lead to one-sided conclusions.
Evaluation design should therefore not chase case count alone. It should also focus on whether cases are close to real engineering practice and whether they genuinely exercise complex code comprehension and logical reasoning.
3. Recall Is Only the Starting Point: Stable Hits Are What Matter
In vulnerability detection evaluation, recall usually refers to the proportion of known vulnerable cases in which the model successfully identifies the problem. It is intuitive and easy to understand, which is why it is often used for rankings. Recall is measured in several different ways, though, and treating them as interchangeable can easily lead to misleading conclusions.
Common measurement methods include:
First-run recall
Each case is run once, and the model is scored on whether it finds the vulnerability. This metric reflects performance on a single attempt and is suitable for assessing a model’s “average” capability.Pooled recall across multiple runs
Each case is run several times, and the model is considered capable of finding the vulnerability if it succeeds in any run. This metric tends to inflate the overall score, since it gives the model multiple chances to get it right.Stable hits
The model must find the vulnerability in every run. This metric better reflects reliability and is well suited to assessing stability in production environments.First successful round
This is the average number of inference, search, or tool-call rounds the model needs before hitting the target. It is directly tied to cost, latency, and engineering efficiency.
As an example, suppose each case is run three times. The metrics can be computed as follows:
total_cases = len(cases)runs = 3first_run_hits = 0pooled_hits = 0stable_hits = 0for case in cases:run_results = [run_case(model, case) for _ in range(runs)]if run_results[0]:first_run_hits += 1if any(run_results):pooled_hits += 1if all(run_results):stable_hits += 1report = {"first_run_recall": first_run_hits / total_cases,"pooled_recall": pooled_hits / total_cases,"stable_recall": stable_hits / total_cases,}
As this simplified flow shows, pooled recall and stable recall can differ substantially. A model with high pooled recall but low stable recall shows some potential. Yet its inconsistent outputs make it hard to use directly in automated security gates or high-risk scenarios.
For enterprises, stable hits often matter more than the “highest recall.” A security system that only catches vulnerabilities occasionally cannot support a trustworthy automated loop.
4. Control the Variables: Make Results Reproducible
Reproducibility is another major challenge. Many evaluation results look impressive, but the numbers can shift noticeably once you change the prompt, organize the context differently, or use a different code indexing strategy. A rigorous evaluation therefore needs to control the variables.
The following experimental conditions usually need to be specified:
Input scope
Can the model see only the current function, the entire file, related modules, or a summary of the whole repository? The scope of context can dramatically affect results.Tool use
Can the model use code search, symbol indexing, static analysis, dependency queries, and other aids? Tools can boost performance, but they also add complexity and cost.Internet access
With internet access, the model may look up vulnerability advisories or patch information. Without it, results depend more on code understanding alone. The settings should match the evaluation objective.Maximum number of reasoning rounds
Too few rounds may leave the model no time to explore. Too many drive up cost and can introduce noise.Output format constraints
Should the model provide file paths, line numbers, vulnerability types, root causes, impact scope, and remediation suggestions, or simply judge whether something is a vulnerability? Structured output makes automated scoring easier.Scoring rules
Does mentioning a relevant file earn a point, or must the model hit the key function, critical logic, or patch location? Scoring that is too lenient overestimates model capability. Scoring that is too strict can overlook the model’s value as an aid.
A fairly rigorous evaluation process might look like this:
Prepare the case set↓Initialize the code repository and runtime environment↓Run the model through multiple rounds of inference↓Record output paths, vulnerability types, evidence chains, and rounds↓Match against ground truth and score↓Generate recall, stable hit rate, cost, and false positive metrics↓Have humans review disputed cases
The value of this workflow is that it turns evaluation from a “gut feeling about which model is stronger” into a trackable, reviewable, repeatable engineering experiment.
5. Don’t Overlook Cost and Efficiency
Vulnerability detection models are not lab toys. Ultimately, they have to fit into the development process, so cost must be part of the evaluation. Cost includes not only model invocation fees, but also the following:
Cost per run
This depends on the length of the input code, context summaries, output length, the number of tool calls, and the number of inference rounds.Average cost per finding
This can be roughly expressed as:Average cost per finding = Total cost / Number of vulnerabilities stably identified
A model with slightly higher recall may need more rounds, longer context, and more frequent calls. If so, its actual cost could be far higher than the leaderboard suggests.
Latency and developer experience
If a model takes a long time to reason, it may not suit real-time checks on code commits. Higher latency is more acceptable for deep audits, scans before a release, or targeted security investigations.Human review cost
LLM output may contain false positives or explanations that are too thin. If every result requires security experts to spend a lot of time reviewing it, overall efficiency may not actually improve.
Evaluation results should therefore present effectiveness and economic metrics side by side. For example:
Model A: higher recall, but more rounds per case and higher costModel B: slightly lower recall, but good stable hit rate and lower costModel C: average on its own, but well suited to use alongside static analysis tools
Different models can play different roles. High-recall models suit initial triage and lead discovery. Models with high stability are better suited to gate validation. Models with strong explanatory power are good at generating reports and assisting human audits.
6. High Recall ≠ High Usability: Reading Evaluation Results Correctly
When reviewing evaluation results, try to avoid three common misjudgments.
First, don’t mistake pooled recall for reliable single-run recall.
If a model needs three runs to find most cases, it may also need multiple attempts in real use. For automated security scans, this instability adds complexity to the workflow.
Second, don’t focus only on the total number of vulnerabilities and ignore severity.
Critical vulnerabilities, edge cases, and business logic flaws are often more valuable than simple, templated ones. If a dataset consists mostly of simple cases, a model’s score may be inflated while its actual defensive value remains limited.
Third, don’t treat model output as the final verdict.
LLMs are best used as assistants for security analysis, not as automatic decision-makers that run without human oversight. For high-risk vulnerabilities, core business code, and compliance-sensitive scenarios, human review and regression verification should remain in place.
A more sensible way to read results is to examine:
- how findings are distributed across vulnerability types;
- how the model performs with complex context;
- whether it provides an evidence chain;
- how easy its output is for engineers to review; and
- whether it can be used sustainably under cost constraints.
7. How Enterprises Can Build Their Own LLM Vulnerability Evaluation System
To genuinely assess LLM vulnerability detection capabilities, enterprises should look beyond external evaluations and build an internal evaluation loop.
7.1 Build an Internal Case Set
You can assemble cases from historical vulnerabilities, security audit records, internal defect databases, and sanitized code. Each case should include:
- Vulnerability background
- Relevant code snippets
- Ground truth root cause
- Fix approach
- Impact scope
- Scoring points
Internal cases are closer to the enterprise’s own business. They are also less likely to have been memorized directly by the model.
7.2 Set Up an Isolated Runtime Environment
Evaluation code should run in an isolated environment so that it never touches real production data or sensitive credentials. A common pattern is to run evaluation tasks on a container platform, store sanitized samples in object storage, use a logging service to record execution details, and rely on monitoring and alerting to track task status and resource consumption. On Baidu AI Cloud, these general-purpose cloud capabilities can be combined into a fairly complete evaluation pipeline.
7.3 Fix the Scoring Rules
Break the score down into multiple dimensions:
Whether the vulnerability type is identifiedWhether the key file is locatedWhether the key function or code line is locatedWhether the root cause is explainedWhether a reasonable remediation direction is providedWhether there is any obvious false positive
This avoids crude scoring in which a model gets credit merely for mentioning a vulnerability keyword.
7.4 Run Regression Tests
After a model version update, rerun the case set and check whether recall, stable hit rate, false positives, and cost have regressed. Security evaluation is not a one-off activity. It should be an ongoing engineering practice with continuous regression testing.
7.5 Combine Static Analysis and Human Audits
LLMs should not replace every traditional security measure. Instead, they should form a layered defense together with static scanning, dependency analysis, secret detection, and human audits. High-confidence findings can be blocked directly, while low-confidence leads are routed to a human review queue.
8. Practical Recommendations for Adoption
Overall, the goal of LLM vulnerability detection evaluation is not to prove that one model is “absolutely the best.” It is to answer a set of engineering questions:
- Can the model consistently find high-value vulnerabilities in real code?
- Is its output explainable and easy to review?
- Is its cost acceptable?
- Does it fit into the existing development workflow?
- Does it remain reliable in complex business scenarios?
- Does its capability hold up, or improve, after updates?
Development teams should start with low-risk use cases, such as code comment assistance, suspected risk hints, patch description generation, and security report summaries. From there, they can gradually move into higher-risk scenarios, such as pre-commit checks, pre-release audits, and vulnerability discovery in critical modules. This approach keeps risk under control while building up evaluation data.
Security teams, meanwhile, should treat evaluation as an ongoing governance process rather than a one-time procurement or ranking exercise. With a case library, a metrics dashboard, regression tasks, and a human review mechanism, they can gradually build a quantifiable, auditable, and continuously improvable LLM security capability system.
Overall, the potential of large language models in vulnerability detection keeps growing. Evaluation, however, must shift from a “leaderboard mindset” to an “engineering mindset.” Recall reflects a model’s ability to find issues, while stable hit rate better reflects its usability in production. Cost, latency, false positives, and explainability ultimately determine whether it can genuinely fit into security workflows. Only with a rigorous, reproducible, regression-ready evaluation system can enterprises assess and adopt LLM security capabilities more rationally.
