# LLM Security Evaluation: How to Balance Recall, Consistency, and Cost
As large language models move into vulnerability discovery, code auditing, and threat intelligence analysis, teams are spending enormous compute to benchmark their security detection abilities. This article explains why a single "top recall" number is misleading, how single-run and pooled recall differ, why consistency matters for automated security pipelines, and how to fold cost into the evaluation model. It includes a Python snippet for measuring stable hit rates and a short checklist for designing a reproducible security benchmark.
Background: Why Security Needs to Rethink Its Metrics
As large language models are increasingly applied to vulnerability discovery, code auditing, threat intelligence analysis, and other security work, the industry has started investing massive compute in systematically benchmarking how well they detect security issues. Recently, one security team spent tens of billions of tokens running a large-scale security capability comparison across several frontier LLMs.
The problems that surfaced are worth a closer look. One model had the highest recall under a “merge three runs” methodology and looked impressive, but a per-run breakdown showed it hit only about half of the targets on its first attempt, and even fewer targets were hit in all three runs. Meanwhile, another model with dramatically lower cost already had recall close to the top tier. Taken together, these findings suggest that reading a single “highest recall” figure can easily lead to a misjudgment of a model’s security capability.
This article does not review specific products. Instead, it extracts the methodological questions that matter: how recall should be computed, why consistency matters, and how cost should enter the picture.
1. Get Recall Right First
In security detection tasks, recall is the share of risk items that should have been identified but were actually caught:
Recall = correctly identified risk items / total risk items that actually exist
Why does security care about recall so much? Because a false negative costs far more than a false positive. A single high-severity vulnerability that goes undetected may lead directly to a system compromise, whereas a false positive only adds manual review effort. In scenarios where “better to over-flag than to miss,” such as vulnerability detection and threat hunting, recall is usually the primary metric.
The caveat: high recall does not necessarily mean high usability. If a model chases recall by outputting aggressively, precision drops sharply and can overwhelm downstream alerting systems. Recall therefore has to be read together with precision and F1; no single number is complete on its own.
2. “Pooled Recall” and “Single-Run Recall” Are Not the Same Thing
This is the most interesting point from the comparison.
LLMs are stochastic: the same input can produce different outputs across calls. Different evaluation methodologies can therefore lead to very different conclusions.
- Single-run recall: run once and measure the hit rate. It reflects how well the model performs in one shot.
- Pooled recall: run multiple times (three, say) and count a risk item as hit if any run caught it, then take the union.
Pooled recall is naturally higher, because it effectively gives the model several attempts. Back to the example above: one model reached a pooled recall of 28/32, but hit only 17/32 on the first attempt, with just 10 items hit in all three runs. Its actual stable capability is therefore closer to 17, and a substantial part of the 28 was assembled by stacking multiple attempts.
In one sentence: pooled recall measures the upper bound of potential, while single-run recall and the stable hit count are much closer to production usability.
3. Consistency: The Engineering Metric That Matters More Than a Single High Score
If a model is going into an automated security pipeline, consistency is unavoidable. The reason is simple: security workflows depend on reproducible, predictable behavior.
- If a model performs unevenly, finding a vulnerability today but missing it tomorrow, the security team cannot build trust in it.
- If automated alerts fluctuate frequently, they generate a lot of noise and actually reduce response efficiency.
A practical way to assess consistency is a “stable hit rate”: across N independent runs, what is the distribution of hits for each target? It can be computed as follows:
def analyze_consistency(run_results):"""run_results: a list of lists; each inner list holds the target ids hit in one runreturns: stable hits, average recall, and the standard deviation of recall"""import statisticstotal_targets = len(set().union(*map(set, run_results)))hit_counts = {}for run in run_results:for target_id in set(run):hit_counts[target_id] = hit_counts.get(target_id, 0) + 1n_runs = len(run_results)# hit in every run is the strictest definition of a stable hitstable_hits = sum(1 for c in hit_counts.values() if c == n_runs)per_run_recall = [len(set(run)) / total_targets for run in run_results]return {"total_targets": total_targets,"stable_hits (hit in every run)": stable_hits,"mean single-run recall": round(statistics.mean(per_run_recall), 4),"recall standard deviation": round(statistics.stdev(per_run_recall), 4) if n_runs > 1 else 0,}
The smaller the standard deviation, the more stable the model’s behavior. A model with a high mean recall but a large standard deviation is actually harder to operate in production.
4. Cost Must Be Part of the Model, Not Just a Benchmark Score
Security detection is a high-frequency, large-scale task, and cost directly determines whether a solution can actually be deployed. Evaluations should establish a notion of “cost per effective detection” rather than looking at unit price or benchmark scores in isolation.
A reasonable cost assessment framework should include:
- Token consumption: large-scale evaluation or production use consumes enormous numbers of tokens, so both total consumption and per-call consumption need to be tracked.
- Cost efficiency: measure recall per unit cost, or cost per successful vulnerability detection.
- Marginal return: when a stronger model recalls only one or two additional samples compared with a mid-tier model, ask whether the price difference is justified.
The case at the start is representative: one model recalled only one sample fewer than the top tier while costing nearly 70% less. In most engineering scenarios, this “good enough and cheap” option is usually more pragmatic than pushing quality to the limit.
5. Methodology: How to Design a Trustworthy Security Capability Comparison
For conclusions to be credible, methodology matters more than any single number. Pay attention to the following:
- Authoritative benchmark set: does the evaluation use industry-recognized, accurately labeled vulnerability or threat datasets as ground truth, rather than a self-selected sample set?
- Multiple runs, report the distribution: run at least three independent times and report the mean, variance, and stable hit rate, not just the highest value.
- Transparent methodology: clearly distinguish single-run recall, pooled recall, and precision, and avoid presenting pooled recall as if it were real-world performance.
- Coverage disclosure: state honestly which model tiers or capability levels the round did not cover, so readers do not assume the conclusion is universal.
- Reproducibility: publish prompts, randomness parameters, and scoring criteria so third parties can reproduce the results.
Randomness parameters have a large effect on results, so fix them or report them explicitly:
# fix randomness-related parameters during evaluation to keep it reproducibleeval_config = {"temperature": 0.0, # keep randomness as low as possible for reproducibility"runs": 3, # multiple runs to observe consistency"metric": ["recall", "precision", "f1", "stable_hit"],}
6. Recommendations for Practitioners
Pulling all of this together, when facing LLM security capability evaluations:
- Do not be swayed by a single “highest recall” figure; always ask whether it is single-run or pooled.
- Include consistency in your decision, especially for pipelines that need automation.
- Choose models on cost efficiency rather than absolute scores; in most scenarios, a “good enough and cheap” model is the better deal.
- Build your own evaluation set and validate repeatedly against your real business risk distribution, instead of copying someone else’s leaderboard.
For adopters, a more robust approach is often “tiered invocation”: use a low-cost model for a broad first-pass screen, then call a stronger model to re-check items that look high-risk, balancing recall against cost.
Conclusion
The real value of a security model evaluation that consumes tens of billions of tokens lies not in the ranking of who came first, but in the methodological details it exposes: how recall is defined, the stable hit count, and the trade-off between cost and consistency. Understanding these is what allows you to avoid being misled by an attractive single metric and to build a genuinely reliable, deployable LLM security capability assessment system.
