LLM Security Detection Evaluation: Recall, Stability, and Cost

    Evaluating large language models for security tasks such as vulnerability discovery, log anomaly detection, and threat intelligence classification is not a matter of reading a single recall number. This article explains why LLM output on security tasks is inherently non-deterministic, how single-run recall and pooled recall can diverge sharply, why cross-run consistency deserves more attention than the average score, and how unit cost becomes the deciding factor once leading models are within one or two samples of each other. It also provides a reusable four-step evaluation methodology, sample scoring code, and practical guidance for engineering teams.

    1. Why Evaluating LLMs for Security Detection Is Harder Than It Looks

    When enterprises bring large language models into security work such as vulnerability discovery, log anomaly detection, and threat intelligence classification, model selection almost always runs into the same question: how do you objectively determine whether a model is actually usable?

    On the surface, recall is the most intuitive metric: given a fixed sample set, how many of the real issues did the model find? In practice, however, treating the recall from a single evaluation run as the final verdict almost always leads you astray.

    The reason is that LLM output on security tasks is strongly non-deterministic. With the same prompt and the same input samples, repeated runs can produce different hits. An issue the model missed in one run may be caught in the next, and the issues it does catch consistently may be only a fraction of the total. So the right way to evaluate is not “run it once and record one number” but to understand the underlying distribution across multiple samples.

    2. Single-Run Recall vs. Pooled Recall: An Easily Missed Distinction

    Two related concepts need to be separated here.

    Single-run recall is the proportion of issues a model actually catches in each individual evaluation run. Pooled recall is computed by taking the union of results across multiple runs, meaning an issue counts as a hit if any run found it.

    The gap between these two can be decisive. Consider a typical evaluation: a frontier model reaches a pooled recall of 28/32 across three runs and ranks first among the participants, which looks impressive. Break it down, though, and its first run caught only 17 items, while just 10 items were caught consistently in all three runs. In other words, the high pooled recall depends largely on the statistical effect of “run it a few more times and one of them will get lucky,” rather than on stable detection capability.

    The implications for engineering are direct:

    • If your detection pipeline only allows one run per sample because of cost or latency constraints, the number that matters is single-run recall, not the attractive pooled figure.
    • If your system can sample multiple times and aggregate results (for example, via voting, union, or weighted fusion), then pooled recall becomes meaningful, provided the aggregation logic is sound.

    A simple illustration of the difference between aggregation and single runs:

    1. # runs: the set of issues found in each run
    2. runs = [set(run1), set(run2), set(run3)]
    3. # Pooled recall: union of all runs
    4. pooled_hit = set().union(*runs)
    5. # Single-run recall: count each run separately
    6. single_hits = [len(r) for r in runs]
    7. # Stable hit: issues that appear in every run
    8. stable_hit = set.intersection(*runs)
    9. total = 32
    10. print("Pooled recall:", len(pooled_hit) / total)
    11. print("Single-run recall:", [round(h / total, 3) for h in single_hits])
    12. print("Stable hit ratio:", len(stable_hit) / total)

    The stable_hit value above (the three-run intersection) is the only figure that reflects detection capability the model reliably has. pooled_hit (the union) is closer to an optimistic upper bound. Reporting only the latter in an evaluation write-up will easily overstate model reliability.

    3. Consistency: A Quality Dimension Worth More Than the Average

    If recall answers “can it find the issue,” then consistency answers “can it find it reliably, again and again.” This is what distinguishes security detection from general question answering.

    In real security operations, a model that “occasionally finds things but frequently misses” can be riskier than a model with slightly lower recall but stable behavior. False negatives are insidious: the security team develops misplaced trust because “it caught this last time,” relaxes manual review, and real threats slip through. Poorly consistent models also carry a hidden cost: results are not reproducible, which makes regression testing, audit trails, and alert baselines impossible to establish.

    For that reason, consistency measures are worth adding explicitly to the evaluation scorecard, such as:

    • Stable hit rate: the proportion of all positive samples that are caught in every run.
    • Recall variance / standard deviation: how much performance fluctuates across runs.
    • First-run hit rate: a proxy for the harshest single-call scenario.

    A model with 25/32 recall, very low variance, and near-identical hits in almost every run is usually more trustworthy in engineering terms than one with 28/32 recall and wild swings. When you evaluate, put both types on the same radar chart or comparison table so decision-makers see the full picture instead of fixating on a single maximum.

    4. Cost: The Second Battleground Once Recall Catches Up with the Frontier

    Another trend worth noting is that recall gaps between model tiers are narrowing as capabilities improve. As an example from the original evaluation data, a newer model’s production version reached a recall of 25/32, only one item behind the top-ranked frontier model, while costing substantially less (at roughly 70%).

    For engineering decisions, this marks an important turning point: once the recall gap shrinks to one or two samples, selection should shift from “recall above all” to joint optimization of recall and cost.

    Security detection is typically a large-scale, high-frequency workload: massive volumes of logs, code commits, and traffic samples processed every day. At that scale, per-call cost is multiplied by call volume and becomes the dominant driver of total cost of ownership. A model with slightly lower recall but much lower cost can, within the same budget, support more sampling rounds or broader detection coverage, and therefore deliver a better outcome at the system level.

    When building a cost model, weigh multiple dimensions rather than unit price alone:

    • Cost per token, together with the average tokens consumed per detection (affected by input length, output length, and whether long context is used).
    • Sampling rounds required to reach the target recall: if a model needs three runs to catch up, its real cost is three times higher.
    • Retry and error-correction overhead: poorly consistent models inflate wasted calls.

    A pragmatic approach is to define metrics such as “recall per unit cost” or “best recall within budget” and compare models under a fixed budget constraint, instead of looking at any single metric in isolation.

    5. A Reusable Evaluation Methodology

    Combining these dimensions, here is a four-step evaluation process:

    1. Define the call pattern. Determine whether production uses a single call or multiple calls with aggregation. This decides whether single-run recall or pooled recall should be the primary metric.
    2. Sample multiple times and record by layer. Run at least 3–5 rounds, record the hit set for each round, and compute the union, intersection, and variance. Never run only once.
    3. Build a multi-dimensional scorecard. Include single-run recall, pooled recall, stable hit rate, recall variance, and recall per unit cost to form a complete profile.
    4. Weight by scenario when deciding. Assign weights to each metric based on the business’s tolerance for false negatives, budget constraints, and latency requirements, then produce an overall ranking.
    1. def evaluate_model(runs, total_positives, unit_cost):
    2. pooled = len(set().union(*runs)) / total_positives
    3. singles = [len(r) / total_positives for r in runs]
    4. stable = len(set.intersection(*runs)) / total_positives
    5. variance = statistics.pvariance(singles)
    6. # Weights for the composite score should be adjusted per business needs
    7. score = 0.3 * stable + 0.3 * (sum(singles)/len(singles)) \
    8. + 0.2 * pooled + 0.2 * (1 / (1 + variance)) \
    9. - 0.1 * unit_cost
    10. return {"stable": stable, "variance": variance,
    11. "score": score}

    6. Practical Engineering Recommendations

    A few lessons from real deployments are worth keeping in mind:

    • Establish a reproducible evaluation baseline. Fix the sample set, prompt templates, and sampling parameters so that comparisons between models happen on the same scale. Persist evaluation samples and expected results in a dedicated logging service with monitoring and alerting so regressions are easy to catch.
    • Use aggregation to offset single-run variance. If cost allows, combining multiple sampling rounds with voting or union fusion can meaningfully raise effective recall while reducing the risk of an occasional missed detection.
    • Put cost into the CI gate. Make “recall per unit cost” one of the admission criteria for shipping a model, so you do not pay a disproportionate price for marginal gains.
    • Re-evaluate continuously. Models keep iterating on their versions, so evaluation should be an ongoing practice rather than a one-time selection exercise.

    7. Conclusion

    Evaluating an LLM’s security detection capability is essentially about finding the best business balance among recall, stability, and cost, rather than chasing a single leaderboard position. An attractive pooled recall can mask volatility risk, while a recall close to the frontier combined with a meaningful cost advantage is often the more realistic engineering choice. Putting single-run recall, cross-run consistency, and unit cost into the same scorecard is what makes selection decisions hold up in production.