Large Language Model Security Evaluation: Balancing Recall, Stability, and Cost

    Evaluating LLMs for security work means answering a harder question than "how accurate was this run?" This article explains why single-run accuracy is not enough for security operations, code auditing, and threat intelligence analysis, how pooled (best-of-N) recall can inflate the apparent quality of a model, and why stable hits across repeated runs matter more than peak scores. It also shows how to factor repeated-run token consumption into an effective cost per unit of recall, and provides a reproducible multi-run evaluation harness in Python, along with practical guidance on choosing between a top-performing but volatile model and a cheaper, more dependable tier.

    1. Why security use cases demand stricter LLM evaluation

    When large language models are introduced into security operations, code auditing, or threat intelligence analysis, one question is unavoidable: how many real issues can the model actually find? Conventional capability benchmarks usually report the overall accuracy of a single run, but in security work a good number from one scoring pass is not enough to support a decision. The core requirement of security tasks is not to miss anything, which means evaluation must check not only whether the model can find problems, but also whether it can find the same problems consistently across multiple independent runs. Evaluation design therefore has to move beyond single-run scores toward a combined framework of multi-run statistics and cost constraints.

    2. Recall: don’t be fooled by pooled recall

    Recall in security discovery tasks is the number of real issues the model finds divided by the total number of real issues. One easily overlooked distinction matters here:

    • Single-run recall: the proportion the model catches in one particular run.
    • Pooled (best-of-N) recall: over multiple runs, a hit counts as a hit if any run caught it, then the de-duplicated total coverage is measured.

    The two can differ dramatically. Consider an illustrative scenario on an evaluation set containing 32 known security samples. A frontier-class model might reach a pooled recall of 28/32 across several runs, which looks close to perfect. But its single run might hit only around 17 samples the first time, and only around 10 samples might be hit consistently across three runs. A high pooled recall is largely an optimistic number produced by taking the union of multiple attempts, not evidence of stable model capability.

    This has a direct impact on model selection. If the model will only be run once in production, single-run recall is what actually matters. If you can run it several times and take the union, pooled recall becomes a meaningful reference, at the cost of repeated computation.

    3. Stability: the hidden metric that matters more than peak performance

    More telling than pooled recall is the number of stable hits — the problems hit in every one of several runs. Why does this matter so much?

    1. It approximates real production conditions. Online tasks rarely give the same input several chances to take a union; stable hits represent the lower bound of model capability.
    2. It exposes the role of luck. A model may happen to get many items right in one round, but that performance cannot be reproduced or guaranteed.
    3. It bears directly on missed-detection risk. In a security context, a model that hits only 10 samples consistently implies that a large share of samples could be missed at any time.

    A practical approach is to report three numbers together — first-run recall, pooled recall, and stable recall — and examine the gap between them. The larger the gap, the more unstable the model’s output, the more it depends on the number of runs and on randomness, and the less suitable it is for security pipelines that require determinism.

    4. Cost: put “catching up to the frontier” on a price axis

    The recall of frontier models is often only marginally higher than that of the next tier down, while the cost can differ by orders of magnitude. A common pattern in evaluation is that a model trails the strongest one by a single sample (for example, 25/32 versus 26/32) while costing roughly 70% less. In that situation, the balance clearly tips toward value for money.

    When assessing cost, look beyond the per-call price and include the number of repeated runs needed to reach a target recall. If a model has low single-run recall and must be run many times to match the union coverage of another, its effective cost — the total cost of reaching equivalent coverage — is actually higher. A better common yardstick is therefore the cost per unit of effective recall, rather than a raw comparison of token prices.

    5. A reproducible multi-run evaluation framework

    The pseudocode below shows how to turn these ideas into a reproducible evaluation process:

    1. import numpy as np
    2. def evaluate_model(model, benchmark, runs=3, temperature=0.7):
    3. """
    4. benchmark: a set of security samples with known answers (N items)
    5. runs: number of independent runs per sample (used to measure stability)
    6. """
    7. N = len(benchmark)
    8. hit_matrix = np.zeros((runs, N), dtype=bool) # records whether each run hit
    9. total_tokens = 0
    10. for r in range(runs):
    11. for i, item in enumerate(benchmark):
    12. result, tokens = model.solve(item, temperature=temperature)
    13. hit_matrix[r, i] = result.correct
    14. total_tokens += tokens
    15. # 1. Single-run recall (per run)
    16. single_recall = hit_matrix.mean(axis=1) # length: runs
    17. # 2. Pooled recall (union)
    18. pooled_recall = hit_matrix.any(axis=0).mean() # hit at least once
    19. # 3. Stable recall (hit every time)
    20. stable_recall = hit_matrix.all(axis=0).mean() # hit in all runs
    21. return {
    22. "first_run_recall": single_recall[0], # first-run performance
    23. "mean_single_recall": single_recall.mean(),
    24. "pooled_recall": pooled_recall, # optimistic upper bound
    25. "stable_recall": stable_recall, # reliable lower bound
    26. "stability_gap": pooled_recall - stable_recall, # variability
    27. "cost_per_recall": total_tokens / (N * pooled_recall) # effective recall cost
    28. }

    Key design points:

    • Fix the benchmark and the scoring criteria. The sample set, the correct answers, and the hit-determination logic must be public and consistent, or comparisons are meaningless.
    • Run multiple independent passes. At least 3 runs, preferably 5 or more, using randomized temperature to observe the output distribution rather than a single snapshot.
    • Report all three recall numbers. First-run, pooled, and stable recall, so the optimistic value and the reliable value are immediately visible.
    • Include cost accounting. Convert the token consumption of repeated runs into an effective recall cost instead of comparing per-call prices alone.
    • Control variables. Compare different models on the same benchmark, with the same scoring criteria and the same budget; only then are the conclusions trustworthy.

    6. Practical conclusions and recommendations

    Overall, judging whether an LLM is reliable on security discovery tasks should not rest on how impressive it looked in one scoring run. Three questions matter:

    1. Does its single-run recall meet production requirements? If only one run happens online, stable hits and first-run hits are the basis for a decision.
    2. How stable is its output? The gap between pooled recall and stable recall reflects the model’s determinism directly; the larger the gap, the more caution is warranted.
    3. How much cost does the target coverage require? Comparing models on effective recall cost rather than per-call price is the only way to see real value.

    A more rational selection logic is to treat stable recall as the capability lower bound, pooled recall as the optimistic upper bound, and effective recall cost as the economic yardstick, then weigh the three together. For security pipelines that prioritize determinism and controllable cost, a better-value model from the next tier down is often a better choice than the strongest model with flashy peak scores but high volatility and high cost. The purpose of evaluation is precisely to quantify these hidden differences, so decisions rest on reproducible data rather than single-run luck.