LLM Security Evaluation Metrics: Recall, Consistency, and Cost

    This article breaks down a recent LLM vulnerability audit benchmark that ran frontier models three times over 32 real vulnerabilities and consumed about 11.7 billion tokens. It explains how to define recall (single-run, pooled, and stable), measure consistency, account for false-positive and retry costs, compare models by cost per true positive, and recognize the limits of small-sample rankings. It also covers why LLM outputs drift (sampling randomness, context dilution, and prompt design) and how to build a tiered pipeline that combines static analysis, lightweight model triage, strong-model deep review, and human verification.

    1. Introduction: What 11.7 Billion Tokens Reveal About Method

    A security vendor recently published a round of LLM vulnerability audit evaluation. On a benchmark of 32 real vulnerability samples, it ran several frontier models three times each and consumed about 11.7 billion tokens in total.

    The headline results were roughly as follows:

    • Model A (an open-weight model family) had the highest pooled recall across three runs (28/32). However, it found only 17 in the first round and hit only 10 in all three rounds.
    • Model B (an overseas closed-source flagship) recalled 26/32, with the fewest lucky hits. Its cost-effectiveness advantage was not pronounced.
    • Model C (a recent release from a China-based vendor) recalled 25/32 in its GA build, one short of Model B, at a cost about 69.8% lower.
    • The round did not cover the most capable flagship tier.

    A rough order-of-magnitude calculation puts the scale in perspective: 11.7 billion tokens ÷ 32 samples ÷ 3 rounds ÷ 4 models is about 30 million tokens per sample per round. This is only an estimate derived from public figures and may differ from the tester’s actual orchestration. At per-thousand-token pricing, a single-sample, single-round audit could cost hundreds of currency units. Repeating this kind of audit across a full codebase every day would quickly make costs spiral out of control.

    These numbers are more valuable than the ranking. They show that when a generative LLM processes code, it is essentially making probabilistic judgments inside a sampling process with inherent randomness. Concretely:

    • Recall shifts depending on how the evaluation is defined.
    • Marginal cost determines cost-effectiveness.
    • A one-off high recall does not equal sustainable engineering capability.

    This article therefore does not argue about which model is stronger. It first defines a rigorous set of metrics, interprets the measured data above against each one, and finally proposes a practical tiered integration plan.

    2. Breaking Down the Evaluation Method: What a Credible Conclusion Requires

    2.1 The benchmark: 32 samples, clean but statistically limited

    The samples came from real security vulnerabilities that had already been fixed. This gives them several strengths:

    1. Confirmed ground truth. Each sample is a human-verified vulnerability, so its ground truth is clear and no second-party adjudication is needed.
    2. Real code, not synthetic examples. The samples come from actual codebases and include real-world noise such as cross-file calls, implicit data flows, and incomplete context.
    3. Clear labeling. Because the fixes are already in place, the faulty locations and root causes can be clearly annotated.

    Even so, 32 samples is a small set. Under a simple normal approximation, when a sample’s true hit probability is about 78%, observing 28/32 hits could simply be “a bit of luck.” It does not prove that the model’s true recall is 87.5%. Rankings become more stable only as the sample size grows and the distribution of vulnerability types becomes more balanced.

    2.2 The audit pipeline: constrained consistency, recall-only scoring, three rounds

    • Tool-assisted auditing. Models were given tools for reading files, searching symbols, and jumping across files. This more closely resembles how developers actually use AI coding assistants, rather than asking the model to “guess” an entire repository out of thin air.
    • Scoring rule. Credit was given only for hitting vulnerabilities on a list predefined by the testers. Whether the location was correct or the description accurate did not count toward the score. This is reasonable for a “recall-first” research setting, but it is still a step away from production readiness.
    • Repeated sampling. The same model with the same prompt was run for 3 rounds. This is one of the most valuable design choices in the evaluation, because it splits “capability” and “stability” into two separately observable dimensions.

    2.3 Hidden costs: tokens are not the only expense

    Beyond model call fees, the assessment should also account for the following hidden costs:

    • Retries and multi-turn calls. Failed tool calls, overlong context, and non-compliant output formats all trigger retries.
    • Long-context caching. When code search results are fed to the model repeatedly, cached and uncached portions are typically priced several times apart, sometimes by an order of magnitude.
    • Human review. Once a vulnerability is reported, someone still has to confirm whether it is real and reproducible.
    • Trust erosion from false positives. Results with too much noise eventually get switched off by engineers within the workflow.

    3. Five Core Metrics: Seeing Which Model Is Actually More Useful

    3.1 Recall: three definitions, three conclusions

    Vulnerability auditing is a classic “better to over-report than to miss” scenario. A missed vulnerability reaches production, at a far higher cost than a few extra manual reviews. Recall is therefore the primary metric, provided its definition is explicitly fixed in advance.

    Metric Definition Results this round Typical scenarios
    Single-run recall (first round) Counts only the first run Model A: 17/32 (53.1%) One-shot scans such as end-to-end scans or CI gates
    Pooled recall (union) After deduplicating and merging three rounds, how many ground-truth vulnerabilities were hit Model A: 28/32 (87.5%); Model B: 26/32; Model C: 25/32 Deep audits and periodic full scans
    Stable recall (intersection) Ground-truth vulnerabilities hit in all three rounds Model A: 10/32 (31.3%) Continuous monitoring and the baseline for alert trustworthiness

    Key insight: pooled recall is inherently optimistically biased. A vulnerability counts as long as it was found in any round, so the score rises with the number of rounds. Pooled recall measures the upper bound of the process, not the actual output of a single audit. Treating pooled recall as the sole conclusion therefore systematically overestimates a model.

    3.2 Stability: consistency and uncertainty

    Stability can be measured along two dimensions:

    • Absolute: the number of ground-truth vulnerabilities hit in all three rounds. For Model A, this was 10.
    • Relative: stable hits ÷ pooled hits. For Model A, 10 ÷ 28 ≈ 35.7%, meaning nearly two-thirds of the pooled hits were found “by luck.”

    By comparison, Model B hit 26 vulnerabilities in the first round alone and had 18 stable hits. Its single-run output was 2 lower than Model A’s, but only 8 were “lucky” hits, giving a relative stability of 69.2%.

    The engineering implication is direct: in scenarios that need trustworthy alerts, stable recall should be the primary acceptance metric. Pooled recall is meaningful only in deep audits where a human can review findings one by one.

    3.3 Precision and false-positive cost: the underestimated hidden metric

    This evaluation did not measure false positives. Yet the real engineering cost is:

    1. Total cost of one audit = Model call fees
    2. + False positives × Manual review time per finding × Labor cost
    3. + Missed vulnerabilities × Remediation cost per miss

    A model with slightly lower recall but very few false positives can have a lower total cost in practice. This is especially true for teams where labor is expensive and review alignment is difficult. When selecting a model, you should ask vendors for false-positive counts under the same prompt, not recall alone.

    3.4 Cost per finding: translating price into “cost per real vulnerability”

    Model C’s cost was about 69.8% lower, which means its cost was about 30.2% of Model B’s.

    A rough estimate based on “cost per true-positive vulnerability discovered” (an order-of-magnitude illustration only):

    • Model B: 1 ÷ 26 ≈ 0.0385. If a single audit costs 100 currency units, each vulnerability costs about 3.85 currency units.
    • Model C: 100 × 0.302 ÷ 25 ≈ 1.21 currency units.

    Under lenient assumptions, Model C’s cost-effectiveness is about 3.2 times that of Model B. But this conclusion is fragile:

    1. Including retries and review costs will enlarge or shrink the ratio. If Model C produces more false positives, its adjusted cost-effectiveness will drop noticeably.
    2. Harder samples (such as cross-service vulnerabilities) may widen Model B’s relative advantage, eroding Model C’s cost edge.
    3. Tiered volume discounts distort static comparisons. At large scale, volume pricing, cache hits, and batch discounts matter far more than a single-call quote.

    Conclusion: do not select models by unit price. Select by cost per true positive.

    3.5 Coverage boundaries: no cross-tier extrapolation

    Models not covered in the evaluation (such as the strongest flagship tier) cannot have their performance inferred. Likewise, a quality ordering that holds on 32 samples may not hold on 320. An ordering that holds on Java repositories may not hold on Go or Rust.

    Engineering recommendation: run a “shadow evaluation” on your own codebase, using your own vulnerability type list. A shadow evaluation costs far less than one real security incident.

    4. Why Models “Drift”: Three Observable Root Causes

    4.1 Sampling randomness

    Most LLM APIs do not set temperature to 0 by default. Even when it is set to 0, floating-point arithmetic, batching, and the order of parallel reductions on different hardware can still produce subtle differences in output. In other words, “the same prompt, the same input, and the same model producing three different outputs” is the norm, not an anomaly.

    1. Same code snippet, same question, three rounds:
    2. Round 1: flags issues A and B
    3. Round 2: flags issues B and C
    4. Round 3: flags issues A and D

    This explains why pooled recall and stable recall differ so much. The gap is not a capability problem but a sampling variance problem. Engineering controls for variance include lowering randomness parameters, fixing the decoding strategy, adding self-checks and second-pass confirmation, and using majority voting on critical conclusions.

    4.2 Context window and attention dilution

    Once code search results are packed into the context, the model has to locate the critical lines among massive numbers of tokens. The “lost in the middle” effect in long contexts leaves the model torn between “having read every file” and “catching the key line.”

    Countermeasure: inject context in layers. Start with a repository summary and call graph, then fetch relevant files on demand, instead of stuffing the entire repository in at once.

    4.3 Prompts and vulnerability type distribution

    Broad instructions such as “please find the security vulnerabilities in this code” push the model toward the most common issues (for example, injection and broken access control). Low-frequency but high-severity types (for example, insecure deserialization and prototype pollution) tend to be ignored. The prompt should explicitly state the vulnerability list and its priorities, since this directly determines the denominator of recall.

    5. Engineering Integration: Turning “Model Evaluation” into a “Stable Production Line”

    5.1 Tiered architecture: coarse filtering → deep review → human verification

    1. Code change (PR / merge request)
    2. │
    3. ▼
    4. [Layer 1] Rule engine (SAST-style static analysis) ← covers known patterns, zero model cost
    5. │ hit → straight into the fix workflow
    6. ▼
    7. [Layer 2] Lightweight model for batch coarse screening (low cost, high throughput)
    8. │ output: suspicious points + confidence
    9. ▼
    10. [Layer 3] Strong model for deep review (triggered only when Layer 2 flags something)
    11. │ output: vulnerability type, location, exploit path, fix recommendation
    12. ▼
    13. [Layer 4] Human review (high-severity + high-confidence results only)
    14. │
    15. ▼
    16. Alert / ticket / merge block

    The core idea is simple: spend expensive model capability only where it is worth spending. Layer 1 handles “known pitfalls,” while Layer 3 handles “unknown pitfalls.”

    5.2 Prompt template: writing the evaluation criteria into production

    1. You are a code security audit expert. In the given code snippet, report only issues of the following types:
    2. [SQL injection / Broken access control / Insecure deserialization / Hardcoded credentials / Insecure randomness]
    3. Output format (strict JSON, no extra text):
    4. {
    5. "findings": [
    6. {"type": "vulnerability type", "file": "file path", "line": line number,
    7. "reason": "one-sentence root cause", "fix": "one-sentence fix recommendation"}
    8. ]
    9. }
    10. If no issues are found, output {"findings": []}.

    The value of this template is that it keeps the evaluation criteria and the production criteria aligned. If you define a “vulnerability type list” in your evaluation, production must use the same list. Otherwise, the recall numbers will not transfer.

    5.3 Consistency: making “stable hits” the acceptance line

    To counter sampling randomness, you can do three things on the engineering side:

    1. Fix sampling parameters. Set the randomness parameter to 0 and pin the model version snapshot (if the platform supports it).
    2. Multi-sample voting. Run the same code 3 times, and report an issue only if at least 2 runs flag the same type (majority voting).
    3. Second-pass confirmation. Have the model perform a “devil’s advocate review” of its own conclusion (“explain why this code is not a vulnerability”). Use the intersection of the two results as the final conclusion.

    These three measures significantly increase token consumption. However, they can raise “stable recall” from just above 30% to a level close to “pooled recall.” The cost is money; the benefit is trustworthiness.

    5.4 Cost control: spending tokens where they matter

    • Use cheaper models for coarse screening first. The vast majority of code has no vulnerabilities, so auditing every line with the strongest model is wasteful.
    • Enable context caching. Cache repeated content such as repository summaries and call graphs. Cached tokens are usually priced significantly lower than uncached ones.
    • Trim by change scope. Audit only the files touched by the diff and their direct dependencies, not the whole repository.
    • Set token budgets and circuit breakers. Cap each audit; if the cap is exceeded, fall back to “report high-confidence issues only.”
    • Build a “cost per finding” dashboard. Track model spend per true-positive vulnerability weekly as the basis for model selection and tuning.

    5.5 Observability: making every audit traceable

    • Keep audit logs: input summary (without storing raw source), model version, prompt hash, output, token consumption, and elapsed time.
    • Reconcile alerts with tickets: record whether each alert was accepted or turned out to be a false positive, and feed the results back into the evaluation set.
    • Close the false-positive feedback loop: false positives flagged by engineers feed into the next round of prompt and rule optimization.

    6. Implementation Checklist

    • Define whether the goal of this audit is recall-first or trustworthy-alert-first, and choose the metric definition accordingly.
    • Build your own vulnerability type list and shadow evaluation set (covering at least high-frequency and high-severity categories).
    • Select models by “cost per true positive” rather than unit price.
    • In production, fix sampling parameters and model versions to avoid “three results from one prompt.”
    • Apply second-pass confirmation or majority voting to high-severity conclusions.
    • Establish token budgets, circuit breakers, and fallback strategies.
    • Feed false-positive feedback back into the evaluation set to close the loop.

    7. Conclusion: The Value of Evaluation Lies in the Criteria, Not the Ranking

    Of the numbers bought with 11.7 billion tokens, only three are really worth remembering:

    1. Pooled recall is an upper bound, not day-to-day performance. It measures whether a vulnerability can be found, not whether it is found every time.
    2. Consistency is the gate to engineering adoption. Stable recall determines how trustworthy alerts are, and whether teams keep using the tool.
    3. Cost-effectiveness determines whether you can scale. Cost per finding reflects the real ledger far better than unit price.

    For developers, the takeaway is simple: do not blindly trust any single leaderboard. Build your own evaluation criteria and shadow evaluation set, and quantify the “recall–stability–cost” triangle before making a selection.

    For organizations, a more pragmatic direction is to embed LLM-based auditing into existing development workflows. It should complement static analysis, dependency governance, and human review, rather than being expected to solve every security problem in a single call.

    There is no silver bullet for security, but there can be a measurable engineering path.