LLM Security Benchmarking: Recall, Stability, and Cost Trade-offs

    This article breaks down how to interpret LLM security evaluations by separating pooled recall, single-run recall, and stable hits, and by calculating cost per successful detection. Using illustrative figures from a public 11.7-billion-token evaluation summary, it explains why leaderboard rankings can overstate practical capability, why consistency matters for CI workflows, and how teams can build their own evaluation sets and deployment pipelines.

    1. Introduction: What an 11.7-billion-token evaluation reveals

    Source and scope: This article is a methodological interpretation based on an evaluation summary discussed in public communities. The original report name, publisher, and date are not clearly provided. Figures such as 11.7 billion tokens, 28/32, 17/32, 10/32, 25–26/32, and 69.8% are illustrative citations and should not be used directly for procurement or selection decisions. Model rankings and cost comparisons are also illustrative; before formal publication or external citation, they should be replaced with verifiable sources.

    Recently, a large-scale cybersecurity model evaluation that consumed about 11.7 billion tokens attracted attention. It asked multiple large language models to identify the same set of known vulnerable code samples over multiple rounds and produced two seemingly counterintuitive conclusions.

    Conclusion 1: Rankings and cost comparisons are illustrative and must be replaced with verifiable sources before formal publication. “Best score” is not the same as “average capability.” A frontier model ranked first in pooled recall, but a single run detected only about half of the samples, and the proportion of vulnerabilities found in all three rounds was also not high.

    Conclusion 2: Rankings and cost comparisons are illustrative and must be replaced with verifiable sources before formal publication. “One sample behind” is not the same as “one tier behind.” An open-source domestic model was close to a frontier model in the same tier, differing by only about one sample in pooled recall, while its cost was significantly lower. If converted to cost per successful detection, the cost-effectiveness gap could widen further, but only when recall scope and number of runs are unified.

    For teams planning to introduce AI-assisted code security auditing into the development process, the value of this kind of evaluation is not just the leaderboard. It highlights three questions:

    • Which capability is being measured? Best-once performance, average-once performance, and stable performance are three different capabilities.
    • Is the evaluation method reliable? Sample design, number of runs, and context construction can all affect results.
    • How should cost be calculated? Looking only at token price can be misleading. It is better to examine cost per successful detection, and comparisons must not mix different evaluation scopes.

    The rest of this article breaks down these issues.

    2. Three recall metrics: pooled, single-run, and stable

    For discussion, the article uses the following labels:

    • Frontier Model A: the model ranked first in pooled recall.
    • Frontier Model B: the frontier model immediately behind Frontier Model A and in the same tier as the open-source domestic model.
    • Open-source domestic model: an officially released model that is open-source and can be privately deployed.
    Metric Definition Question answered Frontier Model A Frontier Model B Open-source domestic model
    Pooled recall Union of multiple runs; a vulnerability is counted if hit in at least one run What is the model’s “capability ceiling”? 28/32, ranked first About 25–26/32 25/32
    Single-run recall Number of vulnerabilities hit in a single run What can one actual run find? 17/32 in the first run Not separately listed Not separately listed
    Stable hit Number of vulnerabilities hit in every run What is the model’s “lower bound”? 10/32 Not separately listed Not separately listed

    Table note: The values in this table are illustrative citations used only to explain metric definitions. They are not confirmed benchmark results or selection criteria.

    Frontier Model B’s pooled recall was about 25–26/32, in the same range as the open-source domestic model’s 25/32. The open-source domestic model was about one sample behind Frontier Model B, while its cost was significantly lower than Frontier Model A; the illustrative cost comparison later in the article treats the open-source model’s three-round pooled cost as about 69.8% lower than Frontier Model A’s.

    Putting these numbers together yields three observations.

    First, the value of being first in pooled recall is limited. Frontier Model A did lead on best-case performance, but its single-run recall was 17/32 and its stable hit count was 10/32. Its higher pooled recall came largely from complementarity across multiple runs: vulnerabilities missed in one round were often caught in another.

    Second, the open-source domestic model is already in a usable range. Being one sample behind in a 32-sample evaluation is closer to statistical noise than a stable capability gap. In addition, open-source private deployment brings more than API price considerations: it also creates room for data compliance, controlled deployment, and cost management. However, “usable” does not mean it can directly replace the strongest model; secondary validation against business-specific samples is still needed.

    Third, consistency is often underestimated. If a model’s vulnerability discovery is unstable across runs, it can cause alert noise, missed detections, and engineering rework when inserted into an automated development workflow.

    3. Methodology breakdown: how this kind of evaluation is performed

    3.1 Dataset design behind 32 samples

    This type of evaluation usually does not select vulnerable code arbitrarily. It typically collects confirmed historical vulnerabilities from real open-source projects and reviews them manually. The samples generally cover multiple common categories:

    • Injection issues, such as command injection and SQL injection.
    • Authentication and authorization defects.
    • Incorrect cryptography usage, such as weak algorithms, hardcoded keys, and misuse of random numbers.
    • Sensitive information leakage.
    • Deserialization issues and insecure dependencies.

    The key question is not merely “can the model find something?” but “can it locate the problem accurately?” Each sample usually has a trusted root cause, such as a specific file, function, or code pattern. Evaluation should consider at least two points: whether the location reported by the model matches the root cause, and whether the report ultimately points to a real, fixable defect.

    Simply saying “this code is unsafe” without providing location, call-chain context, and remediation clues offers limited value to developers.

    Also, 32 samples are enough for a horizontal comparison, but not enough to support overly fine-grained rankings. A small sample size widens confidence intervals, and small differences can be misinterpreted.

    3.2 Why run multiple rounds

    Large language models are inherently uncertain when sampling temperature is nonzero. Even at temperature 0, engineering factors such as batching schedules, floating-point operation order, context truncation methods, and differences in reasoning or “thinking” modes can still produce subtle variations.

    Therefore, a single result only tells you how the model performed in that one run. Multiple independent runs are essentially estimating the model’s hit probability for a given vulnerability. When the sample set is fixed, adding more runs is often more economical than adding more samples.

    A practical run strategy is to start with 3 rounds to identify stable hits, then expand to 5 or 7 rounds depending on budget, risk tolerance, and observed model volatility. More rounds are not always better; the number of rounds should be enough to support statistical judgment.

    The 11.7-billion-token consumption also shows that this kind of evaluation is not cheap. With 32 samples and multiple runs per model, each detection requires the model to read large amounts of code context. Depending on the number of evaluated models, a single invocation may involve token consumption at the million-token scale. Token consumption mainly comes from context construction: reading entire files, retrieving related files, or only extracting function snippets can directly affect detection rates.

    3.3 Four metrics and four different conclusions

    Metric Statistical scope Question answered Common misreading
    Pooled recall Multi-round union: hit in at least one round What could theoretically be found? Treated as single-run capability
    Stable hit Multi-round intersection: hit in every round Which vulnerabilities can be reproduced reliably? Ignored and treated as ordinary recall
    Single-run recall Number of hits in each individual round What does one actual run show? Reporting only the best or worst round
    False positives Number of normal code samples incorrectly flagged as vulnerabilities How credible are the reports? Discussing recall only, without false positives

    Another commonly ignored point is that model output is usually a lead stating “a risk may exist here,” not a confirmed vulnerability. The engineering question is whether the output helps people find and fix problems faster and more accurately. Therefore, recall, false-positive rate, and remediation cost should be evaluated together.

    4. Why pooled recall gets inflated

    After taking the union of multiple runs, recall can only stay the same or increase; it cannot decrease. This is the fundamental reason pooled recall can be naturally higher and disconnected from single-run performance.

    4.1 Estimating with a simple probability model

    Assume the model’s average single-run hit probability across all 32 samples is p. For three independent runs:

    • Expected single-run recall: p
    • Pooled recall: 1 − (1 − p)³
    • Stable hit ratio: p³

    If Frontier Model A’s single-run recall is about 17/32, then p ≈ 0.53.

    Scenario Theoretical value under independence Observed value
    Single-run recall About 17/32 17/32
    Pooled recall 1 − (15/32)³ ≈ 0.897, about 28.7/32 28/32
    Stable hits (17/32)³ ≈ 0.150, about 4.8/32 10/32

    The observed pooled recall is close to the independence assumption, suggesting that much of the “first-place pooled recall” comes from multi-round complementarity. However, the observed stable hit count is clearly higher than the theoretical value: about 10/32 versus about 4.8/32 under independence. A caveat is needed: if the sample size is sufficient and the single-run hit rate is stable, this gap may reflect positive correlation across runs, but it may also include sampling variation. It cannot directly be interpreted as strong positive correlation.

    4.2 Why positive correlation may occur

    For the same vulnerability on the same model, results often look like “almost always found” or “almost never found”:

    • If the vulnerability pattern closely matches common patterns in the model’s training data, the model may hit it every round.
    • If the vulnerability is buried in complex business logic, cross-file data flows, specific framework usage, or requires multi-step reasoning to locate the root cause, the model may occasionally hit it but struggle to reproduce the hit consistently.

    Therefore, a single-run recall of 0.53 should not be understood as “each vulnerability has a 53% chance of being found.” A more realistic interpretation is that roughly half of the samples are relatively easy to detect, while the other half form a stable blind spot.

    4.3 What happens when difficulty is uneven

    Assume 16 of the 32 vulnerabilities are “easy,” with a per-round hit probability of 0.9, and the other 16 are “hard,” with a per-round hit probability of 0.1. The expected values are:

    • Expected single-run recall: 16 × 0.9 + 16 × 0.1 = 16, or 0.5.
    • Expected pooled recall: 16 × (1 − 0.1³) + 16 × (1 − 0.9³) ≈ 20.3/32.
    • Expected stable hits: 16 × 0.9³ + 16 × 0.1³ ≈ 11.7/32.

    These numbers show that when difficulty is uneven, pooled recall can be higher than single-run recall while still leaving some stable hits. However, they do not necessarily reproduce the observed 28/32 and 10/32; the exact values depend on difficulty distribution, model capability, and cross-round correlation.

    The engineering conclusion remains the same: pooled recall reflects a capability ceiling, while production deployment depends more on stable capability.

    4.4 Three implications for engineering deployment

    1. Use majority voting to improve reliability. A report can be included only when it appears in at least M out of N runs. Larger N can reduce false positives, but more runs also affect cost and recall.
    2. Use incremental analysis to control cost. When scanning the same repository repeatedly, cache historical stable hits or perform incremental comparisons, and send only new code to the model.
    3. Use two-stage validation to improve credibility. First let the model quickly screen candidate issues, then verify them with static analysis rules or another model, and finally generate alerts.

    5. Cost and cost-effectiveness: three calculation scopes

    5.1 Single-run cost

    1. Single-run cost = input tokens × input price + output tokens × output price + fixed overhead

    Input tokens mainly come from context construction. Output tokens depend on whether deep reasoning mode is enabled and how detailed the report is. Fixed overhead includes rule engines, result deduplication, caching, and logging/audit costs.

    5.2 Cost-effectiveness: cost per successful detection

    1. Cost per successful detection = total invocation cost / effective detections

    An important prerequisite is that the numerator must match the denominator. Single-run cost can only be divided by the average number of hits from a single run. The cost of three pooled runs must be divided by the number of pooled hits across those three runs. The same formula applies only when the recall scope, number of runs, and context construction method are the same. Direct comparison between pooled recall and single-run cost is not allowed.

    To illustrate the scale, assume Frontier Model A has a relative single-run cost of 1.00 and a relative three-round total cost of 3.00. Its single-run hit count is 17/32, and its three-round pooled hit count is 28/32.

    Model / scope Hit scope Relative total invocation cost Cost per successful detection (relative)
    Frontier Model A (single run) 17/32 1.00 1 ÷ 17 ≈ 0.059
    Frontier Model A (three-round pooled) 28/32 3.00 3 ÷ 28 ≈ 0.107
    Open-source domestic model (three-round pooled) 25/32 3 × 0.302 = 0.906 0.906 ÷ 25 ≈ 0.036

    Note: In the table above, the open-source domestic model’s 25/32 is treated as pooled recall under the original illustrative scope. Because the material does not provide its average single-run hit count, it is not appropriate to estimate a single-run scope using “0.302 ÷ 25.” If average single-run hit data is later provided, the calculation should be redone as “single-run cost ÷ average single-run hits.”

    For architecture decisions, these numbers imply that when single-call cost differs by several times but recall differs by only one or two samples, it is often better to invest the budget in the engineering layer first—such as expanding context coverage, introducing retrieval augmentation, optimizing prompts, and adding cross-validation. These measures usually provide higher marginal benefit than simply switching to a more expensive model.

    6. Why conclusions from 32 samples should not be overinterpreted

    A sample size of 32 is small. For example, with 17/32 hits in a single run, a Wilson interval estimate gives a 95% confidence interval of roughly 36% to 69% for the true recall. Frontier Model B and the open-source domestic model differ by only about one sample, so their confidence intervals would overlap substantially. Statistically, this difference cannot be clearly distinguished from “no difference.”

    Therefore, this kind of evaluation is suitable for answering “which models deserve to be on the candidate list,” but not for answering “who is better by a few percentage points.”

    Model selection should include three actions:

    1. Expand the sample size. Use the team’s own codebase for secondary validation.
    2. Control variables. Fix the model version, context construction method, prompts, and sampling parameters.
    3. Evaluate by language and scenario. Vulnerability patterns vary significantly across languages, so an overall recall score cannot replace scenario-specific evaluation.

    If candidate models have overlapping confidence intervals in public evaluations, they should enter small-traffic validation rather than being eliminated immediately or rolled out fully.

    7. Six-step checklist for building an enterprise LLM code-security evaluation set

    Step Key action Deliverable Acceptance criteria
    1. Define the scenario Decide whether the use case is fast pre-commit checking, periodic CI scanning, or nightly full auditing Scenario description Clarify metric priorities: fast checks focus on false positives; periodic scans focus on recall and coverage
    2. Construct samples Collect historical vulnerability fix records, public vulnerability samples, and normal code snippets Evaluation sample library Samples are manually reviewed for root cause, exploitability, and code location
    3. Set annotation rules Annotate vulnerability type, file, function, severity, fix method, and exploitation prerequisites Annotation specification Inter-annotator agreement is acceptable, and disputed samples can be traced back
    4. Control contamination Rename variables, restructure code, and add recent real defects Sample update mechanism Evaluation samples are isolated from training data, and samples are refreshed periodically
    5. Design the workflow Use multi-round runs, majority voting, and cross-model validation Evaluation workflow Define false-positive ceiling, single-run cost ceiling, acceptable recall floor, and version rollback support
    6. Establish a baseline Record recall, false-positive rate, and single-run cost under the same samples and workflow Baseline table Supports horizontal comparison of candidate models and regression testing across versions

    8. Engineering integration: embedding model-based security capabilities into development workflows

    8.1 Layered design: use different costs at different stages

    • Local pre-check. Before committing code, developers use lightweight rules or a low-cost model to perform quick checks, focusing on pattern-based issues such as injection and hardcoded secrets.
    • CI stage. Perform incremental scans on merge requests, scanning only changed code. Use a model with moderate cost, combine caching and deduplication, and control invocation volume per commit.
    • Periodic full scans. Nightly or scheduled jobs analyze core repositories comprehensively. Use a stronger model or multiple sampling rounds when recall is the priority.

    8.2 Controlling false positives: turn the model from a reporter into a reasoner

    The model usually reports “a potential issue may exist.” An additional verification layer is needed:

    • Cross-validation. Use another model or static analysis rules to review candidate issues. Only confirmed issues should become formal alerts.
    • Attach an evidence chain. Require the model to output the file, line number, call chain, and trigger conditions so developers can quickly judge whether the finding is valid.
    • Closed-loop feedback. After developers mark a finding as a false positive, the sample can be returned to the evaluation set to drive prompt and rule iteration. Sample return requires desensitization and authorization to avoid secondary leakage of sensitive code or personal information.

    8.3 Data and compliance

    When model invocation is hosted in the cloud, considerations include code asset egress, invocation auditing, and cost control. The following generic capability categories can be considered:

    • Access multiple models through a unified model service gateway and route by scenario.
    • Send sensitive code through private networks or dedicated deployments.
    • Connect invocation logs to a logging service, and connect cost metrics to monitoring and alerting.
    • Store evaluation sets and result data in object storage, with permissions isolated by project.

    The cloud capabilities mentioned here are only used to illustrate generic capability categories. They do not constitute product endorsement, ranking conclusions, or competitive comparisons. Specific selection and configuration should follow official documentation, compliance requirements, and actual testing.

    8.4 Organization and process

    • Tiered response. High-severity vulnerabilities can automatically block merges or create remediation tasks. Low-severity issues can be summarized into reports and handled according to team schedules.
    • Metrics loop. Continuously track mean time to remediation, vulnerability reopen rate, and model false-positive rate to evaluate the actual value of AI-assisted auditing.
    • Human fallback. Model capability is limited. Critical systems still require manual audits and security experts; a single model cannot be expected to solve all security problems.

    9. FAQ: How to select and calculate model code-security evaluations

    What is pooled recall?
    Pooled recall takes the union of multiple evaluation runs. If a vulnerability is hit in any round, it counts as a pooled hit. It is closer to a capability ceiling and does not represent stable single-run performance.

    What is a stable hit?
    A stable hit means the same vulnerability is hit in every run. It is closer to reliable production capability and is suitable for automated workflows.

    How is cost per successful detection calculated?
    It is calculated by dividing total invocation cost by the number of effective detections. If a model has low recall, even a low unit price may lead to a higher cost per successful detection. The key is not to mix pooled recall with single-run cost.

    What are the most common mistakes in model selection?
    Common mistakes include looking only at leaderboard scores, confusing single-run recall with pooled recall, ignoring false-positive rates, ignoring context construction methods, making overly fine-grained rankings from 32 samples, and reusing evaluation samples after they have entered the model’s training data.

    How can evaluation contamination be prevented?
    Once evaluation samples enter the model’s training corpus, results can become distorted. Samples should be refreshed periodically, for example through variable renaming and code restructuring, while continuously adding recent real defects observed by the team.

    How should a model be selected?
    A practical order is to first look at the scenario, then the recall scope, and then cost. For high-frequency CI scenarios, prioritize single-run recall, false-positive rate, and single-run cost. For low-frequency full-audit scenarios, it may be acceptable to pay a higher cost for higher pooled recall. Final decisions should be confirmed through small-traffic validation.

    10. Conclusion: from leaderboard thinking to scenario thinking

    What did the 11.7-billion-token evaluation actually prove?

    It proved that frontier models can indeed lead on best-case performance, but the lead is not enough to constitute a statistically significant advantage at a sample size of 32. It also proved that open-source self-deployment has practical value, with clear advantages in cost, compliance, and controllability. More importantly, it reminds us that an LLM’s security capability is not a single scalar value, but a triangle of “recall, consistency, and cost.”

    For enterprises, the right approach is not to chase leaderboards, but to build their own evaluation sets and baselines, define scenarios clearly, control variables, and iterate continuously. The endpoint of model selection is not “choose the strongest model,” but “find the model that best fits the current scenario, cost, and risk constraints.” When code security auditing becomes a standard part of the development workflow rather than an optional add-on, enterprises can quantify the benefits as lower false-positive rates, shorter remediation times, and controlled per-call costs—and let AI security capabilities genuinely support stable delivery.