Measuring LLM Security Performance: How to Evaluate Vulnerability Recall and Consistency

    A large-scale evaluation of 10 leading language models used 11.7 billion tokens to test whether they could identify real vulnerabilities in real-world codebases. The results show why recall alone is not enough: consistency, exploration limits, inference cost, and human oversight are equally important when selecting an AI model for security workflows.

    1. Background: Evaluating LLMs Against Real Vulnerabilities

    A European cybersecurity company recently conducted a large-scale experiment that used approximately 11.7 billion tokens to evaluate 10 mainstream large language models on cybersecurity tasks. The central question was highly practical: Can LLMs identify real security vulnerabilities in real-world codebases?

    The company was not an academic institution. It provides application security posture management (ASPM) and cloud security posture management (CSPM) capabilities for developers. Its platform integrates multiple security tools, including static application security testing (SAST), software composition analysis (SCA), cloud infrastructure checks, container security, and secret detection.

    As a security practitioner, the company had a clear reason for testing LLMs: to determine whether these models could replace or enhance critical parts of traditional security scanning workflows. This gives the results considerable engineering value beyond a simple model leaderboard.

    2. Evaluation Methodology: A Practical Benchmark Design

    LLM evaluations in cybersecurity can easily produce misleading results. Training data for many leading models contains extensive public information about known vulnerabilities, meaning a model may simply be recalling an answer it has already seen.

    This evaluation included several design choices worth adopting.

    1. Using Recent CVEs to Reduce Data Contamination

    The test selected 32 CVEs—publicly disclosed vulnerabilities—from recently reported issues. The vulnerabilities were found in real codebases. Because they had been disclosed relatively recently, the corresponding patches and analyses were unlikely to have been included in the models’ training data.

    This helps reduce the risk of testing memorization rather than actual vulnerability discovery. The lesson is important for any team designing an LLM benchmark: the freshness of the evaluation set has a direct impact on the credibility of the results.

    2. Running Each Case Multiple Times to Measure Stability

    Each model was tested against all 32 cases three times, for a total of 96 runs. Each run allowed up to 30 exploration rounds, and the models could not access the internet.

    This setup produced two key metrics:

    • Recall: How many real vulnerabilities the model discovered.
    • Consistency: Whether the model could reliably identify the same vulnerability across multiple runs.

    3. Limiting Exploration Resources

    Each run was limited to 30 exploration rounds and had no internet access. These constraints were intended to approximate real security operations, where an agent works with limited context, tool calls, and inference budgets rather than unlimited brute-force exploration.

    The evaluation loop can be represented with the following pseudocode:

    1. for model in candidate_models:
    2. for cve in recent_cves: # 32 recent real vulnerabilities
    3. for run in range(3): # Run each case three times
    4. result = agent_explore(
    5. repo=cve.related_repo,
    6. max_rounds=30,
    7. internet_access=False
    8. )
    9. record(model, cve, run, result)
    10. # Aggregate recall and consistency

    3. Key Finding: Recall and Consistency Are Different Metrics

    The evaluation revealed several important findings that are easy to overlook.

    1. An Open-Source Frontier Model Achieved the Highest Recall but Lower Consistency

    One open-source model achieved the highest aggregate recall, identifying 28 of the 32 vulnerabilities across three runs. This exceeded most leading closed-source models, which generally identified approximately 25 or 26 vulnerabilities.

    However, the detailed results told a different story:

    • In its first individual run, the model found only 17 vulnerabilities.
    • It consistently identified just 10 vulnerabilities across all three runs.

    This means much of its lead came from aggregating three runs. Its performance in any single run was relatively volatile.

    That distinction matters in production. If high recall comes at the expense of stability, deployment may require repeated sampling, majority voting, or similar compensation mechanisms. These approaches directly increase inference costs.

    2. A Newer Contender Narrowed the Gap at a Much Lower Cost

    Another model also delivered notable results. It achieved a recall of 25 out of 32 vulnerabilities—just one fewer than the best-performing closed-source model—while costing nearly 70% less to run.

    This reflects a broader trend in the LLM market: the capability gap between open-source models and leading closed-source models is narrowing quickly, while open-source options often provide better cost efficiency.

    For security teams with limited budgets, this suggests that a sufficiently capable open-source model combined with appropriate engineering safeguards may be more economical than automatically choosing the most powerful closed-source model.

    3. Limitations of the Evaluation Scope

    The test did not include every top-tier model, and the evaluation set contained only 32 CVEs. Its conclusions therefore have clear boundaries.

    Any benchmark is a snapshot of model performance under a particular task definition and set of constraints. The results should not be interpreted as a definitive ranking of overall model capabilities.

    4. Practical Lessons for Enterprise Adoption

    The evaluation suggests a reusable methodology for selecting models for security workflows.

    1. Define a complete metric framework.
    In security scenarios, recall—minimizing missed vulnerabilities—is often more important than precision, which focuses on reducing false positives. However, the two metrics must be balanced against the business impact of each type of error. Teams should also track consistency, average exploration rounds, token consumption, and other runtime metrics.

    2. Keep the evaluation set fresh.
    Use recently disclosed vulnerabilities and, where possible, private code samples to reduce contamination from public benchmarks and training data.

    3. Use repeated sampling to offset model volatility.
    For models with inconsistent results, an aggregation strategy could look like this:

    1. votes = [run(cve) for run in range(k)] # Run k independent trials
    2. confirmed = sum(votes) >= threshold # Confirm only after reaching the threshold

    4. Measure effective recall per unit cost.
    Teams can establish cost-efficiency metrics such as “vulnerabilities discovered per dollar.” They can then conduct small-scale pilot tests using hosted LLM services and compute support from providers such as Baidu AI Cloud before committing to broader deployment.

    5. Keep humans in the loop.
    Even the best-performing model failed to identify every real vulnerability. Model output should therefore be treated as input for security engineers and a source of investigation leads—not as the final verdict.

    5. Conclusion

    The most valuable aspect of this evaluation, which consumed 11.7 billion tokens, was not determining which model ranked first. Its real value was demonstrating a rigorous evaluation framework based on real tasks, fresh samples, repeated runs, and cost analysis.

    For any organization investing in LLM-based security capabilities, the methodology is more useful than the leaderboard. Build a reliable evaluation system first, and choose a model second.