LLM Vulnerability Discovery on Real Codebases: Test Methods, Findings, and Takeaways

    A European cybersecurity vendor recently published a large-scale evaluation of LLMs’ ability to find real vulnerabilities in real codebases, using about 11.7 billion tokens, 32 recent CVEs, and an offline multi-turn agent protocol. This article breaks down the test design, key results—including an open-source model’s pooled-recall lead, consistency gaps, and a low-cost Chinese open-source model’s near-parity—and practical recommendations for security teams building LLM-assisted vulnerability discovery workflows.

    Why This Report Is Worth Attention

    A European cybersecurity vendor recently published a real-world evaluation of LLM security capabilities. The team consumed approximately 11.7 billion tokens, had multiple mainstream LLMs search for real vulnerabilities in real codebases, and released the full methodology and ranking results. The report sparked discussion in developer communities for three main reasons:

    1. The tests used real vulnerabilities, not synthetic tasks. Many model evaluation leaderboards rely on public datasets, and models may have “seen” the answers during pre-training, inflating scores. This test selected 32 CVEs from recently disclosed real vulnerabilities and paired them with real codebases, so models could not pass by memorizing answers.
    2. The test process was heavy enough. Ten models, three runs per case, 96 total runs, and up to 30 autonomous exploration turns per run—this scale is uncommon in security-focused evaluations.
    3. The conclusions include counterintuitive findings: the top-ranked model was not the “strongest closed-source model” in the traditional sense, but a high-end version from an open-source family. Meanwhile, another Chinese open-source model has largely caught up with leading closed-source models, at significantly lower inference cost.

    Below, we analyze the test methodology, core findings, and implications for engineering teams.

    Test Methodology: How to Scientifically Evaluate a Security Model

    2.1 Dataset Construction: Real CVEs + Real Codebases

    The evaluation team selected 32 CVEs from recently disclosed real vulnerabilities, covering multiple vulnerability types such as injection, authentication bypass, and deserialization, and placed the vulnerabilities back into their real open-source code contexts. The benefits of this approach are:

    • Avoiding data contamination: recently disclosed vulnerabilities are unlikely to be in the model’s training corpus as “standard answers,” so the test measures reasoning ability rather than memory.
    • Closer to real-world practice: security engineers routinely face tasks such as locating suspicious code paths in large codebases, not isolated CTF-style puzzles.

    2.2 Run Protocol: Multi-Turn Agent-Style Exploration

    Each model ran each of the 32 cases three times, for 96 runs in total. In each run, the model worked as an agent:

    • Up to 30 autonomous exploration turns, with the ability to browse the codebase, search files, and read dependency declarations.
    • Offline operation: the model could not access the internet, eliminating “look it up and report it” cheating.
    • The model ultimately output the location and cause of the vulnerability it believed existed, and the evaluator compared this with the real CVE to determine whether it was a hit.

    This is essentially an evaluation of the model’s “security agent capability”: the model must not only understand vulnerability patterns but also plan, verify, and backtrack over long-horizon tasks.

    2.3 Scoring Dimensions: Recall + Consistency

    The report did not rely on a single score. Instead, it split evaluation into two dimensions:

    • Pooled recall: the union of three runs, measuring the model’s upper bound.
    • Stable hits: the number of cases hit in all three runs, measuring the lower bound and consistency.

    This design is critical. As shown below, these two dimensions reveal large stability differences among top-ranked models.

    Core Findings

    3.1 An Open-Source Model Takes First Place in Recall, but Consistency Is Concerning

    The model with the highest pooled recall found 28 of the 32 vulnerabilities across three runs, exceeding several leading closed-source models, which clustered at 25–26. But a closer look reveals two issues:

    • Mediocre single-run performance: the model found only 17 vulnerabilities on its first run, indicating high variance in single-run output.
    • Only 10 stable hits: few cases were hit in all three runs, and a “lucky run” contributed a large share of the pooled score.

    The implication for engineering teams is: if such models are used for automated auditing in production, do not rely on a single benchmark run. Design repeated runs and voting mechanisms to hedge variance.

    3.2 A Chinese Open-Source Model Reaches Near-Parity at Very Low Cost

    The official release of another open-source model achieved 25/32 recall, only one fewer than the best-performing closed-source model, while inference cost was about 70% lower. This supports an industry trend: on vertical reasoning tasks, the gap between the open-source camp and the closed-source frontier is narrowing quickly. For security teams that need to scan codebases at scale, a large cost gap translates into very different deployment feasibility.

    3.3 Coverage Limitations

    This test did not include some newer flagship models known for security capabilities, so the “open-source first” conclusion should be understood within the range of tested models and should not be overextrapolated. At the same time, 32 samples is small for statistical significance, and single-ranking differences—such as 25 versus 26—should not be interpreted as a generational capability gap.

    Practical Recommendations for Security Teams

    Based on the report’s methodology and findings, teams planning to introduce LLMs for vulnerability discovery should consider the following:

    1. Build an internal evaluation set before choosing a model. Extract 20–30 cases from historical vulnerabilities in your own codebase and run a small-scale evaluation using a protocol of “real context + offline operation + multi-turn exploration.” This is more reliable than any public leaderboard.
    2. Use repeated runs and unions to improve recall. Following the pooled-recall approach, run the same codebase three times and merge the results. This can significantly reduce missed findings, at the cost of linearly increasing cost—which is also where low-cost open-source models add value.
    3. Watch for false positives. LLMs tend to report suspicious points with a “better safe than sorry” bias. Be sure to add human review or a second validation layer based on rules or static analysis in the pipeline. A typical orchestration logic is as follows:
    1. results = []
    2. for run in range(3): # Multiple runs to hedge variance
    3. findings = agent_scan(repo, seed=run)
    4. results.append(set(findings))
    5. pooled = set().union(*results) # Union: improve recall
    6. stable = set.intersection(*results) # Intersection: high confidence
    7. # Send stable directly to human review; pooled minus stable enters the low-priority queue
    1. Treat the model as a “junior security engineer,” not a “final judge.” Current consistency weaknesses make models better suited to triage and lead generation. Final severity rating and remediation decisions should remain human-led.
    2. Pay attention to cost structure. Security scanning is a typical batch task, and token consumption scales with codebase size. When capabilities are similar, choosing a model with lower unit cost—for example, general-purpose LLM services hosted on mainstream cloud platforms—can move a solution from pilot to full coverage.

    Conclusion

    The greatest value of this 11.7-billion-token test is not that a particular model topped the leaderboard, but that it demonstrates a credible evaluation paradigm for security capability: real vulnerabilities, real codebases, offline operation, multi-turn exploration, and dual metrics for recall and consistency. It also reveals two trends: open-source models are approaching the closed-source frontier on vertical tasks, and single-run capability and stability are still two different things. For developers, rather than chasing leaderboard rankings, it is better to borrow the methodology and build a reproducible internal benchmark on their own codebases. That is the right starting point for putting AI-assisted security into practice.