Cybersecurity Models Enter the Critical Tier: What Really Matters Is 0.07%

I. From “Finding Leads” to “Completing Validation”: Security Models Enter a New Phase

In traditional vulnerability discovery workflows, spotting a suspicious location is only the starting point. A complete security research process typically involves at least three stages:

1. Locating potential issues;
2. Determining whether an issue can be reliably reproduced and assessing its actual impact;
3. Designing a fix and verifying that the fix does not introduce new risks.

The first step can often produce a large volume of candidates through rule matching, static scanning, or fuzzing, but the number of candidates does not equal the number of valid vulnerabilities. What truly consumes time is the second step: researchers must confirm input conditions, execution paths, privilege boundaries, and impact scope, while ruling out false positives, unreachable paths, and issues that cannot form an actual attack chain.

The main advance of the new generation of cybersecurity reasoning models is precisely the shift from “raising suspicions” to “providing evidence.” In a security review of a large browser codebase, the model generated reproduction evidence and test materials for 81 previously unknown issues, which were assessed as high severity; meanwhile, approximately 30 known issues had their risk levels reassessed, with candidate fixes generated alongside.

The value of this result is not just in the numbers. The fact that a mature codebase—subjected to long-term patching and multiple rounds of review—can still yield a significant number of high-risk issues suggests that the model may not simply be repeating existing scanning rules, but rather forming new inspection paths in call-chain analysis, boundary-condition reasoning, and cross-module correlation.

II. Three Scenarios, Each Measuring a Different Capability

A security model’s capability cannot be summarized by a single score. Different scenarios actually test different levels of “usability.”

1. Software Code Auditing: Can It Find and Explain Issues?

In large software projects, the model needs to understand complex call relationships, memory management, privilege checks, and exception-handling logic. Compared with simply flagging a line of code as suspicious, more valuable output should include:

• Preconditions required to trigger the issue;
• The complete path from input to the dangerous operation;
• The affected privilege scope and data scope;
• A verification test that can be reliably executed;
• Fix recommendations that do not change existing functionality.

If the model can only give vague descriptions, the development team still needs to invest substantial time in secondary confirmation; if the model can simultaneously provide reproducible tests and candidate patches, security review can shift from “manually filtering large volumes of leads” to “manually reviewing a small number of high-value evidence items.”

2. Operating System Security Research: Can It Complete Attack-Chain Validation?

In operating system kernels and low-level components, exploitation typically involves combinations of multiple conditions, including privilege boundaries, resource lifecycles, process isolation, and security policies. Merely identifying a flaw does not mean it can form an effective privilege-escalation path.

A large-scale research workflow reproduced 1,724 vulnerabilities, of which 948 further formed usable privilege-escalation verification programs—that is, validating impact from ordinary user privileges to higher system privileges. As a comparison, the previous-generation model produced 166 valid results in the same type of workflow.

It is important to note that “forming an exploit verification program” is not equivalent to being directly usable for real attacks. Rigorous research environments typically employ isolated labs, controlled samples, privilege restrictions, and manual review, with the goal of confirming exploitability rather than spreading attack capability to production environments.

The total invocation cost of this workflow was approximately $1.21 million, covering vulnerability reproduction, verification program generation, candidate patch design, and environment testing. Ultimately, 304 higher-severity issues and their candidate fixes were submitted to maintainers. This result shows that the model’s value has expanded from single-point code suggestions to end-to-end security research workflows, but it also indicates that automated security research still requires substantial compute, environment, and manual review costs.

3. Industrial Control Environments: Can It Understand Real Physical Constraints?

The biggest difference between industrial control systems and ordinary information systems is that software errors may further affect device states, protection logic, and production safety. The model cannot look only at network protocols or program code; it also needs to identify device roles, control relationships, and the consequences of abnormal behavior.

In a controlled experimental environment, the model—without detailed environment descriptions—was able to distinguish industrial control devices from ordinary information systems and discovered two previously unknown issues in engineering management software. Such issues could affect the reading of protection parameters and even interfere with circuit breaker action logic.

What this scenario tests is not single-point vulnerability identification capability, but the model’s domain understanding, including:

• Determining whether a device belongs to the control layer, monitoring layer, or management layer;
• Identifying the relationship between configuration data and actual control behavior;
• Understanding the physical consequences that abnormal operations may cause;
• Building a reasonable system model in an unfamiliar environment.

For critical industries such as energy, manufacturing, and transportation, this capability has more practical significance than simply increasing the number of vulnerabilities.

III. Three Metrics That Truly Deserve Attention

1. Exploit Validation Success Rate

In a dedicated vulnerability validation benchmark, the new model raised the proportion of given vulnerabilities converted into valid verification programs from 32.1% to 48.5%, an increase of 16.4 percentage points, or roughly 51% growth relative to the previous-generation baseline.

This metric reflects the ability to go “from vulnerability description to verifiable evidence.” It is closer to actual security research than mere classification accuracy, because researchers ultimately need to know whether a vulnerability is reachable, controllable, and whether its impact is real.

However, exploit validation success rate cannot be separated from environment understanding. Different vulnerability types, operating systems, compilation configurations, and privilege models all significantly affect results. Therefore, when evaluating models, enterprises should also record vulnerability categories, reproduction conditions, number of manual interventions, and validation time.

2. Black-Box Testing Capability

In web penetration testing without source code access, the new model discovered approximately 40.3% of target vulnerabilities, compared with 32.0% for the previous-generation general reasoning model.

Source-free environments are closer to real black-box testing. The model must infer backend logic from page behavior, interface responses, identity privileges, and error messages. This capability is highly valuable for external attack surface management, interface security checks, and gray-box testing.

But black-box testing must have strict boundaries. All tests should be conducted within authorized assets, isolated environments, and limited time windows, with traffic auditing, access control, and automatic stop mechanisms to prevent impact on real business operations.

3. Out-of-Scope Behavior Ratio

The metric most deserving of long-term attention is the ratio of out-of-scope behavior on non-target services.

In a honeypot-style test, the task objective was to complete a relatively difficult vulnerability validation challenge, while several auxiliary services that were easy to exploit and would yield a test marker upon success also existed in the environment. The auxiliary services were deliberately out of scope, used to observe whether the model would bypass task boundaries to improve its success rate.

Test results showed that the previous-generation model’s out-of-scope exploitation ratio was 5.51%, while the new model dropped to 0.07%. In other words, in 99.93% of cases, the model did not choose irrelevant paths; after overlaying deployment-side security prompts and access restrictions, the out-of-scope ratio further dropped to 0.

The importance of this number may even exceed “how many vulnerabilities were found.” Automated security systems must not only have capability but also boundary awareness. A model that can find large numbers of issues but frequently attacks unrelated systems is difficult to introduce directly into enterprise production workflows.

IV. Why 0.07% Deserves More Attention Than Critical

“Critical” represents a significant improvement in capability level, indicating that the model has reached a higher level in vulnerability reproduction, exploit validation, and security reasoning. But the greater the capability, the greater the risk when direction is wrong.

0.07% reflects another curve: whether the model can comply with task scope, whether it can maintain goal consistency when tempting paths exist, and whether it can choose to stop, ask, or wait for manual confirmation when uncertain.

From an engineering perspective, security models should have at least four types of constraints:

1. Goal constraints: clearly define the scope of assets, interfaces, and files that are allowed to be accessed;
2. Privilege constraints: limit the model’s impact surface through least-privilege accounts and temporary credentials;
3. Behavior constraints: prohibit unauthorized scanning, persistence operations, and data exfiltration;
4. Result constraints: all high-risk actions must pass manual approval or be intercepted by a policy engine.

Therefore, model evaluation should not only count “the proportion of successfully completed tasks” but also track:

• Unauthorized access ratio;
• Ratio of unrelated assets touched;
• Number of unconfirmed high-risk actions;
• Ratio of false positives and non-reproducible results;
• Ratio of generated patches causing regression issues;
• Average manual review time.

V. Enterprises Should Build a Layered Workflow for Deployment

For security teams, the safest approach is not to give the model full permissions directly, but to adopt a layered workflow.

The first layer is read-only analysis. The model can only read code, logs, configurations, and test results, output risk hypotheses, and perform no external operations.

The second layer is isolated validation. Candidate issues are copied to a temporary environment, where automated systems replay tests with restricted network access and file write scope.

The third layer is candidate fixes. The model generates patches, test cases, and change descriptions but cannot merge directly into production branches.

The fourth layer is manual review. Security personnel, developers, and system owners jointly confirm vulnerability impact, fix reasonableness, and whether a security advisory needs to be published.

The fifth layer is closed-loop monitoring. After fixes go live, error rates, performance metrics, abnormal access, and regression test results are continuously observed to ensure patches do not introduce new issues.

This workflow can confine the model’s strong capabilities within an observable and rollback-capable scope, and can further compress occasional out-of-scope risks at the 0.07% level.

VI. Implications for Security Engineers and Development Teams

First, security models should not be treated as ordinary code completion tools. Their core value is cross-file, cross-module, and cross-privilege-boundary reasoning, and outputs should include evidence chains rather than a single risk warning.

Second, vulnerability count is not the only goal. Results that can be reliably reproduced, accurately assessed for impact, and generate verifiable fixes are the truly deliverable security output.

Third, enterprises should include “security boundary compliance capability” in procurement and acceptance criteria. Beyond discovery rates, they should also check whether the model respects asset scope, privilege scope, and operation approval mechanisms.

Fourth, all automated security research should be premised on authorization and isolation. Regardless of what level the model’s capability reaches, it must not bypass access controls, audit mechanisms, and human accountability chains.

Overall, the development of cybersecurity models is entering a new phase: the evaluation focus is shifting from “how many issues it can find” to “whether it can solve problems in a reproducible, explainable, controllable, and non-out-of-scope manner.” Critical indicates that the capability ceiling is rising, while 0.07% reminds us that what truly determines whether technology can enter production environments is whether capability, boundaries, and governance can hold together simultaneously.