AI Models in Cybersecurity: Capabilities, Boundaries, and Governance

Expanding Capability Boundaries of AI Security Models

Traditional security research typically relies on manual code analysis, test input construction, and anomaly behavior localization, followed by gradual verification of whether a vulnerability can be stably reproduced. The introduction of AI models helps streamline code paths, generate testing strategies, analyze crash information, and organize reproduction steps in controlled environments. Its value lies not merely in “generating code faster” but in connecting various stages of vulnerability research, thereby reducing the cost of repetitive work.

Recent test results indicate that a new-generation cybersecurity model identified 81 previously undiscovered high-risk vulnerabilities in browser kernel research, submitting test evidence and reproduction plans. In operating system kernel research, the model reproduced 1,724 known vulnerabilities, with 948 corresponding functional exploits. For comparison, the previous-generation model under the same test conditions only achieved 166 successful exploits. In another remote vulnerability exploitation capability test, the new model scored 48.5%, compared to 32.1% for its predecessor.

These figures demonstrate a significant shift in security model capabilities, but they should not be interpreted as the model being able to independently conduct complete security research. Vulnerability discovery, reproduction, exploit code generation, and real-world attacks are fundamentally different tasks with progressively higher risks. When evaluating models, it is crucial to clearly define each metric’s definition, test environment, and success criteria.

From “Discovering Issues” to “Validating Risks”

Vulnerability research can generally be broken down into several stages:

1. Code and Behavior Analysis: Understanding logic such as input handling, state transitions, memory access, and permission checks to identify unexpected paths.
2. Problem Localization: Narrowing down suspicious code areas based on static analysis results, anomaly logs, or crash phenomena.
3. Controlled Reproduction: Constructing minimal inputs to confirm whether the issue consistently appears in an isolated environment.
4. Impact Assessment: Determining the effect on confidentiality, integrity, and availability, and evaluating trigger conditions and exploitation difficulty.
5. Fix and Regression Validation: Proposing remediation suggestions, checking if patches address the root cause, and ensuring no new defects are introduced.

Model reliability varies across these stages. It may excel at generating analysis leads or organizing reproduction steps, yet it can still misjudge anomalies, miss environmental dependencies, or produce unstable code. Therefore, model outputs should be treated as research materials awaiting verification, not as confirmed vulnerability conclusions. Professionals must still review the evidence chain, particularly to confirm whether the issue is reproducible, whether the impact is exaggerated, and whether the fix targets the root cause.

Test Scores Do Not Equate to Real-World Risks

Vulnerability exploitation capability tests typically evaluate whether a model can achieve specific objectives against designated flaws within a pre-configured environment. While such evaluations are useful for横向比较 model performance on specific tasks, the scores themselves do not fully represent actual offensive and defensive capabilities.

First, the vulnerability types, target configurations, and success conditions of the test set heavily influence results. If the test environment differs significantly from production systems, lab success rates may not directly transfer. Second, functional exploits are not necessarily stable and may not bypass additional protections in real-world environments. Third, testing focuses on “whether the task can be completed” and does not necessarily measure whether the model can accurately identify authorization boundaries, avoid destructive operations, or correctly handle sensitive data.

Therefore, enterprises and research institutions should focus on both task success rates and process quality—such as whether outputs are verifiable, whether prerequisites are accurately specified, whether operation scopes are proactively limited, and whether safe fallback strategies are employed upon failure. A single score is suitable for describing a specific capability but cannot replace a comprehensive security assessment.

Capability Enhancement Must Be Paired with Scope Control

The use of cybersecurity tools is typically constrained by authorization scopes. Testers may only be authorized to inspect designated assets, specific time frames, and limited technical paths; exceeding these boundaries, even for research purposes, can cause service disruptions, data exposure, or compliance risks.

Relevant test data shows that without additional safety prompts, the model did not attempt to access services outside the task scope in 99.93% of tests; with safety prompts, this rate reached 100%. Such results indicate that instructional constraints can improve a model’s adherence to scope boundaries, but they do not eliminate risks entirely. In actual deployment, prompt text cannot replace identity verification, access controls, and network isolation, as models may misunderstand context or be manipulated by untrusted inputs.

A more robust approach is to enforce authorization constraints at the system level:

• Limit Target Assets: Clearly define the hosts, applications, and data permitted for each task, and validate them during execution.
• Isolate Execution Environments: Restrict code execution, sample analysis, and test traffic to dedicated sandboxes or lab networks.
• Tiered Authorization: Establish different privilege levels for code reading, vulnerability validation, and exploitation testing, requiring manual approval for sensitive operations.
• Protect Credentials and Data: Avoid providing long-term credentials, production keys, or irrelevant personal data to the model; minimize necessary data processing.
• Maintain Audit Logs: Record task objectives, authorization bases, model suggestions, human confirmations, and actual execution results for traceability and post-mortem review.

Critical Infrastructure Requires Stricter Protections

Critical infrastructure, such as power grids, water utilities, transportation, and industrial control systems, often features long equipment life cycles and strict availability requirements. Security testing in these environments cannot adopt standard software testing procedures: a single anomalous request can alter device states, and some systems lack straightforward rollback mechanisms.

Therefore, high-risk operations involving critical infrastructure should adopt stricter access controls. Prioritize using simulation environments or digital testing environments disconnected from production networks. If live testing is necessary, it must be approved by the asset owner, clearly defining change windows, emergency contacts, and stop conditions, with recovery plans validated beforehand. Models should not be granted unrestricted operational access to production control equipment. Any action that could alter a device’s state must require human confirmation and independent safety safeguards.

Practical Recommendations for Development and Security Teams

For development teams, AI models can assist in code reviews, test case design, and defect explanation, but their outputs should not be directly applied as patches or security conclusions. Teams should establish a review process: supplement minimal reproduction materials for each issue identified by the model, verify that fixes cover the root cause, and validate behavioral changes through regression testing.

For security teams, it is recommended to introduce models starting with low-risk tasks—such as analyzing desensitized code, summarizing log clues, or organizing test reports—before gradually evaluating higher-privilege scenarios. Before expanding the task scope, re-evaluate data flows, network access, execution permissions, and audit requirements. When measuring deployment effectiveness, observe metrics such as false positive rates, reproduction success rates, manual review costs, and unauthorized access attempts, in addition to efficiency.

Conclusion

The advancement of AI security models is making vulnerability analysis and validation more efficient, simultaneously altering the threshold and speed of security assessments. However, greater capability necessitates clearer authorization boundaries, reliable isolation mechanisms, and traceable human reviews. For organizations, the reasonable goal is not to replace security personnel with models, but to integrate them into controlled research processes: letting automation handle repetitive, verifiable tasks while professionals remain responsible for risk assessment, permission approval, and final decision-making.