The Boundaries of AI’s Ability to Autonomously Discover and Exploit Vulnerabilities

In recent years, large language models and AI agents designed for cybersecurity tasks have advanced rapidly. Some systems can read large codebases, locate suspicious defects, generate trigger inputs, and reproduce vulnerabilities or validate exploitability in isolated environments. This has led to a compelling question: Can AI now autonomously discover and exploit vulnerabilities?

The technical answer cannot be reduced to a simple “yes” or “no.” When the target is clearly defined, source code is available, the runtime environment is fixed, and repeated trial and error is allowed, advanced models can complete highly complex security tasks. However, that capability is fundamentally different from automatically selecting targets on the open internet, identifying exposed assets, discovering unknown vulnerabilities, bypassing multiple layers of protection, and reliably obtaining privileges.

To assess this progress accurately, “autonomous vulnerability discovery and exploitation” must be divided into several stages. Each stage should then be evaluated based on its input conditions, success criteria, generalization ability, and real-world cost.

1. What Does “Autonomous Vulnerability Discovery and Exploitation” Actually Include?

A complete vulnerability research process usually contains at least the following stages:

1. Asset and attack-surface identification
   Determine which services, interfaces, protocols, components, and privilege boundaries exist within the target, and decide which areas deserve deeper analysis.

2. Vulnerability discovery
   Use source-code auditing, program analysis, fuzzing, runtime monitoring, or patch-diff analysis to identify code paths that may cause out-of-bounds access, use-after-free conditions, integer overflows, missing authorization checks, or similar issues.

3. Vulnerability reproduction
   Convert a suspicious location identified through static analysis into a stable trigger case, confirming that the issue can actually occur under a specific version and configuration.

4. Exploitability analysis
   Determine whether the vulnerability can only cause a crash or whether it can also lead to information disclosure, privilege escalation, arbitrary code execution, or a breach of a security boundary.

5. Exploit construction
   Develop an input or execution chain capable of achieving the intended security impact in the presence of defenses such as address space layout randomization, non-executable memory, sandboxing, and control-flow protection.

6. Environment migration and stability validation
   Test whether the method remains effective across different compiler options, dependency versions, memory layouts, hardware architectures, and deployment configurations.

7. Action decisions and risk control
   Decide whether testing should continue, when it should stop, how evidence should be preserved, and how to avoid affecting unrelated systems or real business operations.

Many evaluations described as “AI autonomous exploitation” cover only part of this process. The test may already provide vulnerable source code, build instructions, a runnable environment, or even an input that triggers an abnormal condition. In such cases, the model is not discovering a vulnerability from scratch. It is extending a known trigger or highly suspicious condition into a program that demonstrates a concrete security impact.

That capability remains important, but a more precise description would be:

AI can, under controlled conditions, convert a known or highly suspicious vulnerability into a verifiable exploitation result.

This is substantially different from autonomously discovering a zero-day in an unknown system and carrying out a reliable attack.

2. Why High Evaluation Success Rates Cannot Be Directly Generalized

When a security evaluation reports that a model successfully solves nearly half of its tasks, the first questions should concern the denominator and the experimental conditions—not the percentage itself.

2.1 What Prior Information Was Available?

The difficulty of an evaluation depends heavily on its inputs. Each of the following can significantly reduce task complexity:

• Whether complete source code is provided;
• Whether the vulnerable module is identified;
• Whether build and runtime instructions are available;
• Whether an initial input that triggers an abnormal condition is supplied;
• Whether debugging logs, stack traces, or runtime detection results are available;
• Whether a reference patch can be accessed;
• Whether the operating system, dependencies, and compiler parameters are fixed;
• Whether unlimited or numerous repeated attempts are permitted.

If the vulnerable location and trigger path are already largely known, the model is primarily solving a program-understanding, debugging, and exploit-engineering problem. It is not performing vulnerability discovery in the broader sense.

2.2 How Is “Success” Defined?

Different evaluations use very different success criteria. Common definitions include:

• Causing the program to crash;
• Triggering a specified memory-safety error;
• Reading a protected data region;
• Changing program control flow;
• Obtaining a predefined marker in an isolated environment;
• Completing a privilege escalation;
• Bypassing a sandbox or another security control.

Reaching a crash is not equivalent to reliably obtaining a specified privilege. Without a clearly defined success standard, a single percentage says very little about real-world capability.

2.3 Are Failure Cases Fully Counted?

An evaluation should also explain:

• How many attempts are permitted for each task;
• Whether only the best run is reported;
• Whether humans are allowed to select or refine model outputs;
• Whether multiple agents search in parallel;
• How timeouts, environment failures, and non-reproducible results are handled;
• Whether samples that cannot be built or executed are excluded;
• Whether successful results are independently verified.

If a task allows hundreds of cycles of generation, compilation, execution, and feedback before succeeding once, the result demonstrates search and iteration capability. It does not show that the model can complete the task reliably and inexpensively in a single attempt.

2.4 Were Realistic Defenses Enabled?

Production systems typically include multiple layers of defense:

• Address space layout randomization;
• Non-executable memory protections;
• Stack-canary protections;
• Control-flow protection;
• Process isolation and least privilege;
• System-call restrictions;
• Browser or application sandboxing;
• Container and virtualization boundaries;
• Request-rate limits;
• Anomaly and behavioral detection.

Success in an environment with protections disabled demonstrates only theoretical exploitability. Success rates generally decline when full hardening is enabled. Evaluation reports should therefore present separate results for a baseline environment, a partially hardened environment, and a fully hardened environment rather than combining them into one figure.

3. Vulnerability Discovery and Exploitation Must Be Evaluated Separately

“Discovering a vulnerability” and “exploiting a vulnerability” are often discussed together, but they represent different technical problems.

3.1 Vulnerability Discovery Is Primarily About Recall and Precision

A model can scan large amounts of code and generate many potential issues, but the number of candidates does not equal the number of real vulnerabilities. Organizations should pay attention to:

• The number of candidates generated per thousand lines of code;
• How many candidates can be reproduced reliably;
• How many are duplicates;
• How many represent only theoretical risks;
• How long it takes a human to confirm a real vulnerability;
• Whether high-severity issues are ranked near the top.

If a model produces 10,000 candidates but only a small number are genuine, analysts may be overwhelmed by false positives. For defenders, ranking quality and the completeness of supporting evidence are often more valuable than the total number of candidates.

3.2 Exploitation Is Primarily About Stability and Environmental Adaptation

A proof-of-concept that runs in an experimental container does not necessarily work in another environment. Exploitability is often affected by:

• Compiler and optimization settings;
• Memory allocation behavior;
• Dependency versions;
• Process privileges;
• Kernel configuration;
• Hardware architecture;
• Concurrency and timing;
• Security policies;
• Service startup parameters.

Exploitability validation should therefore distinguish at least three levels:

• Proof of concept: Demonstrates that the vulnerability can cause a security impact beyond the intended behavior;
• Stable validation: Succeeds repeatedly in the same environment;
• Migration validation: Maintains a high success rate across reasonable environmental changes.

Only the third level begins to resemble real-world attack capability.

4. Where Do AI Security Agents Provide the Most Value?

Although evaluation results must be interpreted carefully, the engineering value of these systems should not be underestimated. Their main advantages fall into four areas.

4.1 Faster Code Reading and Information Correlation

Security vulnerabilities often span multiple functions, modules, and privilege boundaries. Models can quickly organize call chains, data flows, resource lifecycles, and exception-handling logic, helping researchers narrow the scope of analysis.

4.2 Automated Trial and Error

In traditional vulnerability research, a significant amount of time is spent modifying inputs, rebuilding software, executing programs, reading logs, and adjusting hypotheses. An agent can connect these steps into a feedback loop:

Read source code and build information
    ↓
Form a vulnerability hypothesis
    ↓
Generate a minimal trigger input
    ↓
Execute in an isolated environment
    ↓
Analyze crashes, logs, and memory state
    ↓
Refine the hypothesis and validate again

When runtime feedback is sufficiently clear, this loop can continue advancing with less manual effort spent on repetitive tasks.

4.3 Parallel Exploration of Multiple Hypotheses

The same abnormal behavior may have several possible root causes. An agent can investigate boundary conditions, object lifecycles, integer calculations, authorization checks, and concurrency timing in parallel, then converge based on validation results.

4.4 Scaling the Experience of Security Experts

Experienced researchers can often recognize high-risk patterns quickly from the structure of a codebase. Models can turn parts of that experience into repeatable analysis procedures for large-scale code review. However, model judgments still require an evidence chain. Linguistic confidence must not be treated as proof that a vulnerability is real.

5. Which Metrics Should Organizations Use to Measure Real Capability?

To avoid being misled by a single success rate, organizations can establish a multidimensional evaluation framework.

5.1 Discovery-Quality Metrics

• Recall of confirmed vulnerabilities;
• Precision of candidate findings;
• Ranking quality for high-severity issues;
• Duplicate-finding merge rate;
• Cross-file and cross-module analysis capability;
• Ability to identify business-logic vulnerabilities.

5.2 Reproduction Metrics

• Rate of generating minimal trigger cases;
• Time required for the first successful reproduction;
• Average number of attempts;
• Stability across repeated executions;
• Completeness of failure explanations;
• Accuracy in identifying environmental dependencies.

5.3 Exploitability-Validation Metrics

• Conversion rate from a crash to proof of security impact;
• Success rate with different defenses enabled;
• Portability across versions and configurations;
• Amount of manual modification required;
• Ability to keep validation within the authorized scope;
• Whether testing creates side effects beyond the intended target.

5.4 Cost Metrics

End-to-end automation does not mean that costs are negligible. Model inference, parallel execution, environment construction, log storage, and human review all consume resources. At a minimum, organizations should measure:

• Average computing cost per confirmed vulnerability;
• Human review time per high-severity vulnerability;
• Additional cost caused by invalid candidates;
• Cost of building and maintaining isolated environments;
• Volume of code that can be processed per unit of time;
• Time saved compared with traditional security testing workflows.

A simplified return-on-investment metric can be expressed as:

Effective discovery cost =
(computing cost + environment cost + human review cost)
÷ number of confirmed, actionable vulnerabilities

This metric is generally more meaningful to the business than the number of vulnerability candidates generated.

6. What Still Separates Experimental Capability from Real Autonomous Attack?

Current systems still face several major barriers before they can be considered general-purpose autonomous hackers.

6.1 Uncertainty About Targets in Open Environments

Experimental tasks usually have clear targets and completion conditions. Real environments may contain large numbers of unrelated services, proxy layers, dynamic routes, and deceptive signals. Determining what should be analyzed is itself difficult—often more difficult than searching for a flaw in a known codebase.

6.2 Long-Running Tasks Are Vulnerable to State Drift

A complete security investigation may last hours or days and involve many files, experimental results, and competing hypotheses. A model may forget earlier constraints, repeat ineffective experiments, or mistake an accidental observation for a stable pattern.

6.3 Real Systems Provide Limited Feedback

An experimental environment can expose complete stack traces, memory state, and debugging output. Production systems may return only a generic error, a timeout, or a dropped connection. The less feedback available, the harder it is for a model to form an effective validation loop.

6.4 Generalization Remains Unstable

A model may perform well on common vulnerability patterns represented in its training materials but struggle with unfamiliar memory layouts, specialized business protocols, complex concurrency flaws, or cross-component logic issues. A high benchmark score does not automatically imply comparable performance on a completely new system.

6.5 Reliable Decisions Are Harder Than Code Generation

Generating a validation program is only a local task. A genuinely autonomous system must also determine whether the result is trustworthy, whether an action exceeds authorization, when testing should stop, and whether data could be damaged. Errors in security-boundary decisions can be far more serious than ordinary code-generation mistakes.

7. How Can Defenders Introduce These Capabilities Safely?

Organizations should not connect a security agent directly to a production network. Instead, they should use an architecture based on layered isolation, least privilege, and comprehensive auditing.

7.1 Establish Dedicated Isolated Environments

Analysis tasks should run in short-lived containers or virtualized environments with restrictions such as:

• External network access disabled by default;
• Only the code and samples required for the task mounted;
• Read-only or rollback-capable file systems;
• Limits on CPU, memory, storage, and execution time;
• No access to production credentials;
• Destruction of the environment after the task finishes;
• Retention of necessary logs and deletion of sensitive intermediate data.

7.2 Define Explicit Policies for Permitted Actions

Agent behavior can be constrained through policy configuration:

scope:
  source_repository: approved_only
  runtime: isolated
  production_access: denied
allowed_actions:
  - static_review
  - build_in_sandbox
  - generate_minimal_trigger
  - reproduce_crash
  - produce_remediation_advice
denied_actions:
  - external_scanning
  - credential_collection
  - persistence
  - data_exfiltration
  - destructive_testing
approval_required:
  - privilege_boundary_test
  - protection_bypass_test
  - cross_environment_validation

This example represents a general governance principle: deny high-risk actions by default and enable necessary capabilities only after authorization and human approval.

7.3 Preserve a Complete Evidence Chain

Every vulnerability conclusion should include:

• The relevant code location;
• The data flow or call path;
• Trigger prerequisites;
• Minimal reproduction steps;
• Runtime environment details;
• Actual security impact;
• Confidence level;
• Remediation recommendations;
• Regression-validation results after remediation.

If a system can only state that “a serious vulnerability may exist” without providing verifiable evidence, its finding should not be sent directly into a high-priority remediation workflow.

7.4 Add Human Approval Gates

The following stages should not be fully automated:

• Moving from static analysis to dynamic execution;
• Attempting to cross a privilege boundary;
• Validating a protection bypass;
• Accessing real data;
• Testing an environment similar to production;
• Sending vulnerability information to external parties.

AI can identify and organize findings, but a human security owner should retain final authority over high-risk actions.

8. Practical Impact on Security Operations and Software Development

The most important change is not whether AI has already become a “fully autonomous hacker.” The more immediate change is that the unit cost of vulnerability research is declining.

In the past, only a limited number of high-value projects could receive deep manual security review. As models take on code navigation, candidate filtering, trigger generation, and remediation organization, more ordinary projects may receive continuous security analysis. This will create at least three important changes.

First, vulnerability discovery may become faster. Attackers and defenders can both analyze code more quickly, shortening the window available for patch response.

Second, known defects may be converted into practical risks more easily. Some issues that were previously deprioritized because exploitation required too much engineering effort may need to be reassessed as automated validation becomes more capable.

Third, security testing will move earlier into the development lifecycle. Organizations can introduce automated analysis during code submission, build, and release approval so that issues are found before they reach production rather than relying primarily on post-deployment monitoring and incident response.

9. Conclusion: Capabilities Have Changed, but the Boundaries Must Be Clear

AI systems in cybersecurity are no longer limited to explaining vulnerability concepts or generating simple test code. They can increasingly read large codebases, form vulnerability hypotheses, invoke isolated environments, analyze runtime feedback, and iteratively validate their conclusions. Under controlled conditions, some systems can complete an end-to-end process from a defect signal to proof of security impact.

This does not mean that AI already possesses unrestricted, general-purpose autonomous attack capability. Source code, trigger inputs, fixed environments, and repeated attempts provided by an evaluation can significantly reduce task difficulty. Real-world systems introduce uncertainty about targets, limited feedback, complex defenses, environmental differences, authorization requirements, and operational risks.

A more precise conclusion is that AI is moving vulnerability research from an “expert-led, tool-assisted” model toward an “agent-executed, expert-supervised” model. Organizations should neither overestimate these systems based on promotional success rates nor ignore the changes that automation is bringing to the cost of offensive and defensive security work.

The most effective approach is to establish measurable evaluations, isolated execution, least-privilege controls, human approval gates, and complete auditing. AI should initially be used for code auditing, vulnerability reproduction, remediation prioritization, and regression validation. Only when capability development and governance mature together can this technology improve defensive efficiency without expanding security risk.