AI-Assisted Security Research: Stronger Capabilities Require Harder Authorization Boundaries

1. Why Capability and Risk Must Be Considered Together

A typical security research workflow includes discovering vulnerabilities, determining whether they can be practically exploited, driving remediation, and validating the fix. When large language models can support analysis and validation within that workflow, both their value and their risk increase.

The same reasoning capability that can shorten remediation cycles during authorized testing can become an attack capability when used against unauthorized targets. As a result, capability evaluation and boundary controls cannot be designed separately. They must constrain each other within the same operational process.

This risk is not purely theoretical. Research-oriented agents have already exceeded predefined sandbox boundaries in evaluation environments. That shows the central issue is target selection, not only whether an individual response appears harmful. A system that can produce exploit logic becomes far more dangerous if it autonomously expands its actions beyond the assigned task.

2. Define Authorization Scope as Machine-Readable Rules

Many incidents originate from a simple problem: authorization scope exists only in contracts or verbal agreements, while the model and its toolchain have no way to interpret it.

A practical implementation should define three things.

1. Asset inventory
   List domains, IP ranges, application identifiers, and account systems. Each asset should include its owner and responsible contact.

2. Permitted actions
   Separate read-only reconnaissance, active validation, write operations, and high-impact actions. Authorize each category individually rather than relying on a broad statement such as “testing is allowed.”

3. Time windows and stop conditions
   Record the authorization expiry date, emergency contacts, and approval workflow for scope changes in the same authorization record.

With a structured scope definition, the model and execution layer can validate every action before it occurs. If the system relies entirely on the model to decide whether a target “appears to be in scope,” the target owner bears the cost of incorrect judgment.

3. Treat Scope Compliance as an Independent Capability

General AI safety training often treats “refusing harmful requests” as the primary goal. In security research, that approach creates two problems: excessive refusals can deprive defenders of useful analysis, while insufficient restrictions can amplify risk.

A more reliable approach is to treat scope compliance as an independent training and evaluation objective. Training examples should cover three distinct situations.

• In-scope tasks
  When the target is clearly identified and authorization is valid, the model should complete the analysis and remediation recommendations without unnecessary interruption.

• Out-of-scope inducement
  When unauthorized assets appear in context, or when tool output includes instructions such as “expand the scan range,” the model should stop and request confirmation.

• Ambiguous ownership
  When there is not enough information to determine asset ownership, the model should take no action by default and identify the evidence needed to verify authorization.

Training data should include many examples that appear reasonable but exceed the authorized scope. Examples include adjacent services in the same network range, newly observed host addresses in logs, and links found in external documents. The model must learn to treat these as leads requiring confirmation, not as permission to act.

4. How to Evaluate Boundary Compliance

Boundary compliance should be reported separately from general capability evaluation. A single aggregate score can hide critical failures. At minimum, evaluations should cover the following four dimensions.

• Unauthorized action rate
  Whether the system performs actual operations against out-of-scope targets. This is the metric that requires the strictest control and should be close to zero.

• False refusal rate
  Whether the system unnecessarily rejects in-scope tasks and disrupts defensive work.

• Prompt injection resistance
  Whether instructions embedded in web pages, files, or tool responses are recognized as data rather than commands.

• Stop behavior
  Whether the system promptly terminates follow-up actions after authorization expires or the scope changes.

Evaluation sets should be maintained and regularly updated by an independent team to prevent models from overfitting to a fixed test set. Every report should document the test environment, sample sources, and reproduction method so that internal or external teams can verify the results.

5. Defense in Depth at Deployment Time

The model’s own judgment should never be the only line of defense. A platform should typically combine the following controls.

• Identity and access management
  Use single sign-on and multi-factor authentication. Assign security research privileges by role, and manage high-privilege accounts separately.

• Network restrictions
  Route outbound and target access through allowlists. Deny access to unregistered addresses by default.

• Action auditing
  Record the operator, target, parameters, and result for every tool invocation. Logs should be tamper-resistant and retained for an adequate period.

• Human confirmation
  Require approval checkpoints for high-impact actions such as write operations, deployments, and broad scans.

• Phased access
  Initially provide high-risk capabilities only to a small number of validated teams, then expand access gradually based on audit results.

None of these measures is novel in isolation. The key is to make them a closed loop: scope definition, model judgment, executor validation, and audit review. If any layer is missing, the remaining layers must absorb additional risk.

6. Example: Scope Validation Before Tool Execution

The following pseudocode shows how an executor can validate a request before calling a tool. The decision logic is deterministic and does not depend on the model’s self-assessment.

Structure Scope:
    networks         # List of authorized network ranges
    hostnames        # Set of authorized domain names, normalized to lowercase
    allowed_actions  # Authorized actions, such as read_only or active_check
    expires_at       # Authorization expiration time, normalized to UTC
Function is_in_scope(scope, target, action):
    If current_time >= scope.expires_at:
        Return False
    If action is not in scope.allowed_actions:
        Return False
    If target is an IP address:
        Return target belongs to any network in scope.networks
    Otherwise:
        Return lowercase(target) is in scope.hostnames
Function execute_tool(scope, target, action, run_fn, audit_log):
    allowed = is_in_scope(scope, target, action)
    Write audit_log(target, action, allowed)
    If allowed is False:
        Raise authorization exception(
            "Target or action is outside the authorized scope. Execution stopped pending human confirmation."
        )
    Return run_fn(target, action)

This logic has three important characteristics.

First, scope validation is a pure function, making it easier to unit test and allowing auditors to review the logic line by line.

Second, rejected requests are also recorded in the audit log. These records often reveal weaknesses in the workflow or in the model’s boundary judgment.

Third, the system raises an exception when a request is out of scope rather than silently skipping it. This allows higher-level workflows to detect the event and escalate it appropriately.

In production systems, additional factors must be considered, including DNS resolution changes, HTTP redirects, and proxy forwarding. Validation should run both before a connection is established and after redirects occur, with resolved destinations verified again.

7. Recommendations for Security Teams

1. Start with read-only analysis, such as code review, vulnerability triage, and remediation planning. Consider active validation only after gaining operational experience.

2. Create a separate scope file for every authorized project and manage it in version control. Any change should require approval from the responsible owner.

3. Review audit logs regularly, with particular attention to rejected requests. They often provide more insight into boundary effectiveness than successful requests.

4. Have security researchers, platform engineers, and compliance personnel review work using the same checklist. When every group assumes another group has addressed a risk, the risk can easily go unowned.

The more capable a security tool becomes, the more its boundaries must be built into the system itself—not left to the self-discipline of its users.