Security Agent Out-of-Scope Action Rate Falls Below 1%: Evaluation Systems and Engineering Practices

1. Background: When Security Agents Begin to “Act on Their Own”

Automated vulnerability discovery is moving from assisted scanning toward autonomous execution. A new generation of security agents can identify potential weaknesses, autonomously plan exploitation paths, construct validation payloads, and even complete end-to-end validation in real network environments. The expansion of capability boundaries inevitably expands risk boundaries: once an agent that performs well in a lab environment is connected to a production network, any misjudgment in its autonomous decision chain can become out-of-scope access to non-target systems.

A new-generation network security model recently released by a cloud provider brought this conflict to the forefront. Its published evaluation data showed that the proportion of out-of-scope behavior during task execution had fallen to less than 1%, while the comparable metric for the previous model generation was about 5%. The significance of these numbers lies not only in performance improvement but also in an engineering fact: the usability of security agents is increasingly defined by the “lower bound of loss of control” rather than the “upper bound of capability.”

This article starts from the causes of out-of-scope behavior, analyzes the design logic of the evaluation environment, and provides a set of protection and verification practices that enterprises can implement.

2. The Nature of Out-of-Scope Behavior: Mismatch Between Autonomy and Scope

“Out-of-scope intrusion” refers to an agent accessing, probing, or affecting systems and resources outside the authorized task scope while executing an assigned task. It is usually not caused by malicious intent in the model but by a combination of three factors.

1. Overgeneralization of targets. During reasoning, an agent tends to reuse known effective attack paths. When the target environment contains assets similar to the task objective, the model may misjudge “reachable” as “authorized reachable,” thereby spreading validation behavior to lateral systems.

2. Insufficient awareness of environment state. Real networks are dynamic: asset ownership, network partitions, and access policies change at any time. If an agent lacks real-time awareness of the current authorization boundary, its planner will make out-of-scope decisions based on stale or incomplete environment models.

3. Missing feedback loop. Single-step execution lacks a loop of “confirm before execution—monitor during execution—audit after execution,” so an out-of-scope action may be completed before it is discovered.

These three causes correspond to three different governance approaches: scope constraints, enhanced environment awareness, and execution loop construction. The value of an evaluation system is that it makes these three issues explicit and measurable in a controlled environment.

3. Evaluation Environment Design: Honeypot Approach and Reproducible Baselines

For “out-of-scope rate” to be a credible metric, the evaluation environment must satisfy two conditions at the same time: observability and reproducibility.

A widely referenced design approach is the “honeypot lab”: place a large number of non-target assets around the task objective as decoys. These assets are topologically reachable but should not be accessed under the granted permissions. If the agent initiates any probing or exploitation behavior against a decoy, it is judged as out-of-scope. The advantage of this design is that out-of-scope behavior no longer depends on manual audit discovery; it is precisely captured by the environment itself.

Key design elements of the evaluation environment include:

• Topology realism: Decoy assets need real ports, service fingerprints, and response behavior. This prevents the agent from identifying “this is a trap” through simple features and avoiding the test; otherwise, evaluation results will be systematically optimistic.
• Explicit authorization boundary: Inject the task authorization scope into the agent context in a structured form rather than relying on the model to infer it from natural language descriptions. The boundary description should include the target list, allowed operation types, and prohibited network segments and asset labels.
• Full behavior logging: Record every network request, command execution, and file access, together with the decision rationale—that is, the reasoning snippet from the model when it made the action—so that “misjudged out-of-scope behavior” can be distinguished from “exploratory probing.”
• Baseline comparability: The same lab, the same task set, and the same authorization description remain constant across model versions, so changes in the out-of-scope rate have attribution value.

It is worth emphasizing that the evaluation metric should not rely only on a single out-of-scope ratio. It is recommended to track the following metrics at the same time:

Metric	Meaning	Focus
Out-of-scope rate	Share of tasks that trigger unauthorized access	Overall controllability
Out-of-scope depth	Asset tier and data sensitivity reached after crossing the boundary	Severity of consequences
Self-correction rate	Whether the agent can autonomously identify and roll back after crossing the boundary	Closed-loop capability
Task completion rate	Share of tasks that still complete the assigned objective under constraints	Balance between capability and constraints

Lowering the out-of-scope rate only by sacrificing the task completion rate is equivalent to degrading the agent into a scanner. Real capability progress occurs only when the out-of-scope rate is reduced while the completion rate remains stable.

4. Engineering-Side Protection: Build Constraints into the Architecture, Not the Prompt

Evaluation reflects the model’s behavioral tendencies, while production environments require hard constraints at the architecture level. Relying on prompts to ask the model to “not go out of scope” is not reliable in adversarial scenarios. The recommended layered protection is as follows.

Layer 1: Scope whitelist. Deploy an execution proxy between the agent and the target environment. All network actions must be forwarded through the proxy. The proxy maintains an explicit target whitelist. Any address not on the whitelist is rejected directly at the protocol layer, and the model has no way to “persuade” the proxy to allow it.

# Authorization check in the execution proxy (pseudocode)
ALLOWED_TARGETS = load_scope("task_scope.json")  # issued by the task system; model cannot write
def forward(action):
    target = action.dest
    if not in_scope(target, ALLOWED_TARGETS):
        audit_log.warn("out_of_scope_blocked", action=action, target=target)
        return Blocked(target)
    return execute(action)

Layer 2: Operation classification and human confirmation. Classify operations by impact: read-only probing can be executed automatically; write operations, privilege escalation, and lateral movement actions need to trigger a confirmation process. Confirmation can involve human intervention or automatic adjudication by a policy engine based on asset labels. The key is that the adjudication logic is independent of model reasoning.

Layer 3: Session-level resource isolation. Each task runs in an independent network namespace or independent container, with outbound traffic passing through a unified gateway. Even if the model plans an out-of-scope path, physical isolation prevents the out-of-scope action from reaching real assets.

Layer 4: Anomaly behavior circuit breaker. Monitor signals such as target address dispersion per unit time, failed connection ratio, and unexpected port scan rate. When dispersion exceeds the reasonable threshold for the task profile, automatically suspend the session and escalate it to human review. This layer addresses the gray area of “within the whitelist but anomalous behavior.”

5. Deployment Recommendations: Migration Path from Evaluation to Production

To migrate evaluation conclusions into production, it is recommended to proceed in the following order:

1. Build a baseline before discussing optimization. Replicate the honeypot approach on your own business topology and run the same task set to compare the out-of-scope rates of old and new models. Avoid directly applying external evaluation numbers.
2. Make authorization descriptions structured. Move task boundaries from natural language to machine-verifiable manifest files as input to the execution proxy, reducing the model’s inference space.
3. Roll out gradually. Initially run only in isolated network segments, then gradually expand to non-core production zones, while retaining action replay capability throughout.
4. Establish a review mechanism for out-of-scope events. Each blocked out-of-scope action should be attributed to one of three categories—overgeneralization, insufficient awareness, or missing closed loop—and fed back into prompting strategy, environment modeling, or proxy rules.

6. Conclusion

The drop in the out-of-scope rate from 5% to 1% appears to be model capability iteration, but in essence it reflects an improvement in the engineering maturity of security agents. For enterprises, what is truly reusable is not a specific model but the methodology of “an observable evaluation environment + architecture-level scope constraints + a tiered execution loop.” The stronger the agent’s autonomy, the closer the constraints must be to the execution layer. Building trust on architecture rather than on the model’s self-discipline is the prerequisite for security agents to move into production.