A Low Out-of-Scope Rate Cannot Guarantee That a Single Operation Is Within Scope

1. An Out-of-Scope Rate Answers a Different Question

An out-of-scope rate is a statistical result calculated across a batch of actions. The numerator is the number of actions classified as out of scope, and the denominator is the total number of actions.

Suppose a public evaluation reports an out-of-scope rate of 0.07%. This means that, among the evaluated samples, approximately seven out of every 10,000 actions were out of scope. Such a figure is useful for comparing two approaches and monitoring long-term trends.

A sign-off, however, applies to one specific task. The person providing the sign-off must determine whether every action in that task is within the authorized scope. These are different questions at different levels of granularity. One conclusion cannot replace the other.

(1) More Actions Mean Higher Risk

Assume that actions are independent. The probability that a task involving at least one out-of-scope action is:

[
1 - (1 - p)^n
]

where p is the out-of-scope rate for a single action and n is the number of actions.

Using p = 0.0007:

• 100 actions: approximately 6.8%
• 200 actions: approximately 13%
• 1,000 actions: approximately 50%
• 3,000 actions: approximately 88%

A test involving thousands of requests across multiple hosts is therefore likely to contain at least one out-of-scope action, even when the per-action rate is very low.

The independence assumption may not hold in real-world environments. A single misunderstanding of the authorized scope can be repeated across many actions. In that case, the risk may be concentrated in a particular class of actions and become more serious than the independent-action estimate suggests.

(2) The Rate Itself Has Uncertainty

If the 0.07% figure comes from sample-based review, the sample size must be considered first. Suppose 10,000 actions were reviewed and seven out-of-scope actions were identified. Using an exact Poisson method, the 95% confidence interval for the true out-of-scope rate is approximately 0.03% to 0.14%.

The smaller the sample, the wider the interval. Treating a point estimate as a commitment for a single task is therefore not statistically valid.

(3) Unlogged Actions Cannot Be Included in the Rate

A statistical measure covers only the actions that are recorded. If execution includes paths that bypass logging—for example, direct calls to a lower-level interface or network requests embedded in a script—the reported out-of-scope rate may appear low while actual coverage is also low.

A low rate and high coverage must therefore be reported separately.

2. Define Scope as Checkable Rules

If scope exists only in a natural-language authorization statement, an execution system cannot reliably evaluate it. At a minimum, a checkable scope should include the following elements:

• Target assets: Specific identifiers such as domain names, address ranges, code repositories, or accounts. Avoid vague descriptions such as “related systems.”
• Allowed action types: Clearly distinguish read-only probing, vulnerability validation, configuration changes, and other action categories.
• Time window: Start and end times, including the applicable time zone.
• Prohibited actions: For example, data export, high-volume stress testing, or write operations against production data.
• Stop conditions: Circumstances that require immediate suspension and escalation.
• Data-handling requirements: Retention periods for samples and applicable redaction methods.

The following is an illustrative scope structure and evaluation function. The field and function names are defined for this article and do not correspond to any specific product:

from dataclasses import dataclass, field
from datetime import datetime, timezone
from fnmatch import fnmatch
@dataclass
class ScopeSpec:
    task_id: str
    allowed_hosts: list[str]
    allowed_actions: set[str]
    valid_from: datetime
    valid_until: datetime
    forbidden_act