Demystifying “Evasion” in AI Security: Adversarial Examples vs. Agent Boundary Violations

In AI system security discussions, the term “evasion” frequently appears, yet it is often assigned drastically different meanings. Some use it to describe a model being fooled by a carefully crafted image, while others use it to describe an AI agent bypassing permission constraints and executing unintended operations. These two phenomena differ vastly in technical causes, threat models, and defense mechanisms. Conflating them leads to misallocated monitoring resources and can cause teams to misjudge real risks. This article starts by clarifying the terminology, dissects the essential differences between the two types of “evasion,” and discusses why certain monitoring methods are effective for one but inadequate for the other.

I. Clarifying the Two Types of “Evasion”

1.1 Recognition-Level Evasion: Adversarial Examples

In traditional software security, “evasion” typically refers to breaking out of a sandbox, container, or permission isolation to move from a restricted environment to the host environment. However, in the context of AI model recognition tasks, “evasion” refers to samples “escaping” the model’s correct classification boundary.

A typical scenario is an adversarial example attack: an attacker overlays almost imperceptible perturbation noise onto an image of a panda. Human observers still see a panda, but the image classification model might classify it as a gibbon with extremely high confidence. The root cause of this phenomenon lies in the high-dimensional geometric properties of the model’s decision boundary—there are numerous directions in the input space that are imperceptible to humans but can significantly alter the model’s output. The model is not “compromised”; it simply provides an incorrect answer based on its own judgment logic.

From an engineering perspective, the essence of this type of evasion is a model robustness defect. The model performs well on in-distribution samples but lacks stability against deliberately constructed neighboring samples. It does not involve the model actively bypassing any permission constraints; the model faithfully executes inference, but the result is wrong.

1.2 Behavioral-Level Evasion: Permission Boundary Violations

Another type of “evasion” occurs in agentic systems equipped with tool-calling capabilities. When a model is granted abilities such as executing code, accessing the file system, or initiating network requests, it may bypass expected constraints and execute unauthorized operations. The essence of this issue is a security boundary design defect: mechanisms like permission models, sandbox isolation, and operation auditing fail to cover the entire behavioral space of the agent.

It is crucial to emphasize that such boundary violations do not necessarily mean the model possesses “subjective intent.” A more common scenario is that the model generates a piece of code that seems reasonable but actually triggers an underlying system exception, or unexpectedly bypasses single-step checks through a combination of multi-step tool calls. For instance, model-generated code might trigger a kernel panic or deadlock in the host operating system, thereby affecting the stability of the entire training cluster. While this phenomenon is easily likened to a “jailbreak,” technically, it is closer to an unexpected side effect rather than a deliberate breach.

1.3 Why the Distinction is Necessary

The defense logic for the two types of issues is entirely different:

• Defending against adversarial examples focuses on the input side: sanitizing inputs, detecting anomalous perturbations, improving model robustness, and adding ensemble voting.
• Defending against permission violations focuses on the behavioral side: least privilege, sandbox isolation, operation auditing, secondary confirmation for hazardous operations, and resource quotas.

Applying adversarial sample monitoring concepts to agentic systems will underestimate the complexity of the behavioral space. Conversely, using permission control concepts to address adversarial examples will not fix the model’s judgment errors. The first step in threat modeling is to clearly identify which type of risk the current system faces.

II. Where the Differences in Monitoring Feasibility Come From

A key observation is that some systems can effectively monitor recognition-level evasion, while others face significantly higher difficulties when dealing with behavioral-level violations. This difference is not due to implementation quality but is determined by the system’s structure.

2.1 Converged Monitoring Surface with Limited Input Channels

Consider a system that receives external information only through limited channels: its input sources are clear, formats are controllable, and call paths are transparent. In this case, defenders can focus monitoring efforts on these entry points—performing consistency checks on inputs, detecting anomalous patterns, and adding secondary verification for critical operations (such as resource transfers or sensitive commands). Since the pathways to being “deceived” are enumerable, the monitoring surface naturally converges.

This is similar to anti-fraud scenarios: if a person has limited channels to access external information, controlling those channels and adding verification for high-risk actions can significantly reduce the probability of being deceived. Input sanitization, perturbation detection, and confidence threshold alerts in adversarial sample defense are reflections of this same logic.

2.2 Diverged Monitoring Surface with Open Behavioral Space

When a system is granted broad operational permissions—capable of calling various tools, accessing OS resources, and combining multi-step actions—the behavioral space becomes difficult to enumerate. Violations can occur in the parameters of any single tool call, in the combined effects of multi-step calls, or at the interaction boundary between the underlying runtime and the model’s output.

The monitoring challenges for such systems include:

1. Combinatorial explosion: Individual single-step calls are legitimate, but their combination produces unexpected effects.
2. Semantic gap: The model-generated code is syntactically correct but semantically triggers an underlying exception.
3. Observation cost: Determining whether an operation is “out of bounds” often requires understanding the contextual intent, and intent itself is difficult to formalize.
4. Latency requirements: If auditing occurs after the operation, the damage is already done; requiring pre-emptive interception demands low-latency judgment capabilities.

Therefore, monitoring behavioral-level evasion cannot be solved by simply “adding a check.” It requires layered design: ex-ante least privilege and sandboxing, in-process runtime constraints and resource quotas, and post-hoc audit tracing and anomaly review.

III. Layered Defense in Engineering Practice

3.1 Recognition-Layer Defense: Focusing on Inputs

Common methods against adversarial example evasion include:

• Input preprocessing: Denoising, compression, and randomized transformations to weaken deliberately constructed perturbations.
• Robustness training: Introducing adversarial examples during the training phase to enhance decision boundary stability.
• Ensemble and voting: Cross-validating with multiple models to reduce the probability of a single model being deceived.
• Confidence monitoring: Triggering manual review for low-confidence or out-of-distribution samples.

The common premise of these methods is that the attack surface is primarily on the input side, and the input channels are observable and intervenable.

3.2 Behavioral-Layer Defense: Focusing on Permissions and Runtime

For permission violation risks, the defense focus shifts to runtime constraints:

# Example: Least-privilege wrapper for tool calls
def safe_execute(tool, args, policy):
    if not policy.allows(tool, args):
        raise PermissionError("operation outside declared policy")
    with sandbox_limits(cpu=..., mem=..., net=...):
        result = tool.run(args)
    audit_log.record(tool, args, result)
    return result

Key design points include:

1. Least privilege: Granting each tool and call only the capabilities essential for completing the task.
2. Sandbox isolation: Restricting CPU, memory, network, and file system access to prevent a single-point anomaly from spreading to the host.
3. Secondary confirmation for hazardous operations: Adding explicit confirmation steps for resource changes and external communications.
4. Full-chain auditing: Recording tool call sequences to support post-hoc reconstruction and anomaly detection.
5. Resource quotas and circuit breaking: Triggering rate limiting or abortion for abnormally high-frequency calls or abnormal resource consumption.

It should be noted that these mechanisms cannot completely eliminate the risk of boundary violations, but they can compress the probability of a “single-point failure causing a global fault” to an acceptable range. Just as network security infrastructure cannot reduce attacks to zero, it can significantly lower the attack success rate and the scale of damage.

3.3 The Two Defenses are Not Interchangeable

A common misconception is: since input-side monitoring is already in place, are runtime constraints unnecessary? The answer is no. Input sanitization cannot solve the problem of “model-generated code crashing the underlying system.” Similarly, sandbox isolation cannot solve the problem of “the model mistaking a panda for a gibbon.” The two methods target different failure paths and must be implemented in parallel.

IV. Reflecting on Descriptive Terminology

The wording in technical documentation shapes the reader’s risk perception. When a description uses terms like “kernel panic,” “deadlock,” or “threatening the entire cluster,” readers easily associate it with narratives of “jailbreaking” or “hacker warfare.” Returning to technical facts, however, this is often just an unexpected code generated by the model triggering a host OS exception. It is a runtime stability issue, not a deliberate breach.

An accurate description should answer three questions: At which layer did the failure occur (model judgment, tool call, runtime, host)? What are the trigger conditions (input perturbation, parameter combination, resource competition)? What is the impact scope (single call, single instance, entire cluster)? Only by clarifying these questions can defense resources be invested in the right places.

V. Conclusion

“Evasion” in AI security encompasses at least two types of issues. Recognition-level adversarial example evasion is essentially a model robustness defect, and its monitoring surface converges on input channels. Behavioral-level permission violation is essentially a security boundary design issue, and its monitoring surface diverges across an open behavioral space. The two have different causes and defense logics; they cannot replace each other and should not be directly compared.

For developers, the pragmatic approach is to first define the threat model, then deploy defenses by layer: perform sanitization and robustness building on the input side; enforce least privilege, sandbox isolation, and full-chain auditing on the behavioral side; and implement resource constraints and circuit breaking at the runtime level. Layered defense does not pursue perfection at any single layer but aims to prevent single-point failures from cascading into systemic faults. This is both an engineering trade-off and basic common sense in security building.