Rethinking Test Boundaries for AI Agent Systems: From Model Evaluation to Full-Chain Validation

    This article explains why testing long-running, stateful AI agents that call external tools must move beyond checking answers to validating the full process. It covers how test focus shifts for agent systems, how restricted frontier model testing is changing security validation, and how persistent state creates long-term contamination risks. It also presents a three-dimension framework of capability, identity, and knowledge for designing tests.

    1. Testing Targets Are Shifting from Models to Systems

    Evaluating a large language model has usually meant preparing a set of questions and comparing the answers for accuracy, format, and response time. This method still works for single-turn question answering. But for agents that work continuously for hours, read and write memory, and call external tools, it covers only the surface behavior.

    Test focus needs to shift when a system has the following characteristics:

    • Long-running operation: Tasks span multiple reasoning rounds and may be interrupted, restarted, or taken over manually.
    • Persistent state: The system remembers user preferences, past actions, and intermediate results, and this state affects later decisions.
    • Tool use: The system can read and write files, send requests, and change configurations. Wrong actions have real consequences.
    • Edge or hybrid deployment: The same capability runs on different devices and network environments, so behavior varies more widely.

    Under these conditions, one correct answer does not prove that a system is reliable. Testing needs to answer other questions. Is each action within authorized scope? Has the intermediate state been contaminated? Can the same input produce explainable results across different environments? Can a problem be reproduced after it occurs?

    The focus of industry competition is also changing. Parameter count and leaderboard rankings still have reference value. However, enterprises care most about whether a system can reliably complete a business task and stop in a controlled way when something goes wrong. The value of quality assurance is shifting from verifying answers to verifying processes.

    2. Restricted Testing of Frontier Models: Changing Participants in Security Validation

    Some recent frontier models were not released publicly in standard versions. Instead, they entered restricted testing programs, where specific teams evaluate their capability boundaries in controlled environments. This approach suggests that these models have reached a level of capability in code understanding, vulnerability identification, and call-chain reasoning that requires dedicated constraints.

    This affects testing teams in three main ways.

    First, security testers will rely more on automated reasoning. Previously, vulnerability analysis depended mainly on expert experience, with tools handling scanning and support. Now models can help read code, summarize suspicious paths, and generate verification ideas. Testing teams need to define which steps models may participate in and which steps must be reviewed by people.

    Second, red team validation must include the agent itself. The systems under test are no longer limited to business interfaces. They also include the tools a model can call, the data sources it can access, and its permission settings. Red teams need to simulate unauthorized calls, misleading inputs, and multi-step combined attacks.

    Third, validation requires controlled boundaries. Such testing needs isolated environments, least-privilege credentials, logging of all actions, and human confirmation checkpoints. Any operation that could cause external side effects should be replaced with mocks or sandboxes in the test environment.

    3. New Threats from Persistent State: From One-Time Injection to Long-Term Contamination

    Single-turn prompt injection was one of the first widely discussed risks. An attacker hides instructions in one input to steer the model away from its intended behavior. For systems that handle only one request at a time, the impact usually stays within the current session.

    Persistent state changes this. When an agent writes external content into memory or a knowledge base, malicious content may be read days later and take effect in unrelated tasks. Recent public research reports that, under specific experimental settings, poisoning attacks against persistent state reached a success rate of 60% to 70%. This figure shows the risk should not be ignored, but it depends on experimental conditions and should not be directly extended to all systems.

    To support test design, the persistent state of a personal agent can be divided into three dimensions.

    1. Capability

    This covers what the system can do, including the tools it can call, the resources it can access, and the rules it runs automatically. Testing should focus on whether permissions are narrowed. Check whether tools are limited to the task, whether high-risk actions require confirmation, and whether permissions are unintentionally expanded as state accumulates.

    2. Identity

    This covers whom the system acts for, including accounts, roles, delegation relationships, and credential lifecycles. Testing should focus on whether identity switches happen correctly, whether expired credentials are revoked, and whether subtasks inherit more permissions than they need.

    3. Knowledge

    This covers what the system remembers, including conversation summaries, user preferences, retrieved documents, and intermediate conclusions. Testing should focus on whether the source of each piece of knowledge can be traced and whether untrusted content is isolated.