Dynamic Attack-Defense Evaluation Reveals Clear Capability Gaps Among AI Agents

    This article explores how dynamic attack-defense evaluation frameworks expose significant capability differences among AI agents in real-world cybersecurity scenarios. Unlike static question-based tests, dynamic evaluations require agents to simultaneously attack opponent services and defend their own, revealing critical differences in reconnaissance, planning, recovery, and resource allocation. The findings challenge traditional assessment methods and offer practical guidance for enterprises looking to evaluate and deploy security AI agents.

    Moving from “Answering Questions” to “Active Combat”

    In the past, evaluating an AI agent’s cybersecurity capabilities typically involved a set of predefined tasks: vulnerability exploitation, configuration analysis, or script generation. Scoring was based on whether the model produced correct answers. This approach can reflect a model’s knowledge base, code generation ability, and analytical skills, but it has a notable flaw: the test environment is static. The attack target does not actively patch vulnerabilities, and the defender does not adjust its strategy based on the attacker’s behavior.

    Real-world network environments operate differently. Attackers continuously scan, probe, and exploit, while defenders patch vulnerabilities, adjust access controls, restart services, update rules, and strive to maintain business continuity. Attack and defense actions influence each other—each decision changes the conditions of the next round. Therefore, a truly valuable security agent must not only “know how to attack” but also be able to judge, execute, and correct course in uncertain, constrained, and constantly changing environments.

    To address this, a research team designed a dynamic attack-defense evaluation environment. The environment incorporates several frontier AI models with their accompanying execution frameworks. Each agent takes on two roles simultaneously: first, it must identify weaknesses in the opponent’s services and execute attacks; second, it must maintain the availability of its own services, handling vulnerabilities, abnormal processes, and configuration risks in a timely manner. The evaluation no longer focuses on whether individual questions are answered correctly, but comprehensively examines attack effectiveness, defensive performance, service stability, and sequential decision-making ability.

    Evaluation Results Show Clear Tiers

    After 22 batches and 466 valid matches, the systems’ scores formed distinct tiers. The top two systems scored approximately 78.6 and 73.8 points respectively. The third-ranked system scored around 68.5 points. The remaining systems were mostly distributed between 56.3 and 67.1 points.

    What is even more revealing is the difference in win rates. The top two systems achieved win rates of 0.83 and 0.86, while the highest win rate among the remaining systems was only about 0.59. This suggests that dynamic confrontation does not simply amplify static capabilities proportionally—it can further magnify differences in planning, execution, feedback utilization, and resource management.

    This gap typically stems from the accumulation of differences across multiple stages:

    1. Different reconnaissance sequences
      Some agents first build a comprehensive view of assets, ports, service versions, and permission relationships before deciding on an attack path. Others jump straight to common exploitation attempts, wasting time on low-value targets.

    2. Different action planning capabilities
      Security confrontation rarely succeeds in a single operation. An attacker may need to gain low-level access first, then find credentials, escalate privileges, and finally impact critical services. Systems that can decompose objectives into multiple phases and adjust plans based on intermediate results are more likely to achieve consistent gains.

    3. Different recovery capabilities after failures
      A single command failure does not mean the path is entirely infeasible. A strong agent will diagnose the cause of failure—distinguishing between insufficient permissions, incorrect paths, services already patched, or environmental changes—rather than repeating the same action.

    4. Different resource allocation between offense and defense
      An agent cannot devote all resources to attacking. If it neglects its own service patching, process states, and permission configurations, it may be defeated by the opponent before its offensive succeeds.

    Static Test Scores Do Not Represent Real Combat Capability

    The study also included a single-player mode as a control. With no active defense and two attempts allowed per question, 30 out of 40 questions could be fully solved by multiple systems. On the surface, these systems appeared proficient in security knowledge, vulnerability analysis, and script generation.

    However, when the same 30 questions were placed in a bidirectional adversarial environment, results deteriorated rapidly. As defenders continuously patched vulnerabilities, by the third round, the attack capture rates of three representative systems had dropped to approximately 52%, 37%, and 35% respectively. In other words, attack paths that appeared stable in a single-machine environment could fail by half or more once placed in dynamic interactivity.

    This phenomenon reveals an important fact: security capability is not a simple extension of “question-solving rate.” It is a systemic competency spanning multiple dimensions.

    First: Correct Knowledge Does Not Equal Effective Action

    A model may accurately describe a vulnerability’s principle and generate plausible exploit code, but real-world execution is also affected by target versions, service states, permission boundaries, network connectivity, and defensive actions. A theoretically sound approach can fail entirely if it is not validated against the current environment.

    Second: One-Time Success Does Not Equal Sustained Control

    Attackers must deal with service restarts, invalidated credentials, patch updates, and changes in access policies. A single successful exploit only proves that an exploitable condition existed at that specific point in time. It does not prove that the agent can maintain control and expand its foothold.

    Third: Local Optimization Can Lead to Overall Failure

    In pursuit of quickly capturing a low-value target, an agent may expose its own behavioral characteristics, exhaust its execution attempts, or even damage its own services. Dynamic confrontation requires systems to weigh short-term gains against long-term risks, rather than pursuing an immediate score.

    How to Decompose Agent Security Capabilities

    For enterprises and development teams, evaluating an agent should not be limited to a single overall score. A multi-dimensional indicator system should be established.

    Reconnaissance and situational awareness.
    Can the system identify assets, services, open interfaces, permission relationships, and anomalous changes? Can it organize discrete information into a usable environmental model? Without stable situational awareness, later decisions will be built on outdated information.

    Attack path planning.
    Can the system prioritize actions based on target value, success probability, and exposure risk? When multiple paths exist, can it select one that is lower cost, more covert, and recoverable? This reflects practical capability better than merely generating an exploit snippet.

    Defense and remediation capabilities.
    Can the defending agent locate the source of anomalies, assess the impact scope of vulnerabilities, and take minimal-impact remediation measures? Good defense should not simply shut down ports or stop services; it must balance business continuity, configuration consistency, and follow-up validation.

    Feedback utilization capability.
    Every execution should produce new evidence: return codes, logs, process changes, network connections, and permission states. Whether an agent can correctly interpret this feedback and use it in subsequent actions is the key differentiator between “automated script” and “autonomous agent.”

    Service availability.
    If an agent deletes critical configurations while trying to patch vulnerabilities, or neglects its own services while attacking an opponent, its security value remains limited. Evaluation must account for service survival rate, recovery time, and misoperation rate.

    System Design Recommendations for Engineering Deployment

    In practical deployment, it is not advisable to grant a large language model unrestricted system access. A more robust architecture can be divided into four layers.

    The first layer is the environment observation layer, responsible for collecting asset inventories, service states, logs, alerts, configurations, and permission information, while desensitizing sensitive data. Observations should carry timestamps to prevent the agent from making decisions based on stale state.

    The second layer is the analysis and planning layer, responsible for generating candidate paths, assessing risks, calculating priorities, and clarifying the expected outcomes of each action. Planning results should retain reasoning and dependency conditions for auditing and traceability.

    The third layer is the controlled execution layer, which restricts high-risk actions through permission tiers, operation whitelists, sandboxes, timeout controls, and rollback mechanisms. Operations involving data deletion, modification of critical configurations, privilege escalation, or service interruption should default to human approval or secondary confirmation.

    The fourth layer is the validation and feedback layer, which checks whether actions achieved their expected results, confirms whether services remain available, and feeds results back to the planning module. If an action fails, the system should enter a diagnostic flow rather than retry indefinitely.

    A simplified decision process can be represented as:

    1. Collect environment state
    2. ↓
    3. Identify assets and risks
    4. ↓
    5. Generate multiple candidate actions
    6. ↓
    7. Evaluate gains, costs, and impact scope
    8. ↓
    9. Execute low-risk actions or request approval
    10. ↓
    11. Validate results and update environment model
    12. ↓
    13. Continue planning, roll back, or escalate to human

    Why More Dynamic Evaluations Are Needed

    Static benchmarks still have value. They are suitable for measuring foundational knowledge, code comprehension, and point-in-time problem-solving. But if the goal is to build security agents for production environments, dynamic scenarios must be added, including:

    • Defenders will actively patch vulnerabilities;
    • Target services may restart or change configurations randomly;
    • Network access experiences latency and restrictions;
    • Credentials may expire or be rotated;
    • Attack actions trigger logs, alerts, and behavioral exposure;
    • Agents must allocate resources between attack and self-defense;
    • Some operations require approval and cannot be executed automatically without limits.

    Additionally, evaluation results should report confidence intervals, variance across scenarios, and failure types—not just a single average score. The value of 466 matches lies not only in producing a ranking, but in helping researchers analyze under which conditions systems fail: whether they cannot discover vulnerabilities, cannot complete exploitation, cannot respond to remediation, or lose points due to poor self-service maintenance.

    Implications for Enterprise Security

    This research demonstrates that as agents enter the cybersecurity domain, the evaluation standard is shifting from “can generate the correct answer” to “can persistently complete tasks under constraints.” Enterprises should avoid making decisions based on live demos or single-round tests.

    A more prudent approach includes: conducting red-team/blue-team tests in an isolated environment first; then gradually opening read-only observation, low-risk remediation, and limited automated patching permissions; establishing a complete audit trail so that all decisions are explainable, traceable, and reversible; and retaining human takeover mechanisms, especially for core data, identity permissions, and critical business services.

    Looking further ahead, agents may become an important component of security operations, but they will not replace foundational security engineering. Clear asset management, least privilege, timely patching, reliable backups, centralized logging, and incident response plans remain the foundation of any defense system. The value of agents lies in helping teams understand environments faster, detect risks earlier, and execute repetitive work within clearly defined boundaries.

    The true significance of dynamic attack-defense evaluation is not merely comparing which model ranks higher. It is about pushing the industry to redefine what “security intelligence” means. A mature security agent should be able to observe, reason, act, validate, and recover—all while balancing offensive gains against service stability, permission constraints, and operational responsibility. Only when these capabilities withstand sustained adversarial pressure and real engineering conditions can an agent qualify to enter production security environments.