Cybersecurity AI Agent Adversarial Evaluation: What 466 Battle Rounds Reveal

    This article examines a large-scale dynamic adversarial evaluation in which 466 agent-vs-agent matches were run across 12 frontier model and agent-framework combinations. It explains why cybersecurity assessment is shifting from static question-answering toward continuous offense-and-defense games, presents the evaluation design and results (including top scores around 78.6 and 73.8 with win rates of 0.83 and 0.86, and how attack capture rates fell to 52%, 37%, and 35% by round three once defenders began patching), and analyzes why dynamic adversarial testing is harder than static benchmarks. It also breaks down the key capabilities these agents must demonstrate, the role of the agent execution framework beyond the underlying model, how enterprises can adopt a continuous red-team/blue-team and evaluation workflow, a general cloud architecture for engineering agent security capabilities, and the governance risks that greater automation introduces.

    1. From Static Q&A to Dynamic Games: The Cybersecurity Evaluation Paradigm Is Shifting

    Historically, cybersecurity capability evaluation was often carried out in a “question-style” or “scan-style” manner: a piece of code, a service, or a known vulnerable scenario was presented, and a model or tool was asked to determine whether a risk existed and to suggest a fix. This approach is cost-controlled, easy to reproduce, and well suited to measuring baseline security knowledge. However, a real network environment is not a static exam paper; it is a dynamic system that changes continuously, with attackers and defenders responding to each other.

    Once attackers begin using automated agents, defenders must also contend with threats that are more frequent, more continuous, and more adaptive. An attack is no longer a one-time attempt but may pass through multiple stages: reconnaissance, verification, bypass, lateral movement, and persistence. Defense is no longer a one-time scan either; it requires real-time monitoring, rapid patching, isolation of affected assets, and a strong effort to maintain business continuity. As a result, cybersecurity evaluation is shifting from “can it solve the problem?” to “can it keep winning under adversarial conditions?”

    This shift raises a key question: is an agent that is strong at static question-answering necessarily capable in real offensive and defensive operations? A large-scale dynamic adversarial evaluation conducted by multiple research institutions offers an instructive answer. The evaluation was built around 466 agent matches, pitting several frontier model and agent execution framework combinations against one another in the same dynamic environment. Each agent had to act both as an attacker, trying to break through the opponent’s defenses, and as a defender and operator, protecting its own services and keeping them stable. This “combined offense and defense” design brings the evaluation much closer to real-world cybersecurity scenarios.

    2. The Evaluation Design Behind 466 Matches: Agents That Both Attack and Defend

    The core idea of this evaluation is to turn cybersecurity from a single task into a continuous game. Rather than focusing only on whether a model can generate exploit code or only on whether it can identify vulnerabilities, the evaluation placed attack, defense, operations, recovery, and scoring into a single closed loop.

    Specifically, the evaluators introduced 12 different frontier model and agent execution framework combinations. Each combination was not an isolated “question-answering machine” but an agent with some tool-calling capability. They executed real tasks in a controlled environment: the attacker needed to discover weaknesses in the target service, construct an exploitation path, and attempt to gain privileges or cause service disruption; the defender needed to monitor anomalous behavior, patch exposed surfaces, adjust strategies, and keep the business available as attacks evolved continuously.

    This design has several important characteristics.

    First, attack and defense occur simultaneously. The attacker does not face a static target but a defender that responds, patches, and adjusts strategy. The defender’s repair actions change the attack surface, forcing the attacker to re-plan its path.

    Second, agents must maintain their own operating state. A defender cannot simply “plug the vulnerability”; it must also keep the service running. This means an agent must understand not only security but also operations, resource scheduling, fault recovery, and business continuity.

    Third, scoring is not a single point of correctness but a comprehensive game outcome. The evaluation focuses on an agent’s expected score, win rate, stability, and capability distribution across multiple rounds of adversarial play. Such metrics are closer to real security operations than traditional multiple-choice questions.

    Fourth, the evaluation was large enough to be statistically meaningful. The final run completed 22 batches and 466 valid matches, giving the results good value for layered observation. The gaps between different model and execution framework combinations were no longer mere random fluctuation but revealed a relatively clear capability gradient.

    This evaluation approach shows that an agent’s cybersecurity capability is not a single knowledge-based Q&A ability but a comprehensive one: it must understand vulnerability principles and be able to execute multi-step operations; it must be able to attack and to defend; it must combine strategic reasoning with tool coordination and risk control.

    3. Evaluation Results: Clear Top-Tier Stratification, and Static Advantage Does Not Equal Combat Advantage

    In the final results, the top agent combinations showed a clear advantage. Some leading combinations reached final scores of about 78.6 and 73.8, with win rates of 0.83 and 0.86 respectively, while the highest win rate among the remaining participants was only 0.59. At the same time, a representative open-source agent combination scored about 68.5, with the remaining models distributed between 56.3 and 67.1.

    This set of data sends a clear signal: in a dynamic adversarial environment, the capability gaps between agents are significantly amplified. During static question-answering, different models may all produce seemingly reasonable vulnerability analyses or fix recommendations; but once continuous offense and defense begin, the differences show up in multi-round decision-making, tool calling, error recovery, defensive response, and resource management.

    More interesting is a comparison the research team made between static and dynamic conditions. In single-player mode, the target was undefended and each question allowed two attempts. Of 40 questions, 30 could be fully solved by all three representative agent groups. This shows that under idealized, non-adversarial conditions, these agents have strong baseline problem-solving ability.

    However, when the same 30 questions were placed into two-way adversarial matches, the results changed markedly. Defenders began actively patching vulnerabilities, and the attackers’ capture rate dropped round by round. By the third round, the offensive capture rates of the three agent groups had fallen to 52%, 37%, and 35% respectively. In other words, an attack that looked nearly guaranteed in a single-machine environment was neutralized or missed by defenders in roughly half to more than sixty percent of cases once dynamic gameplay began.

    The takeaway for enterprises and researchers is direct: static benchmark questions can no longer fully measure an agent’s security capability. Real attack environments have feedback, repair, and adaptation mechanisms, and any successful exploitation may be only a brief window. The true value of an agent lies not in whether it can solve a single problem, but in whether it can keep making effective decisions as the environment changes, defenders respond, and business constraints apply.

    4. Why Dynamic Adversarial Testing Is Harder: Agents Must Face a Continuously Changing Environment

    The core reason dynamic adversarial testing is harder than static evaluation is that the environment state is no longer fixed. In traditional evaluation, the target vulnerability, inputs and outputs, and success conditions are usually clear; but in real offense and defense, system state changes continuously.

    First, there is the shrinking attack window created by defender patching. The attacker may have just found an exploitable path when the defender has already completed a patch, adjusted rules, or isolated the relevant components. The attacker must re-probe, re-verify, and even abandon the original route. This places higher demands on an agent’s planning ability.

    Second, there are business continuity constraints. A defender cannot simply shut down all services to guarantee security. Real scenarios require defenders to balance security and service availability. Agents must understand which operations affect the business, which can be executed safely, and which require human confirmation. This turns cybersecurity from a purely technical contest of offense and defense into a comprehensive decision-making problem involving operations and risk trade-offs.

    Third, the tool-calling chain is longer. A complete offense-and-defense workflow may involve asset discovery, port scanning, vulnerability verification, privilege escalation, lateral movement, log cleanup, and monitoring bypass. An agent must not only call tools but also revise its plan based on intermediate results. Any single failed step can render the entire attack chain ineffective.

    Finally, the evaluation metrics focus more on net gain. A defender’s success is not just “not being breached”; it also includes timely detection, rapid response, reducing the scope of impact, and restoring the business. An attacker’s success is not just “getting into the system”; it also considers whether the attack causes service interruption, whether it triggers alerts, and whether the effect can be maintained. Such metrics are closer to real security operations.

    Dynamic adversarial evaluation, therefore, is really examining an agent’s comprehensive decision-making ability under uncertainty. The model is no longer just a knowledge base but an actor; no longer just an answerer but a task executor.

    5. Key Capabilities Revealed: From Vulnerability Understanding to Sustained Action

    Analyzing the 466 matches reveals several categories of key capability. These capabilities are important for understanding future agent-based security systems.

    5.1 Vulnerability Understanding and Exploitation Orchestration

    The attacker must understand the cause of a vulnerability, its trigger conditions, its impact scope, and its exploitation path. But vulnerability knowledge alone is not enough; an agent must also be able to chain multiple vulnerabilities into an attack chain. For example, an information disclosure vulnerability may be only the starting point, and the real impact depends on whether it can later be combined with permission misconfiguration, credential leakage, or exposed services to form a complete exploitation path.

    This kind of capability requires strong contextual reasoning rather than mechanical rule matching. The agent must judge which information is worth pursuing, which actions might trigger defenses, and which paths offer a better risk-reward ratio.

    5.2 Tool Calling and Execution Stability

    Real offense and defense are rarely completed through a single natural-language output. Agents must call tools for scanning, detection, verification, patching, and monitoring. Whether tool calls are accurate, whether parameters are safe, whether failures can be retried, and whether results can be parsed all affect the final performance.

    In the evaluation, tool-calling stability often determined whether an agent could turn ideas into action. An agent that can reliably call tools may achieve better results through execution framework optimization, even if the underlying model is not the strongest.

    5.3 Defensive Response and Fix Verification

    A defender must not only detect attacks but also verify whether fixes are effective. An agent must judge whether a patch truly eliminates the risk, whether it introduces new compatibility issues, whether a rollback is needed, and whether relevant modules must be notified. This requires a defensive agent to have both security analysis and operational verification abilities.

    In multi-round adversarial play, the speed and quality of a defender’s fixes directly affect the attacker’s capture rate. The sharp drop in capture rate by the third round shows that continuous defender patching can effectively compress the attack space.

    5.4 Business Continuity and Risk Control

    An agent’s actions cannot be divorced from business objectives. If a defender frequently restarts services, blocks traffic, or shuts down core interfaces in order to intercept attacks, short-term security metrics may improve, but business availability will decline. Agents must therefore learn to weigh security strength against business impact.

    This capability is especially important in real enterprises. Security teams cannot pursue only “zero vulnerabilities”; they must also consider change windows, rollback mechanisms, service levels, and user experience.

    5.5 Resistance to Disturbance and Robustness

    Inputs in a dynamic environment often contain noise, misleading information, and adversarial content. Attackers may deliberately plant false leads, and defenders may trigger large volumes of alerts. If an agent is easily misled by surface signals, it can fall into inefficient loops.

    The evaluation results show that agents with stable context management and evidence-chain reasoning are more likely to keep taking effective actions in complex adversarial conditions.

    6. Beyond the Model: The Agent Execution Framework Determines Real-World Effectiveness

    Another important lesson from this evaluation is that model capability is not the only variable. The same model placed in different execution frameworks can perform quite differently. The agent execution framework is responsible for task decomposition, state management, tool calling, error recovery, permission control, and result aggregation.

    An agent system can be understood as a combination of “model brain + execution framework + tool environment.” The model handles reasoning and decision-making, the execution framework turns decisions into reliable actions, and the tool environment interacts with the real system. If the execution framework is poorly designed, even a model with strong reasoning ability will struggle to complete complex tasks reliably.

    An ideal agent execution framework typically needs the following capabilities.

    First, task decomposition. Complex security tasks need to be broken into steps such as reconnaissance, analysis, decision, execution, and verification, rather than generating a final conclusion all at once.

    Second, state management. The agent needs to preserve current progress, actions already taken, reasons for failure, and hypotheses awaiting verification, to avoid repeated operations or lost context.

    Third, tool orchestration. Different tools have different output formats, risk levels, and permission requirements. The execution framework must schedule them uniformly and validate parameters and results.

    Fourth, failure recovery. Network requests may time out, services may behave abnormally, and tools may return empty results. The agent needs to be able to retry, switch paths, or escalate to a human.

    Fifth, security and permission control. An agent should not have overly broad permissions. The execution framework should limit accessible resources, log operations, and require confirmation for high-risk actions.

    Sixth, observability. Every round of attack or defense should be traceable, replayable, and auditable. Only then can evaluation results be reproduced and security incidents be attributed.

    Schematically, the evaluation flow can be abstracted into the following logic:

    1. for battle in range(466):
    2. env = CyberRange()
    3. agent = SecurityAgent(
    4. model=MODEL_CONFIG,
    5. tools=["scan", "detect", "patch", "monitor", "isolate"],
    6. policy=ACTION_POLICY,
    7. )
    8. result = env.run_battle(agent)
    9. metrics.append({
    10. "score": result.score,
    11. "win_rate": result.win_rate,
    12. "attack_success": result.attack_success,
    13. "defense_success": result.defense_success,
    14. })

    This pseudocode does not refer to any specific product; it expresses a general evaluation approach: place an agent in a controlled environment, collect metrics through multiple rounds of action, and then statistically analyze and compare the results.

    7. How Enterprises Can Learn From This: From a Single Evaluation to Continuous Security Operations

    For enterprises, the value of these 466 agent matches lies not only in watching how a particular model scores, but more importantly in borrowing its evaluation methods and operational thinking. Cybersecurity agents are moving from demonstration tools toward production systems, and enterprises need to build their own dynamic evaluation and continuous exercise mechanisms.

    7.1 Define the Evaluation Scenario

    An enterprise must first clarify the problem the agent is meant to solve. Is it external attack simulation, internal lateral movement detection, vulnerability fix verification, security alert analysis, or a security operations assistant? Different scenarios demand different agent capabilities. Attack simulation places more weight on vulnerability understanding and exploitation orchestration; defensive response places more weight on detection accuracy and handling speed; operations and remediation emphasize change safety and business impact assessment.

    7.2 Build an Isolated Range

    Before executing high-risk operations, agents must be validated in an isolated environment. Enterprises can use container platforms, virtual networks, traffic mirroring, and controlled test accounts to build a range environment isolated from production. The range should be able to simulate typical business systems, common vulnerabilities, permission misconfigurations, and anomalous behavior, and support repeatable resets.

    7.3 Establish a Red-Team/Blue-Team Closed Loop

    Traditional red-team/blue-team exercises rely heavily on manual work. Agents can take on repetitive reconnaissance, preliminary analysis, and automated verification, freeing the security team to focus on high-risk decisions. A reasonable loop is: an attack agent generates attack hypotheses, a defense agent performs detection and response, the evaluation system records actions and results, and finally the security team reviews the strategy.

    7.4 Build a Multi-Dimensional Metric System

    A single “breached or not” indicator is not enough to evaluate an agent. Enterprises can combine metrics such as attack success rate, mean time to detect, mean time to respond, fix time, business interruption rate, false positive rate, tool-calling success rate, and human intervention rate. This avoids pursuing only attack success while ignoring the quality of security operations.

    7.5 Keep Human Approval Gates

    Agents can automate many low-risk actions, but high-risk operations must retain human approval. For example, deleting data, modifying core configuration, blocking production traffic, or bulk rolling back versions should be subject to permission checks, two-person confirmation, or circuit-breaker mechanisms. Automation does not mean removing humans; agents should enhance team capability rather than replace key controls.

    8. A General Cloud Architecture: How to Engineer Agent Security Capabilities

    If an enterprise wants to build agent security evaluation or security operations capability in the cloud, it can adopt a combination of general cloud components without relying on any specific vendor product. The key is to connect agent execution, data collection, task scheduling, log auditing, and monitoring and alerting.

    A typical architecture can include the following layers.

    8.1 Execution Isolation Layer

    Use a container platform or isolated compute environment to run agents and range services. Each task should use an independent resource quota, network policy, and lifecycle management to avoid cross-contamination. For high-risk evaluations, use temporary environments and destroy them when finished.

    8.2 Task Scheduling Layer

    Distribute tasks through message queues or workflow scheduling components. Each offense-and-defense batch, each agent action, and each tool call can be recorded as a task unit. The scheduling layer should support priority, concurrency control, failure retry, and timeout handling.

    8.3 Data Storage Layer

    The evaluation process generates large volumes of logs, request and response data, attack payloads, alert records, and report files. Object storage can be used to retain long-term evidence and evaluation reports, and a log service can be used to centrally search key events. All data should have access controls and retention policies.

    8.4 Monitoring and Alerting Layer

    Monitor agent operating state, tool-calling success rates, resource consumption, anomalous behavior, and security incidents. Monitoring serves not only evaluation but also behavioral auditing of agents in production. Abnormal metrics should trigger alerts and support human intervention.

    8.5 Identity and Permission Layer

    Agents need to access assets and tools but should not hold excessive privileges. Enterprises can apply the principle of least privilege, assigning different roles to different agents and restricting the scope of accessible resources. Sensitive credentials should be managed through a general key management capability, with rotation and auditing mechanisms.

    The value of this architecture is that it turns agent security from a one-time experiment into sustainable engineering. Enterprises can periodically replay historical adversarial data, compare the performance of different agent versions, and feed evaluation results into security operations to form a continuous regression mechanism.

    9. Security Governance: New Risks from Automated Adversarial Capabilities

    Agent-based offense-and-defense evaluation also reminds us that the stronger the automation, the higher the governance requirements. If an agent is misconfigured or granted excessive permissions, it may introduce new security risks.

    First is the risk of misoperation. An agent may mistakenly apply a test action to the production environment, or modify a critical configuration without approval. This can be reduced through environment isolation, operation allowlists, and human confirmation.

    Second is the risk of data leakage. Attack agents need to read logs, configurations, and vulnerability information, which may contain sensitive content. Enterprises need to apply tiered control over the data an agent can access, preventing it from touching sensitive information beyond what the task requires.

    Third is prompt injection and adversarial misdirection. When processing external content, an agent may be influenced by malicious input and execute incorrect instructions. For example, web content, log text, and API responses may all contain misleading descriptions. The instruction layer, data layer, and tool layer of an agent should be separated, and critical actions must not be triggered solely by natural-language content.

    Finally, there are supply-chain and tool-dependency risks. When calling external tools, an agent must confirm the tool’s source, permissions, and output trustworthiness. Enterprises should establish tool onboarding standards and avoid using unaudited plugins or scripts.

    Agent security governance should therefore include mechanisms such as least privilege, environment isolation, operation logging, result review, and emergency circuit-breaking. Only by unleashing automation within controllable boundaries can automated offense and defense deliver sustainable value.

    The trend revealed by the 466 matches is clear: cybersecurity capability is shifting from static knowledge competition toward dynamic game competition. Agents not only answer questions but also execute tasks; not only propose fix recommendations but also actually verify fix effectiveness; not only imitate attacks but also adjust strategies based on defensive changes.

    In the future, a cybersecurity system may simultaneously possess capabilities for attack simulation, defensive response, vulnerability remediation, and strategy optimization. Security teams can use agents for continuous red-team/blue-team exercises: finding risks before a business goes live, monitoring anomalies while it runs, and quickly reviewing events after they occur. Agents will not completely replace security experts, but they will significantly raise the degree of automation and the speed of response in security work.

    For enterprises, what truly matters is not the single performance of any one model or tool, but whether they can build an evaluable, governable, and iterable agent security system. This system includes model capability, execution framework, tool environment, permission control, data quality, evaluation metrics, and organizational processes. Only when these elements work together can agents deliver stable value in real network environments.

    In summary, dynamic adversarial evaluation acts like a mirror: it reflects the limitations of static evaluation and reveals the true boundaries of agent security capability. Cybersecurity is entering a new phase of continuous evolution and two-way competition. Whoever establishes scientific evaluation, risk control, and engineered operations earlier will hold the initiative in the future competition for intelligent security.