From Beating a Game to Evaluating AGI: How to Assess Autonomous Agent Capabilities

What a Game Experiment Reveals About Agent Capabilities

Having an AI model autonomously complete a 3D puzzle game may seem like an entertaining experiment, but it actually serves as a comprehensive test of multiple technical capabilities. The model must recognize environmental states from the visuals, understand goals and rules, plan subsequent actions, execute operations via external control tools, and correct its judgments based on new feedback.

In a public experiment, researchers enabled a model to control a game through a tool interface. The game paused whenever the model performed inference; subsequently, it decided the next step based on screenshots and character position. The entire process lasted about 24 hours, generated 3,336 tool calls, and incurred approximately $571 in API costs. The researchers also created a two-hour edited video to facilitate observation of key processes.

The value of such experiments lies not in proving that the model can “play games like a human,” but in placing agent capabilities within an observable and repeatable environment. Because game states are clear, action spaces are relatively limited, and success criteria are intuitive, games are well-suited for testing whether a model can form a closed loop of “perception, decision-making, execution, and feedback.”

From Perception to Execution: The Task Chain is Not Simple

Autonomously completing game tasks typically requires the coordination of multiple stages. A failure in any single stage can stall progress.

1. State Recognition and Information Compression

Models often receive raw information such as images and coordinates. They must identify key elements in the scene, distinguish interactable objects from the background, infer the character’s region, and evaluate the gap between the current state and the objective. Occlusions, perspective changes, or blurred details in the image can lead to misjudgments.

Furthermore, environmental information cannot be retained indefinitely in full at every inference step. The agent must compress observations into useful state summaries for subsequent decisions, such as recording explored areas, mechanism states, available paths, and failed attempts. If the summary is too brief, key conditions may be lost; if too detailed, it increases the reasoning burden.

2. Planning and Decomposition

Puzzle tasks are rarely completed with a single action; they usually consist of a series of preconditions. The model must determine which actions can be executed directly, which goals require preparatory steps, and how to recover if an action fails.

A robust planning approach typically breaks tasks into short stages: define the current local goal, select a few actions, observe the results, and replan. Compared to generating a long sequence of actions at once, this closed-loop approach makes it easier to detect deviations and recover from unexpected states.

3. Tool Invocation and Result Verification

An agent must not only propose actions but also correctly pass them to external control tools and verify whether the actions take effect as expected. If there are delays in tool invocation, uncertainties in execution results, or incomplete feedback, the model might continue reasoning based on outdated states.

Therefore, evaluating tool usage should go beyond counting call frequencies. It should focus on whether the calls are effective, whether actions produce expected changes, whether the agent can identify causes of failure, and whether it proactively re-observes when the state is uncertain.

Why “Beating a Game” Does Not Equate to General Intelligence

While success in a single task demonstrates that a system possesses certain comprehensive capabilities within a specific environment, it is insufficient to prove broad and stable general capabilities. First, the task rules, scene structure, and operation methods might inherently favor the model. Second, experimental results can be influenced by manual configurations, prompting methods, environmental observability, and tool interface design. If these factors are not fully disclosed, it is difficult to determine whether success stems from the model itself or from the conveniences provided by the environment.

Additionally, a long runtime does not automatically imply high autonomy. A system might achieve its goal through massive trial and error while consuming excessive calls, time, and manual maintenance. Evaluations should distinguish between “whether it ultimately succeeded” and “at what cost,” examining whether human intervention was required, whether unrecoverable errors occurred, and whether existing strategies can be transferred to new scenarios.

More importantly, the scope of game tasks is limited. Good performance in a structured environment does not mean a model can reliably handle ambiguous goals, complex constraints, and real-world risks in open-world scenarios. General intelligence is a concept without unified boundaries. Without clear definitions and measurable standards, declaring its realization risks misinterpreting localized capability progress as a holistic capability leap.

How to More Rigorously Evaluate Autonomous Agents

To determine whether an agent has truly made progress, evaluation design must shift from “showcasing a single success” to “systematically measuring capabilities.”

1. Task Diversity. Beyond a single level, testing should cover different rules, map structures, and difficulty levels, incorporating variations the model has not seen before. This helps observe whether the system has learned transferable methods rather than memorizing fixed procedures.

2. Process Quality Recording. In addition to success rates, evaluations should track task completion time, tool call volume, proportion of invalid operations, failure recovery capabilities, and the number of human interventions. Process metrics reveal efficiency differences behind identical success results and help identify bottlenecks.

3. Controlling Experimental Conditions. Evaluations should clarify what observational information the model receives, what actions it can execute, whether retries are allowed, and whether the environment pauses during inference. Controlled experiments should also keep inputs, tool permissions, and task difficulty as consistent as possible to avoid mistaking environmental advantages for model capabilities.

4. Evaluating Reliability and Safety. Agents need to handle not just “whether it can be completed,” but also “when it should stop.” When goals are ambiguous, states conflict, or actions might cause irreversible consequences, the system should recognize uncertainty, request clarification, or choose a conservative strategy. This is particularly crucial for agents connected to real-world software and physical devices.

“AGI Has Arrived” Requires Answering the Definition Question First

Discussions surrounding Artificial General Intelligence (AGI) often yield conflicting conclusions due to differing definitions. Some emphasize cross-task learning, others focus on reasoning and planning, and still others prioritize autonomy, reliability, and adaptability in real-world environments. Without clear evaluation criteria beforehand, a single demonstration can hardly support universal conclusions.

A more practical approach is to decompose broad judgments into a set of testable dimensions: Can the system complete unfamiliar tasks? Can it learn with limited feedback? Can it transfer knowledge across domains? Can it maintain goal consistency over long periods? Can it identify its own errors? Can it operate stably under controllable costs and risks? Different dimensions can be measured separately, gradually accumulating evidence through public benchmarks, repeated experiments, and independent peer reviews.

Implications for Developers and Enterprises

These experiments indicate that when combined with external tools, models can handle longer and more complex tasks than single-turn Q&A. However, to apply these capabilities to actual business operations, a reliable execution loop must be established: define authorization scopes, restrict high-risk actions, preserve critical states and operation logs, design recovery mechanisms for failures and timeouts, incorporate human confirmation before high-impact operations, and continuously monitor task success rates, call costs, and anomalous behaviors.

Therefore, judging the maturity of an autonomous agent should not rely solely on an impressive demo or a blanket statement that “general intelligence has been achieved.” The more valuable questions are: Under what conditions is it effective? Is it robust when facing changes? Can it recover from failures? And are the time, cost, and risk required to complete tasks acceptable? Incorporating these questions into evaluations is the only way to translate exciting experimental progress into verifiable, reusable engineering capabilities.