From Operating to Completing: Capability Boundaries of AI Agents on Game Tasks

What a Game Experiment Reveals

Having an AI agent operate a classic role-playing game looks like an entertaining demo, but it is really a comprehensive technical test. A game environment usually includes map exploration, character movement, resource management, combat decisions, quest triggers, non-player character interaction, and multi-stage goal progression. The agent must not only “understand the screen.” It must continually judge its current state, decide what to do next, and check whether earlier actions produced the expected result.

In a public livestream-style experiment, researchers had a conversational AI complete game tasks through screen observation and input control. At the start, many viewers offered suggestions in a public discussion area and watched whether the agent could advance on its own. The experiment quickly exposed an important fact: being able to perform some operations does not mean the agent can reliably finish the game.

It may know how to move a character, select a menu, or start a battle, yet still fail to organize those local actions into a continuous, reliable task flow. In scenes that require exploration, memory, and reasoning, the system may loop around the same area, repeat actions, ignore key clues, or stall for a long time.

“Can Operate” Is Not the Same as “Can Complete the Task”

From a technical perspective, a game-playing agent’s capability can be divided into at least four levels.

The first level is perception. The agent must extract information from images, text prompts, interface elements, and character positions. For example, it needs to recognize roads, obstacles, entrances, enemies, items, and interactive objects.

The second level is action execution. The agent must turn decisions into concrete inputs such as directional keys, confirm, cancel, and menu selection. The actions themselves are often not complex, but input timing, duration, and combination order affect the outcome.

The third level is local decision-making. The agent must answer “what should I do next?”—for example, which direction to move, whether to enter a building, whether to switch characters, or whether to save resources.

The fourth level is long-term planning. The system must understand the relationship between the current action and the final goal, then confirm after multiple operations whether the task has actually progressed.

Many models perform reasonably well at the first two levels and can also offer seemingly reasonable suggestions at the third. Once long-term planning is required, however, a capability gap often appears. A locally correct action does not guarantee a correct overall path. Moving left may be a reasonable action, but without a clear goal, map state, and fallback condition, the agent may simply keep executing in the wrong direction.

Technical Causes Behind Typical Failures

1. Incomplete state memory

Game environments are strongly sequential. Whether a switch has already been triggered, whether a road has been explored, and whether an item has already been obtained all affect later decisions. If the agent judges only from the current screen and does not maintain structured state, it can easily forget what happened before.

For example, the system may pass through an area and, minutes later, explore it again because the screen looks similar. In a more serious case, it has already tried a route, confirmed that it does not work, failed to record why, and then tries the same plan again.

Effective state records should include at least:

• the current area and approximate coordinates;
• the current task goal and its completion conditions;
• visited locations and verified routes;
• key items, resources, and character status;
• the results of recent actions;
• strategies and trigger conditions already confirmed to be ineffective.

This information should not all be stored as long natural-language text. It should be structured as much as possible. Structured state is easier to retrieve, update, and verify, and it reduces the reasoning burden caused by ever-growing context.

2. Lack of a map and spatial model

Judging direction from the current screenshot alone is reactive control. It can handle short-distance movement, but it is poorly suited to navigation that requires many steps.

In a complex map, the agent needs some internal spatial model. At minimum, it should know which areas connect, which positions are blocked, which entrances can be entered, and in which direction a target lies relative to the current area. Without map memory, the system treats every frame as an independent problem and repeatedly gets lost in similar scenes.

In engineering practice, an incremental map representation can treat areas as nodes and passable paths as edges, then update the graph after each exploration. When a new obstacle or an impassable path is found, the existing map is revised. Even if a complete map cannot be built, a simplified “area–entrance–target” relationship can significantly reduce repeated exploration.

3. Insufficient goal decomposition

“Complete the main quest” is usually not a directly executable action. It consists of multiple subgoals, such as:

1. find the target area;
2. obtain the item required to enter that area;
3. solve the mechanism or obstacle blocking the path;
4. interact with the specified object;
5. complete the battle or event;
6. return and confirm that the quest status has updated.

Without goal decomposition, the agent may look for a direct action that does not exist on the current screen, or loop endlessly on one local step.

Decomposition also needs explicit completion criteria. “Enter the building,” for example, should not mean only that the character is near the entrance; the scene must actually change. “Complete the battle” should not be judged from a single attack; enemy status, rewards, and the quest log should also be checked for changes.

4. Weak error recovery

Errors are unavoidable in a real environment. The agent may misjudge a path, trigger the wrong menu, consume unnecessary resources, or produce a result different from what was expected because of input delay. The key to a mature system is not never making mistakes. It is recognizing mistakes quickly and recovering.

Many experimental agents, however, lack a clear failure-detection mechanism. They continue the original plan instead of asking whether the current state still matches expectations. When an action produces no state change after several consecutive attempts, the system should trigger exception handling, such as:

• returning to the most recent safe position;
• reading the current screen again;
• checking whether a different scene has been entered;
• changing the route or interaction method;
• pausing and requesting human confirmation.

Without these mechanisms, a simple misjudgment can expand into hours—or even longer—of ineffective execution.

Why a Simple Obstacle Can Cause a Long Stall

Human players often use common sense to eliminate possibilities quickly. When a road is blocked by a tree, a door, or a mechanism, a player naturally thinks, “I need the matching tool or trigger condition.” For an agent, that judgment involves several implicit reasoning steps:

• recognize what the obstacle is;
• judge whether the obstacle is interactive;
• infer which ability or item is required;
• recall whether a related clue was seen earlier;
• decide whether to leave the current location or keep trying;
• return to the original location and verify the result after finding the trigger condition.

If the model focuses only on “which button to press next” and does not build a causal chain, it may keep moving in front of the obstacle or even conclude that the entrance location was identified incorrectly. On the surface, this looks like a very basic mistake. From a system-design perspective, it shows that perception, memory, planning, and verification have not formed a closed loop.

Metrics for Evaluating Agents

Simply counting whether the game was cleared does not fully measure an agent’s capability. A more reasonable evaluation should cover the following metrics.

First, task success rate: the proportion of goals completed within the specified resources and time.

Second, effective action rate: the share of all actions that actually change the environment state. Repeated movement, ineffective clicks, and repeatedly opening menus lower this metric.

Third, recovery ability: the time needed to recognize an error, the number of steps needed to return to a valid path, and whether the same type of error is repeated.

Fourth, exploration efficiency: the number of actions consumed to discover new areas, key items, or task clues.

Fifth, resource management: whether limited items are used reasonably, and whether short-term gains interfere with later goals.

Sixth, planning stability: whether multiple runs under the same initial conditions produce similar results. Completely different routes every time indicate that internal state modeling is still unstable.

A More Reliable Agent Control Architecture

The system can be divided into five modules.

The perception module reads the screen, text, and input feedback, and extracts character position, obstacles, target objects, and scene changes.

The state module maintains long-term memory, including the map, tasks, resources, and action history, and updates that memory with new observations.

The planning module splits long-term goals into verifiable short-term subgoals and sets completion and failure conditions for each subgoal.

The execution module turns the plan into concrete actions and controls their pace, avoiding error accumulation from outputting an overly long sequence at once.

The evaluation module compares the expected state with the actual state. If an action does not produce the expected result, it pauses the original plan, perceives the environment again, and selects a recovery strategy.

The core of this architecture is an “execute–observe–verify–correct” loop, not one-shot generation of a very long action sequence. For environments with clear feedback and complex task steps, a short closed loop is usually more reliable than long-sequence automatic execution.

Practical Engineering Improvements

First, set a maximum number of attempts for each action. If the same action is executed several times in a row without progress, the system must be forced to change strategy.

Second, save snapshots of key checkpoints. A snapshot does not have to be a complete game save; it can be a state summary that includes the current location, task stage, and resources. When a serious deviation occurs, the system can quickly return to a controllable state.

Third, separate exploration mode from execution mode. During exploration, unknown routes may be tried, but the results must be recorded. During execution, verified paths should be preferred so the agent does not take the same risks again.

Uncertainty management should also be introduced. When the agent cannot tell whether an object is a key clue, it should not make a high-cost decision immediately. It should choose a low-risk observation, interaction, or fallback action.

For scenarios that require human participation, define clear handoff points. For example, when several consecutive rounds produce no state change, resources fall below a safety threshold, or an unexplained scene transition occurs, the system should actively request confirmation instead of continuing to consume time.

From Game Experiments to Real Applications

A game is only an observable, repeatable test environment. Similar problems also appear in office automation, software testing, web operation, device control, and business-process execution.

In these settings, an agent can likewise be locally correct while failing the overall goal. It may fill out a form but miss the submission confirmation, run a command but fail to verify the output, or call a tool and then continue using invalid parameters after an exception.

Evaluating an agent therefore cannot stop at whether it can complete one demonstration. It must also consider long-term memory, goal decomposition, result verification, exception recovery, and risk control. A genuinely usable system should be able to explain its current state, state the next plan, and adjust promptly when that plan fails.

Conclusion

The real value of letting an agent operate a complex game is not showing that it can move a character or complete a simple action. It is observing the agent’s overall capability on long-running tasks. Seemingly absurd circling, repetition, and stalling reveal the engineering gap that current agents still need to cross in moving from “action generators” to “reliable executors.”

Future agent design should not focus only on larger models or more actions. It should build more robust state representation, spatial memory, goal planning, and error recovery. Only when a system can continually understand its environment, verify results, and correct itself after failure can artificial intelligence truly be said to complete complex tasks.