Real-Time Interaction Benchmarks: Why LLMs Miss Critical Action Windows

1. The Core Challenge: Models Think While the World Keeps Moving

Traditional question-answering and offline tasks usually give models time to reason over relatively stable inputs. Once a request is submitted, the external environment generally remains unchanged. The model primarily needs to balance response quality against latency.

Real-time interaction is fundamentally different. Instead of a static image, fixed text, or one-time request, the model faces an environment that continues to evolve. Inventory on a page may change unexpectedly, other participants may update trading prices, a countdown keeps running, and a game character may continue moving while the model generates its next instruction.

The central question is therefore no longer simply whether the model can produce a correct answer. It must also determine:

• Whether its latest observations are still valid.
• Whether it has identified a genuine opportunity to act.
• Whether reasoning is taking too long.
• Whether a generated action will still be appropriate when executed.
• Whether waiting is a reasonable strategy when no action is currently available.

Real-time agents must solve both reasoning and control problems. They need to understand their environment, manage time, estimate risk, and schedule actions appropriately.

2. Three Core Capabilities: Perception, Decision-Making, and Action

Real-Time Perception: Detecting Change, Not Just Reading the Screen

Real-time perception involves more than taking periodic screenshots and recognizing text. An effective perception module needs to answer at least four questions:

1. What is the environment’s current state?
2. What has changed since the previous observation?
3. Which information is now stale?
4. Is there an actionable opportunity right now?

In a time-limited purchase task, for example, the model must distinguish among several states: purchasing has not opened, purchasing is open but inventory is insufficient, the item remains available, or the purchase control is no longer valid. Detecting a button on the page does not establish that clicking it will work.

An implementation can attach a timestamp, state version, and estimated expiration time to each observation:

Observation {
    state: current environment state,
    observed_at: observation timestamp,
    version: state version,
    valid_until: estimated expiration time,
    confidence: recognition confidence
}

Before executing an action, the system should recheck critical state rather than unconditionally relying on an earlier observation. High-risk actions can use a short observe–confirm–execute loop to reduce failures caused by stale information.

Real-Time Reasoning: Deciding How Long to Think

In static tasks, additional reasoning often improves answer quality. In real-time environments, however, it also carries an opportunity cost. A model comparing several options may miss the only valid action window.

The reasoning module therefore faces a dynamic stopping problem: does the expected benefit of further thinking outweigh the potential loss caused by delay?

A rough expression for decision value is:

ExpectedValue =
    P(action remains valid) × P(action succeeds) × reward
    - execution_cost
    - delay_risk

Here, P(action remains valid) declines over time, while delay_risk depends on how quickly the environment changes and how much of the action window remains. When the window is about to close, an action with a sufficiently high probability of success may be preferable to further searching, even if it is not theoretically optimal.

This calls for layered reasoning:

• Fast layer: Detect urgent events and determine whether immediate action is necessary.
• Decision layer: Compare candidate actions within the available time.
• Review layer: Analyze execution results and update the strategy for subsequent steps.

This design is better suited to real-time settings than requiring a complete, lengthy reasoning process at every step.

Real-Time Action: Correctness Is Not Enough

An action module must consider not only what to do, but also when to do it and whether the action will still be valid at execution time.

In countdown-based tasks, acting too early may trigger an incorrect state, while acting too late may miss the opportunity. In a continuously moving game environment, even a correct turn command may arrive too late to avoid an obstacle.

An action should therefore include at least the following information:

Action {
    type: action type,
    parameters: action parameters,
    issued_at: issue timestamp,
    deadline: latest allowed execution time,
    precondition: conditions required before execution,
    fallback: alternative strategy if conditions are not met
}

The executor should check preconditions immediately before submitting the action. For example, it should confirm that a page still accepts submissions before placing an order, verify that a character’s position has not changed irreversibly before issuing a movement command, and check that the target has not been replaced by another event before clicking a confirmation button.

3. What a Real-Time Benchmark Must Test

A useful real-time interaction benchmark should measure more than final task completion. It should also capture how a model behaves under time pressure.

Continuously Changing Environments

The test environment must evolve independently. Changes may come from countdowns, random events, other participants, resource updates, or automatic character movement.

If the environment changes only after receiving model input, the evaluation remains closer to a conventional turn-based task than a real-time interaction test.

Limited Action Windows

Tasks should define clear windows of opportunity. For example:

• A resource is available only briefly.
• An operation must finish before a countdown expires.
• A route is accessible only in a particular state.
• A target moves continuously, making static plans quickly obsolete.

In these tasks, execution timing directly affects the outcome. Early, late, and repeated actions may produce different results.

Coverage Across Task Types

To avoid testing adaptation to a single interface, benchmarks can cover several general scenarios:

• Travel and scheduling: Select feasible options based on live conditions.
• Trading and purchasing: Complete operations as prices, inventory, or eligibility change.
• Service workflows: Maintain progress as page states change.
• Safety control: Take protective action promptly when unexpected events occur.
• Reaction-based games: Continuously track positions and provide low-latency control.
• Puzzles and strategy: Balance search depth against action speed as the state evolves.
• Simulation tasks: Operate under combined resource, rule, and time constraints.

Different tasks can emphasize reaction speed, state memory, planning, or risk control, producing a more complete evaluation.

4. Environment Clocks: Does Reasoning Time Count?

A critical benchmark design choice is how to account for the time a model spends reasoning. Two common approaches are real-time and paused-clock modes.

Real-Time Clock Mode

In real-time clock mode, the environment advances for as long as the model takes to generate a response. If the model spends more time analyzing a situation, countdowns, character positions, and resource states continue to change.

This mode most closely resembles practical applications and exposes the actual cost of latency. However, it also places greater demands on the execution environment, time synchronization, and experimental reproducibility.

Paused-Clock Mode

In paused-clock mode, the environment freezes while the model reasons and resumes after it finishes producing its output.

This approach helps isolate decision-making ability and reduces interference from factors such as network latency and differences in inference hardware. However, it cannot answer a crucial question: would the model still catch the opportunity if its thinking time counted?

Why Both Modes Matter

The two modes are complementary rather than interchangeable:

• Paused-clock mode is better suited to evaluating understanding, planning, and action selection.
• Real-time clock mode is better suited to evaluating end-to-end latency, scheduling, and timing control.

If a model performs well with a paused clock but degrades substantially with a real-time clock, it may have strong planning ability but lack efficient perception, fast decision-making, or low-latency execution.

5. Recommended Evaluation Metrics

Task success rate remains important, but a real-time benchmark should also measure the following dimensions.

Opportunity Capture Rate

Measure the proportion of valid opportunities the model detects and successfully uses. Tracking detection and execution separately helps distinguish opportunities that were never recognized from those that were recognized but not successfully acted upon.

Timing Error

Record the deviation between actual execution time and the optimal action window. Larger timing errors indicate weaker alignment with the environment’s pace.

Stale Action Rate

Measure the proportion of issued actions that become invalid before execution. This metric directly reflects the reliability of state caching, time estimation, and pre-execution validation.

Decision Latency

Measure the interval between the appearance of a critical event and the issuance of a valid action. Report both average and tail latency: a small number of very slow responses can substantially damage the overall experience.

Replanning Ability

Evaluate whether the model promptly abandons an invalid plan and generates a new one when the environment diverges from expectations. A real-time system should not remain committed to a plan that no longer applies.

Resource and Risk Costs

Track invalid clicks, duplicate operations, failed attempts, and unnecessary waiting alongside task completion. In production systems, completing a task with low risk is often more important than maximizing success rate alone.

6. Engineering Practices for Real-Time Agents

A practical architecture can combine event-driven perception, tiered reasoning, and conditional execution.

First, trigger perception in response to environmental changes rather than relying entirely on fixed-interval polling. Increase observation frequency around high-value events such as countdowns, state transitions, and resource updates. Reduce it when the environment is stable to conserve compute.

Second, split reasoning into urgent and regular paths. The urgent path should handle only events requiring an immediate response, using a simplified state representation and a limited action set. The regular path should handle complex planning, strategy comparisons, and subsequent optimization.

Third, add expiration checks, precondition validation, timeout cancellation, and failure rollback to the execution layer. Model output should not translate directly into an environmental operation. A safety-aware executor should validate and control actions between generation and execution.

Finally, retain complete event logs, including observations, observation timestamps, model decisions, action issue times, execution results, and failure reasons. Connecting these records makes it possible to determine whether a failure arose from delayed perception, slow reasoning, incorrect timing, or an unreliable execution interface.

7. Conclusion

Real-time interaction marks an important shift from AI systems that answer questions to systems that participate in changing environments. Models must close the perception–reasoning–action loop while the world continues moving. They cannot pursue increasingly elaborate reasoning without regard for time, nor can they sacrifice sound judgment simply to respond faster.

Evaluating these agents requires more than asking whether they selected the correct action. A benchmark must also ask whether they detected the opportunity in time, knew when to stop reasoning, recognized stale information, and recovered quickly after unexpected changes.

Only by evaluating time, state, and action within a shared framework can we meaningfully assess whether a model supports reliable real-time interaction.