Realtime Interaction Benchmark Released: Frontier Models Achieve Only 44% Success Rate

Background: When inference speed becomes part of the task

Getting a language model to complete a web operation or play a simple game is no longer difficult today. But if the scenario changes to “grabbing a hot ticket within a fleeting window” or “rotating and moving a block before it lands,” model performance drops sharply. The reason is straightforward: in these tasks, the timeline is controlled by the environment, not by the model.

Most traditional evaluations assume a static environment: the model can spend as much time as it wants thinking, and the environment state remains unchanged during inference. This assumption is reasonable for offline Q&A, code generation, and similar scenarios, but it completely breaks down in realtime interaction settings. Ticket windows close, opponents raise bids, and game characters keep moving. The longer the model “thinks,” the fewer actionable opportunities remain.

To systematically measure this capability gap, the research community has proposed realtime interaction benchmarks such as Realtime-Bench, which place models in continuously changing environments and test whether they can observe, reason, and act under time constraints. The results show that even the best-performing frontier models achieve only around 44% task success rate, meaning realtime capability remains a clear weakness in current model systems.

Three core capabilities for realtime interaction

Realtime-Bench breaks realtime interaction into three interrelated capability dimensions. This decomposition offers a useful reference for engineering practice.

1. RT-Perceive: Observing environmental changes

The model must continuously perceive the environment state and determine three things: what the current state is, whether the information at hand is already stale, and whether the action opportunity still exists. The hard part is “staleness detection”: a button on the interface may still be visible while the corresponding inventory or quota has already dropped to zero. The model must distinguish between “visible” and “valid.”

2. RT-Deliberate: Adjusting reasoning according to time

The model needs to decide “how much longer it can think” and “whether continued thinking is worth it.” This is a form of meta-reasoning: perform deep planning when time is abundant, and degrade to fast heuristic decision-making when time is tight. Without this layer of control, the model either acts prematurely and makes errors, or overthinks and misses the window.

3. RT-Act: Timing the action correctly

The model needs to judge whether to wait or act immediately, and choose an action that will “still be valid at execution time.” This is fundamentally different from action selection in static environments. In a static environment, an action’s validity depends only on the current state. In a realtime environment, there is a delay between decision and execution, so the model must predict the environment state at the moment of execution.

Take ticket purchasing as an example: the model must first detect the ticket-release signal, then control its thinking budget based on the remaining time of the purchase window, and finally submit the order in time. A timing error at any step causes the entire pipeline to fail.

Benchmark design: Making the timing of actions matter

Realtime-Bench’s task design follows two principles: the environment evolves on its own over time, and opportunities to complete the goal are time-limited. Even if the operation itself is correct, executing it too early or too late can still lead to failure.

Typical task formats include:

• Real-time auctions: The model must place a bid before the countdown ends while responding to opponents’ unpredictable price increases.
• Snake-like games: The character keeps moving forward, and the model must turn in time to reach the target.
• Flash sales: Inventory and interface states change frequently, leaving only short decision windows.

The benchmark includes 30 web tasks and 20 game tasks, covering eight categories: travel, e-commerce, services, security, as well as arcade, puzzle, strategy, and simulation. This coverage matters because realtime capability is not a single skill but a composite performance across different interaction rhythms.

Clock strategies: Three ways to measure time cost

The key to evaluating realtime capability lies in how “environment time” is calculated. Realtime-Bench compares three ways of advancing the environment, and this design is essential for understanding why models fail.

Clock strategy	Advancing method	Purpose
Wall Clock	Advances the environment according to the real time consumed by inference	Closest to real deployment; measures end-to-end usability
Paused Clock	Freezes the environment during inference and advances it after inference completes	Idealized control; removes time pressure
Hybrid / Bounded Clock	Sets a cap on inference duration or scales it proportionally	Isolates specific latency-sensitive stages

The combination of these strategies creates a clear attribution path:

• If a model succeeds under Paused Clock but fails under Wall Clock, the bottleneck lies in inference latency and time management rather than task understanding.
• If a model fails under both strategies, its environment perception or action selection itself has flaws.
• If a model’s performance drops sharply as the budget tightens under a bounded clock, it lacks the reasoning-budget adjustment capability required by RT-Deliberate.

This controlled-comparison evaluation design is more useful for guiding optimization than a single success-rate metric.

Failure mode analysis: Where models actually get stuck

From public results, frontier model failures cluster into several reproducible patterns:

1. Perception lag: The model makes decisions based on stale screenshots, and the environment has already changed by the time the action is submitted. A typical symptom is repeatedly clicking on controls that are no longer valid.
2. Reasoning-budget mismatch: The model carries over the “long thinking” habit from static evaluations and consumes too much time in short-window tasks, or produces premature actions in long-window tasks.
3. Action timing misjudgment: The model selects the right type of action but at the wrong execution moment, such as submitting a request at the exact instant the countdown ends, only to have it invalidated by network and rendering latency.
4. Lack of abandonment and retry strategies: After an opportunity window closes, the model keeps trying along the original path instead of switching to a fallback plan.

These failure modes show that realtime capability gaps often lie not in “knowing what to do” but in “when to do it, how long to spend on it, and whether it will still be in time once done.”

Engineering recommendations: Treat time as a first-class citizen

Based on the analysis above, model applications targeting realtime interaction scenarios can make the following engineering-level improvements:

(1) Explicitly incorporate time budgets into prompts and state

Inject metadata such as remaining time and environment evolution rate into each interaction turn so the model has a basis for budget decisions:

[state] The current interface has remained unchanged for 2.4s
[deadline] 6.0s remaining until the window closes
[latency] The previous action took 1.2s end-to-end
[instruction] If remaining time < 2x latency, submit the current best action immediately

(2) Adopt a layered decision architecture

Separate “slow thinking” from “fast reaction”: a lightweight policy handles high-frequency perception and emergency action triggering, while a large model handles low-frequency planning. The emergency path does not depend on large-model inference to complete, thereby avoiding inference latency.

(3) Validate action validity

Before execution, run a low-cost check to confirm that the target control is still in an interactive state. For operations with insufficient idempotency, such as order submission, add deduplication and timeout protection.

(4) Build timing regression test sets

Introduce interaction replays with clock advancement into continuous integration, and track the “success rate versus latency” curve as a regular metric instead of looking only at final success rate. This can expose reasoning-budget mismatch problems early.

(5) Design degradation and abandonment strategies

Preset fallback paths and abandonment thresholds for each task to avoid continuing ineffective attempts after a window closes, which can pollute subsequent states.

Conclusion

The value of Realtime-Bench lies not only in producing a 44% success-rate number but also in turning “time” from an implicit evaluation assumption into an explicit variable. For developers, this means realtime interaction capability cannot be solved by simply scaling up model size. Perception freshness, reasoning-budget scheduling, and action-timing calibration all need to be designed separately at the system level. Treating time as a first-class citizen in architecture and testing is a practical path to improving realtime interaction success rates at the current stage.