Realtime Interaction Benchmarks: When “Can Answer” Must Become “Can Keep Up”

Over the past two years, evaluations of large language models have focused primarily on static tasks: given a complete input, the model generates a complete output, and the answer is compared against a reference. This approach is mature, reproducible, and easy to compare across models. But it rests on a strong assumption — the environment is static, and all information is provided up front.

Real-world interactions rarely satisfy that assumption. Voice assistants must continuously understand users as they speak and remain ready for interruptions. Collaborative document editors must maintain consistent state across streaming events from multiple users. Automated operations agents must make decisions while monitoring metrics that change from moment to moment. These scenarios share a defining trait: input is a stream that advances over time, and the model must respond promptly under incomplete information and a continuously changing environment. Static evaluations cannot capture this capability, which is why realtime interaction task evaluation has become a distinct and difficult problem.

A recent research effort introduced a benchmark specifically for realtime interaction tasks and systematically evaluated more than ten leading models. The results showed that even the best-performing model achieved a task success rate of only about 44%. That number deserves close attention. It suggests that realtime capability is not something that can be solved by simply adding more compute. It is a systemic engineering problem involving task modeling, state management, and temporal decision-making.

1. The Nature of Realtime Interaction Tasks: Three Coupled Challenges

To understand why the success rate is so low, it is necessary to first understand the structural differences between realtime interaction tasks and static tasks. These differences can be broken down into three coupled challenges.

First, input is streaming, and arrival timing itself carries information. In a static task, “what the user said” is all the information available. In a realtime task, “when the user said it, how long they spoke, and how long they paused” is also information. A model that responds after a 0.3-second pause and one that waits for a 1.5-second pause may achieve the same accuracy, but the interaction quality is entirely different. The time dimension is not an add-on; it is part of the task definition.

Second, the environment state continues to change while the model is reasoning. There is latency between when a model receives input and when it produces output. During that interval, the environment may have already advanced. If a model makes a decision based on observations at time t but executes it at time t+Δ, the premise behind that decision may no longer hold. This means evaluations must consider not only “is the decision correct” but also “within what time window does the decision remain valid.”

Third, interaction is a closed loop, and model actions change subsequent inputs. In static tasks, the input distribution is independent of model output. In realtime tasks, a single response from the model triggers the next step in the environment, creating a feedback loop. This means that correctness at a single step does not guarantee overall success. Errors accumulate and amplify, so evaluations must examine multi-turn trajectories rather than single-point accuracy.

Together, these three challenges mean that realtime interaction task evaluation cannot simply reuse the design of static benchmarks.

2. Key Design Considerations for a Realtime Evaluation Benchmark

A usable realtime interaction benchmark requires explicit design decisions across several dimensions.

2.1 Task Modeling: Abstracting the Environment as a Progression State Machine

The core of the evaluation is constructing an environment that can advance in real time. A common approach is to model the task as a state machine: the environment maintains an internal state, advances at fixed or variable time steps, exposes partial observations to the model at each step, and receives actions from the model. Model actions trigger state transitions, which in turn affect subsequent observations.

The advantage of this approach is that it is reproducible, parallelizable, and allows precise control over timing. The following is a simplified environment interface skeleton expressed in pseudocode:

class RealtimeEnv:
    def __init__(self, config):
        self.state = initial_state(config)
        self.clock = 0.0
    def observe(self):
        # Expose only the partial information visible at the current moment
        return partial_view(self.state, self.clock)
    def step(self, action, dt):
        # The environment advances with time, and actions affect state transitions
        self.state = transition(self.state, action, dt)
        self.clock += dt
        return self.observe(), self.is_done(), self.reward()
    def is_done(self):
        return terminal(self.state)

On the model side, the loop consists of observing, reasoning, outputting an action, and waiting for the environment to advance. The key point is that reasoning itself consumes time, and the evaluation framework must decide whether that time counts against the environment clock — this directly affects how “realtime performance” is measured.

2.2 Time Accounting: Inference Latency Must Be Included

If the evaluation allows the model to “pause the world” and think at length, it is not measuring realtime capability. A reasonable approach is to map the wall-clock time consumed by model inference to progress in environment time. In other words, the longer the model thinks, the further the environment advances, and the more stale the model’s observations become.

This design turns latency from an engineering metric into an evaluation metric. An accurate but slow model will miss critical windows as the environment progresses and will fail at the task level. This mirrors the reality of production deployments.

2.3 Metrics: Success Rate Alone Is Not Enough

A single success rate metric carries limited information. It is advisable to cover at least three categories of metrics:

• Task success rate: whether the full trajectory achieves the goal, serving as the terminal metric.
• Response latency distribution: the distribution of delays from observation to action, reflecting interaction smoothness.
• Decision validity window: whether an action still holds at execution time, capturing cases where the model “thought correctly but too late.”
• Trajectory stability: whether errors accumulate across multiple rounds, measurable through trajectory-level variance or the number of fallback events.

Taken together, these four metric categories can distinguish between “insufficient capability” and “incorrect timing” as failure modes — two problems with entirely different optimization paths.

2.4 Task Diversity: Covering Different Types of Realtime Pressure

Realtime pressure comes in many forms: high-frequency event streams with many small events per second, low-frequency long-duration tasks that require maintaining state for minutes, sudden disturbances where the environment changes sharply in a short period, and adversarial interruptions where inputs are deliberately cut off or injected with noise. An evaluation set should span these dimensions; otherwise, the measured success rate will only reflect one type of scenario.

3. What the 44% Figure Tells Us

Placing this number back into the framework above makes its implications clearer.

First, it suggests that current models may perform reasonably well on single-step decisions but fall short on closed-loop trajectories. High single-point accuracy combined with low full-trajectory success typically means errors accumulate: a small early deviation goes uncorrected, subsequent observations drift from expectations, and the task eventually fails. This points to gaps in state management and self-correction capabilities.

Second, it suggests that time accounting is one of the main bottlenecks. When inference latency counts against the environment clock, the model’s “thinking cost” directly translates into environment drift. Long-chain reasoning is an advantage in static tasks but can become a liability in realtime tasks. This is not a matter of the model being insufficiently intelligent; it is a mismatch between the reasoning paradigm and realtime constraints.

Third, it suggests that realtime capability cannot be extrapolated from static benchmark scores. A model that performs strongly on static question answering will not necessarily maintain the same level of performance in streaming interaction. Evaluation must be designed specifically for realtime characteristics, which is precisely why a dedicated benchmark is necessary.

4. Engineering Practices: Improving Realtime Interaction Reliability

From an engineering perspective, improving realtime capability is not a single-point optimization but a set of coordinated design decisions.

4.1 Break Inference into Interruptible Segments

Rather than having the model perform one long reasoning pass before producing output, break the decision process into multiple short segments. At the end of each segment, the model can output a “current best action” or “no action for now.” This ensures that as the environment advances, the model can always provide a usable response instead of blocking while waiting. In practice, this can be implemented with streaming inference combined with an action buffer:

def realtime_loop(env, model, budget_ms):
    obs = env.observe()
    while not env.is_done():
        # Perform incremental inference within the time budget,
        # allowing interruption at any point
        action = model.infer_incremental(obs, budget_ms)
        if action is None:
            action = model.fallback_action(obs)  # Fallback action
        obs, done, reward = env.step(action, dt=model.elapsed())
        if done:
            break

The key is the existence of fallback_action: when inference cannot complete in time, the system can still provide a conservative but safe action, avoiding the more severe timing failures caused by getting stuck.

4.2 Explicitly Maintain Environment State Estimates

Models should not rely solely on the current observation to make decisions. They should maintain an internal estimate of the environment state and update it after each observation. This way, even when observations are delayed or missing, the state estimate can provide continuity. In engineering terms, this is equivalent to equipping the model with a lightweight state tracker, moving “memory” out of the model’s internal parameters and into a component that can be inspected and debugged.

4.3 Drive Inference Depth with a Time Budget

Inference depth should be a function of time, not a fixed value. Perform deeper analysis when time is abundant, and take a fast path when time is tight. This adaptive strategy is common in realtime systems and applies equally to large models: output a rough decision first to ensure timeliness, then refine it in the background and replace it at the next decision point.

4.4 Embed Evaluation into the Development Loop

Degradation in realtime capability often only becomes apparent at the integration stage. It is advisable to introduce a progression simulation environment into the development loop and run trajectory-level evaluations on every change, tracking changes across success rate, latency distribution, and trajectory stability. This is far more economical than discovering problems through production metrics after deployment.

5. Conclusion

The value of realtime interaction task evaluation lies not in producing a ranking but in decomposing an ambiguous “realtime capability” into a measurable, optimizable engineering problem. The 44% figure is not an endpoint but a starting point. It reminds us that capability accumulated on static tasks does not automatically translate into reliability in realtime scenarios. The latter requires independent task modeling, independent time accounting, and independent metrics.

For developers, the practical takeaways are clear: model the environment as a progression state machine, include inference latency in evaluation, break decisions into interruptible segments, and make state estimation explicit. These practices are independent of any specific model and represent general engineering principles for realtime interactive systems. Only on that foundation can model capabilities be meaningfully evaluated and selected.