Why AI Can't Read the Next Second: The Technical Essence of State-Transition Reasoning
Multimodal LLMs struggle to predict what happens next in a scene, and this gap persists even as models scale up. This article argues that spatial reasoning, physical commonsense, embodied control, and temporal ordering are not four separate weaknesses but four faces of a single missing capability: modeling world dynamics. By unifying these tasks under the (s, a, s′) triple, it explains why world dynamics is so hard to learn and outlines practical engineering approaches—from explicit world models and simulated data to physics-informed inductive bias and uncertainty-aware prediction.
A Simple Problem That Keeps Tripping Models Up
Imagine a routine scenario: show a multimodal model a photo of a kitchen, tell it “someone is reaching toward a pot on the stove,” then ask “what would you see next?” For a human, the answer comes without hesitation—the pot gets lifted, or the hand grips the handle. For today’s top multimodal large language models, this question is surprisingly hard.
A study evaluating several mainstream flagship models found that on these “next-scene prediction” tasks, even the strongest model hovers around two-thirds accuracy, while ordinary people score above ninety percent on the same test. More concerning is that this gap does not close automatically as parameter counts grow—making models larger and feeding them more data does not systematically erase the divide.
This suggests the problem is not “insufficient effort” but likely “effort in the wrong direction.” Multimodal large language models (MLLMs) excel at the pattern associations learned from massive image-text corpora, but “what happens in the next second” tests something else entirely: the ability to model how a scene evolves over time.
Fragmentation: Four Kinds of “Failure,” Never Treated as One
Academia has long noticed these shortcomings, but has studied them as four separate tracks that never meet.
- Spatial reasoning: whether a model understands how a scene should change after the camera translates or rotates. Related benchmarks typically build “change the viewpoint—predict the new view” tasks.
- Physical commonsense: whether a model grasps naive laws such as objects not vanishing out of thin air, or an object above falling once its support is removed.
- Embodied control: whether a model can complete tasks like “pick up the cup” that require continuous action decisions in virtual or real environments.
- Temporal ordering: whether a model can restore shuffled frames from a video into the correct causal sequence.
The problem is that each of these tests uses its own scoring criteria and picks its own distractors. Read through the entire literature and you will repeatedly see “the model performs poorly in all four areas,” but no one answers a more fundamental question: are these four failures four isolated defects, or four branches growing from the same root?
A Unified View: Reducing Everything to (s, a, s′)
The hypothesis proposed by a joint study gives an affirmative answer. It argues that although these four task types look different on the surface, they are all fundamentally asking the same thing—given an initial state of a scene plus an action, can the model infer what the state will become after the action?
We can write this cleanly with a triple: (s, a, s′). Here s is the state before the action, a is the action taken, and s′ is the state after the action. Once we have this unified abstraction, the four seemingly different tasks become reasoning about the same triple from different directions:
- Spatial reasoning: the camera performs a rotation action
a, and the view changes fromstos′; asking whats′looks like—this is forward prediction. - Physical commonsense: a ball loses its support (the “action/condition”), and we ask its subsequent state
s′—again forward prediction. - Embodied control: given the current state
sand a goal (the desireds′), infer which actionato take—this is inverse inference. - Temporal ordering: given a set of states, determine their transition direction—this is order recovery.
In pseudocode, this reduction is very clear:
class WorldState:def __init__(self, obs, context):self.obs = obs # current observable state sself.context = context # scene priors, physical constraints, historydef predict_next(state, action, world_model):"""Forward: given s and a, infer s′ — covers prediction/imagination, spatial and physical commonsense"""return world_model.forward(state, action)def infer_action(state_before, state_after, world_model):"""Inverse: given s and s′, infer a — covers action decisions in embodied control"""return world_model.invert(state_before, state_after)def order_states(states, world_model):"""Ordering: given several states, recover the transition order — covers temporal reasoning"""return world_model.rank_transitions(states)
Once we accept this framework, the conclusion becomes profound: what the model truly lacks is a “world model” capable of modeling how a scene evolves over time, not four independent skill modules. Tuning and benchmarking each of them separately has been treating symptoms; only by building the foundation of world-dynamics modeling can we hope to improve all four directions at once.
Root-Cause Analysis: Why “World Dynamics” Is So Hard to Learn
Understanding the unified framework, we still need to answer a tougher question—why can’t models learn this set of dynamic laws? There are at least four reasons.
First, the state space is continuous and high-dimensional. The success of language models rests on the statistical regularities of discrete tokens, but the state of the physical world is continuous: object position, velocity, orientation, lighting, material deformation… between any two “next seconds” there are infinitely many possibilities. Using a “predict the next word” approach to approximate “the next continuous state” is inherently mismatched.
Second, the training objective rewards correlation rather than causation. What models learn from massive static image-text data is a co-occurrence relationship like “pot” and “stove” frequently appearing together, not the causal dynamics of “a hand applies force, the pot gains acceleration, and it moves under gravity.” The former lets a model describe an image but not reason about the next step.
Third, out-of-distribution generalization is weak. Even if a person has never seen a particular kitchen, they can use naive physical intuition to imagine a reasonable next step; once a model leaves the training distribution, its predictions collapse. And the “next step” is precisely almost always a combination never exactly seen in the training set.
Fourth, there is no explicit representation of uncertainty. In real scenarios, the same initial state may have multiple reasonable continuations (the pot is lifted, or the hand merely adjusts its angle). Models tend to produce a single “most training-distribution-like” answer and cannot express “there are several possibilities here, and how probable each one is”—which is fatal in embodied tasks that require continuous decision-making and risk trade-offs.
Engineering and Algorithmic Paths Forward
Once we understand the root causes, the direction for improvement becomes clear. The following are technical paths worth investing in today.
1. Explicitly introduce a world model as a separate component. Instead of hoping a general-purpose large model will “incidentally” learn physics in its parameters, separately train a dynamics model that can perform forward prediction of state transitions. Let the general model handle high-level understanding and planning, and let the world model handle low-level reasoning about “what the next moment looks like.” Division of labor, rather than forcing one model to bear it all.
2. Use synthetic and simulated data to supplement the dynamic supervision signal. Static image-text corpora inherently lack paired “action–outcome” samples. Generating large numbers of (s, a, s′) triples with physics engines can provide exactly the kind of supervision the model most lacks, especially suited to filling in physical commonsense and spatial-transformation areas.
3. Write physical constraints into the architecture as inductive bias. Rather than hoping the model will “guess” conservation laws, support relations, and gravity direction from the data, inject such priors as architectural constraints (for example, explicit modeling of occlusion and contact relations) to reduce the degrees of freedom that must be learned.
4. Make uncertainty a first-class citizen. When predicting s′, the model should output a distribution rather than a point estimate, combined with Monte Carlo sampling or ensemble disagreement, so that downstream planning knows “where it can trust and where it should be cautious.” This is critical for the safety and robustness of embodied control.
5. Cloud support for scalable training and evaluation. This kind of research involves storing and scheduling massive multimodal datasets, running large-scale parallel simulation environments, and elastically scaling distributed training. General cloud capabilities—object storage to hold datasets, a container platform to orchestrate simulation and training jobs, message queues to decouple data pipelines, and monitoring and alerting services to track training metrics—can turn “getting an experiment to run once” into “a pipeline for sustainable iteration.” Baidu AI Cloud provides related support in these general capabilities and can serve as a reference option for teams building such experimental platforms.
Conclusion
The question “why can’t AI read the next second” has an answer that is not mysterious: it is not four isolated defects but four faces of the same capability gap—a lack of world-dynamics modeling for how scenes evolve over time. Unifying spatial reasoning, physical commonsense, embodied control, and temporal ordering under the (s, a, s′) triple lets us both see the essence of the problem and find more targeted technical paths.
For developers and researchers, what is truly worth investing in may not be continuing to chase scores on four separate tracks, but filling in the foundation that lets a model “understand how the world changes over time.” This is the key step that takes AI from “understanding a single image” to “reading a chain of cause and effect.”
