Active Spatial Mentalization: Completing the Final Piece of Large Model Spatial Reasoning
Vision-language models handle image captioning, object recognition, and basic relative-position questions well, yet they routinely fail on viewpoint changes, cross-image association, and reasoning about regions that are not visible in the image. This article examines why this happens and introduces Active Spatial Mentalization, an approach that splits the ability into two modelable stages—perspective planning and perspective imagination. It explains the root cause (training data that teaches answers but not the reasoning process), outlines an explicit multi-stage inference framework with verifiable coordinate transformation, and covers data-construction strategies, deployment guidance, and evaluation practices. Readers will learn how to diagnose where spatial reasoning breaks down and how to improve multimodal spatial reliability through process supervision rather than simply scaling up the model.
1. A Capability Humans Take for Granted, but AI Often Gets Wrong
Picture this scenario: you are sitting in your living room when someone asks, “If you walk to the kitchen doorway and turn around to look back, is the sofa on your left or your right?”
Almost everyone can answer instantly. Behind that “instant” answer lies a whole set of complex cognitive operations: you first mentally “relocate” yourself to the kitchen doorway, switch to a viewpoint that does not currently exist, mentally “render” the living room as it would appear from that angle, and finally make a left-or-right judgment based on the imagined scene.
The whole process happens so fast that you never realize you did any of it, but the ability it requires is far from trivial: actively reconstructing a hypothetical observation viewpoint and reasonably imagining and inferring the scene content under that viewpoint.
This ability is almost instinctive in humans, yet when you pose the same question to today’s most advanced large vision-language models (AI systems that can understand both images and text), they tend to stumble, giving wrong answers or even reversing left and right.
This is precisely the challenge that spatial reasoning research has been trying to tackle in recent years. The research team named this “innate in humans, generally missing in AI” ability Active Spatial Mentalization and designed its methods around it.
2. Clarifying the Concept: What Is “Active Spatial Mentalization”
It is worth stressing that mainstream vision-language models already perform reasonably well on many common visual tasks: describing images, basic object recognition, and simple relative-position judgments (such as “A is to the left of B”), most of which they can handle.
The problems emerge in three categories of tasks:
- Viewpoint transformation: re-judging spatial relations from a new hypothesized observation point;
- Cross-image association: aligning different parts or views of the same scene across multiple images;
- Unseen-region inference: inferring content that does not appear in the current image but must logically exist.
Active Spatial Mentalization is precisely the core ability targeting these three task categories. It does not passively read information that is “already presented” in the image; instead, it actively generates a viewpoint plan beyond the current observation position, constructs a mental representation of the scene under that viewpoint, and reasons accordingly.
Consider a typical cross-image example: given two photos of the same office, the question is, “Taking the direction the person sitting in the black chair is facing as forward, in which direction is the computer located relative to them?” To answer correctly, the model must:
- Locate the chair and the computer in the two images respectively;
- Associate the information scattered across the two images into a unified spatial representation;
- Imagine what the whole room would look like when “a person is sitting in the chair looking forward”;
- Judge the direction of the computer relative to “forward” under this imagined viewpoint.
In tests, many mainstream vision-language models simply answer incorrectly. The reason is not that they cannot understand a single image, but that they cannot link information across images, nor can they proactively “fill in” the picture under that key viewpoint.
3. Root Cause: Why Models Fail to “See from a Different Angle”
Behind this lies a more fundamental contradiction, one concentrated in the training data.
The vast majority of datasets currently used to train spatial reasoning provide only “question–answer” pairs, for example:
Q: In which direction is A relative to B?A: Left
Yet they almost never explain “what thinking process is required to arrive at this answer.” Training on such data, the model effectively learns surface-level pattern matching, not the complete cognitive chain that humans follow: “plan the viewpoint → construct the scene → reason and judge.”
Worse still, a natural remedy—simply having the model generate its own “how to think” process data—often does not work. A model lacking genuine spatial perception is prone to:
- fabricating incorrect viewpoint plans (for instance, misjudging which direction to observe from);
- conjuring objects or layouts that do not exist in the scene at all.
This is like asking someone who has never cooked to write a recipe: the steps sound plausible, but in practice everything goes wrong. The generated chain of thought may look reasonable, yet it is disconnected from the real spatial structure, and instead “hardens” errors into the training process.
4. Breaking Down the Capability: Two Stages Hold the Key
The core idea of the research is to split “Active Spatial Mentalization” into two sub-abilities that can be modeled independently yet must work in coordination:
First, Perspective Planning: judging “where to stand and which direction to face” in order to find the answer, based on the question. This is a reasoning and decision-making ability.
Second, Perspective Imagination: after the observation point is determined, constructing in the mind roughly how the scene would appear from that viewpoint. This is a spatial representation and generation ability.
Only when the two are combined do they form a complete “Active Spatial Mentalization.” The value of this decomposition is that it turns the vague, sweeping problem of “the model cannot put itself in another’s position” into two engineering problems that can be diagnosed and optimized separately—whether the error is that the planning stage chose the wrong observation point, or that the imagination stage built a distorted scene.
5. A Practical Framework: Making the Mentalization Process Explicit
Based on the above decomposition, a practical implementation idea is to make the originally implicit thought process explicit as a multi-stage reasoning pipeline, rather than hoping the model will answer correctly end-to-end in one shot.
The following pseudocode describes a general skeleton for spatial mentalization reasoning:
def active_spatial_mentalization(images, question):# Stage 1: Scene parsing and entity localizationentities = detect_and_localize(images, question) # Locate the key objects the question refers to# Stage 2: Perspective planning (decide "where to look from")viewpoint = plan_viewpoint(question=question,entities=entities)# Example output: {anchor: "black chair", direction: "facing direction", mode: "first-person"}# Stage 3: Perspective imagination (construct "what is seen")mental_scene = imagine_scene(entities=entities,viewpoint=viewpoint) # Reorder entity relations under the new viewpoint coordinate system# Stage 4: Reason and judge based on the imagined viewpointanswer = reason_over_scene(mental_scene, question)return answer, {"viewpoint": viewpoint, "mental_scene": mental_scene}
This skeleton has three key design points:
- Making intermediate planning explicit: exposing the “observation point” as a checkable, intervenable intermediate result, so it is easier to diagnose whether the error lies in planning or imagination.
- Coordinate transformation as the verifiable core step: viewpoint transformation is essentially re-expressing the relative positions of entities in a new coordinate system. The coordinate transformation illustration below helps explain the mathematical core of this step.
import numpy as npdef transform_to_new_viewpoint(point_world, eye_pos, forward, up=(0, 0, 1)):"""Project a point given in world coordinates into a new viewpointcoordinate system with eye_pos as the origin and forward as the front."""forward = np.array(forward, dtype=float)forward /= np.linalg.norm(forward)up = np.array(up, dtype=float)right = np.cross(forward, up) # right directionright /= np.linalg.norm(right)true_up = np.cross(right, forward) # corrected up directiondelta = np.array(point_world, dtype=float) - np.array(eye_pos, dtype=float)# Project onto the right / up / front basis vectors respectivelyx = np.dot(delta, right) # >0 means to the right in the new viewpointy = np.dot(delta, true_up) # >0 means abovez = np.dot(delta, forward) # >0 means in frontreturn {"right": x, "up": y, "front": z}
- Cross-image information alignment: multi-image scenarios require first unifying the entities from each image into the same spatial representation before perspective planning; otherwise, the planning stage lacks complete context.
6. Training Data: Teaching the Model “How to Think”
The framework solves how reasoning is organized at inference time, but whether the model can truly learn depends on how the training data is constructed. The core difficulty here was already noted earlier: having the model self-generate its chain of thought directly introduces hallucination. Viable directions for improvement include:
- Prioritizing process annotation over answer annotation: complete the explicit “plan–imagine–judge” reasoning process for each sample, rather than providing only the final choice, so the model learns a generalizable reasoning structure.
- Strong-constraint validation of intermediate results: validate intermediate results such as the perspective plan for correctness individually; filter out or correct any sample whose plan is wrong or whose imagined content contradicts the real scene structure, cutting off the propagation of erroneous reasoning.
- Using verifiable geometric constraints as anchors: spatial orientation relations have many constraints that can be directly verified through geometric computation (such as relative bearings and the sign of projections in a coordinate system). Such “automatically checkable” signals can be used to screen and reward high-quality samples, reducing reliance on manual annotation.
- Placing positive and negative examples side by side: present “correct viewpoint + correct judgment” paired with “wrong viewpoint + wrong judgment,” so the model learns that different observation points lead to different conclusions, strengthening its sensitivity to the importance of the planning step.
7. Engineering Deployment and Evaluation
When introducing this capability into real systems, several practical suggestions are worth noting:
- Treat mentalization as an optional reasoning module: not every question requires viewpoint transformation. A front-end classifier can determine whether a question belongs to the viewpoint/cross-image/unseen-inference category, triggering the full multi-stage pipeline only for such questions to balance effectiveness against cost.
- Output explainable intermediate results: presenting the perspective plan and imagined scene as displayable intermediate products aids debugging and can also improve answer credibility in user-facing products.
- Build a multi-dimensional evaluation set: do not look only at overall accuracy; instead, evaluate “whether the perspective plan is correct,” “whether the imagined scene is distorted,” and “whether the final judgment is correct” in layers, so the bottleneck can be located to a specific stage.
- Watch out for spurious reasoning: a chain of thought that looks reasonable but is unrelated to the real spatial structure is more dangerous than “no chain of thought” at all. Evaluation should specifically check the geometric consistency between the intermediate process and the scene.
8. Conclusion
Viewpoint transformation, cross-image association, and unseen-region inference have long been weak spots in the spatial abilities of vision-language models. The root cause does not lie in the model’s inability to “understand images,” but in a training paradigm that teaches answers without teaching the process, leaving the model stuck at pattern matching.
The idea offered by Active Spatial Mentalization is instructive: it decomposes the vague human intuition of “putting yourself in another’s position” into two stages that can be explicitly modeled and optimized separately—Perspective Planning and Perspective Imagination—and systematically fills in this last piece through constrained reasoning pipelines and reconstructed training data. For engineering practitioners, this suggests that improving the spatial reliability of multimodal models often depends not on blindly scaling up the model, but on “unfolding” the implicit cognitive steps, making them explicit, and applying constraints at both the data and reasoning levels.
