Spatial Reasoning in Vision-Language Models: Breaking the Bottleneck with Active Mentalization
Large vision-language models (LVLMs) handle many visual tasks well, yet they stumble once a question requires reasoning about how a scene would look from a different viewpoint—covering perspective transformation, cross-image association, and inference about occluded or out-of-frame regions. This article examines why current training data, which pairs questions only with answers, pushes models toward rote pattern matching, and why letting models freely generate their own reasoning introduces hallucinations. It then proposes decomposing the ability into two supervised sub-skills—perspective planning and perspective imagination—forming what can be called active spatial mentalization. It outlines how to build trustworthy training data from multi-view, 3D-annotated, and synthetically rendered sources, how to structure explicit reasoning chains, and how to apply stepwise supervision, joint optimization, and fine-grained evaluation so that every step of the reasoning holds up.
The Deceptive Breadth: Spatial Blind Spots in Vision-Language Models
Large vision-language models (LVLMs) process images and text simultaneously and have grown quite mature at many visual tasks. Describing images, recognizing common objects, and judging basic left/right/up/down relations are tasks they usually handle with ease, with accuracy high enough to support many production scenarios.
But the moment a question shifts from “what is in the image” to “what would the image look like from a different viewpoint,” they start to struggle. The typical weaknesses cluster into three cases:
- Perspective transformation: using some object’s or a fictional observer’s orientation as a new reference frame and re-describing the relative positions of other objects.
- Cross-image association: linking clues scattered across two or more images into a unified understanding of the scene.
- Invisible-region inference: mentally filling in areas that are occluded or outside the frame to answer “what does the unseen area probably look like.”
Take a practical example: given two photos of the same office, ask “If you treat the orientation of the person sitting on the black chair as forward, in which direction is the computer relative to that person?” To answer correctly, the model must complete an entire reasoning chain—first locate the chair and computer separately in each image, then imagine a person actually sitting there and looking forward, then mentally reconstruct “how the whole room is arranged when viewed from that perspective,” and finally judge whether the computer is front-left, front-right, or some other direction.
In actual testing, a mainstream vision-language model gave a wrong answer directly to this kind of question. The reason isn’t that it couldn’t recognize the chair or computer, but that it could neither effectively associate cross-image information nor proactively construct that key mental picture from the given viewpoint.
The Root Cause: Only “Correct Answers,” No “Reasoning Path”
To understand why models stumble on these tasks, start with how training data is composed.
Most public datasets used to train spatial reasoning provide only “question-answer” pairs, such as “In which direction is object A relative to object B? Answer: left side.” The data never explains what viewpoint selection, spatial imagination, and step-by-step derivation are needed to reach that directional judgment.
This creates a fundamental contradiction: the model is exposed to large numbers of isolated input-output samples, so what it naturally learns is a kind of rote pattern matching—mapping specific question phrasings to specific answer texts—rather than, like a human, proactively planning which angle to observe from, constructing a mental picture, and then reasoning based on that picture.
The reason humans can easily answer the opening office question is that we unconsciously “stand” on that chair and rearrange the scene in front of us. The current training paradigm never teaches the model to do this; it has neither an explicit “switch viewpoint” step nor a “construct mental image” intermediate step. The two most critical actions in the spatial reasoning chain are exactly the parts missing from the data.
Why “Letting the Model Figure It Out” Falls Short
A natural idea is: since the reasoning process is missing, let the model generate “how to think” data itself to fill in the intermediate steps. This sounds reasonable but hides a trap.
When you directly let the model self-generate viewpoint planning and scene imagination, it easily produces hallucinations: fabricating a viewpoint plan that is logically coherent but actually wrong; worse, conjuring content that doesn’t exist in the scene at all—like inventing a door that isn’t there or a piece of furniture that isn’t present. Once these errors enter the training data, they get repeatedly reinforced, and the model learns its way into deeper and deeper error.
It’s like asking someone who has never cooked to write a recipe. The steps they write may sound flawless—“heat the pan, add oil, sauté the aromatics, then stir-fry”—and each step looks reasonable in isolation, but in practice it’s full of pitfalls, because they lack real kitchen experience and can’t guarantee the steps connect seamlessly or that the result stays controllable. The model’s self-generated reasoning data is the same: the surface structure looks neat, but underneath it lacks consistency constraints tied to real space.
So simply “adding data” isn’t enough; the key is to supply a trustworthy reasoning step in a trustworthy way.
Decomposing “Change Your Perspective” into Two Steps
The starting point for the solution is to break this seemingly mysterious ability into two sub-abilities that can be modeled and supervised separately:
- Perspective Planning: based on the question itself, decide which angle to observe the scene from in order to have a chance of finding the answer. It solves the “where to stand” problem—choosing which object and which orientation to use as the new reference frame.
- Perspective Imagination: from the fixed viewpoint, construct in the “mind’s eye” roughly what the scene looks like from that angle. It solves the “what to see” problem—mentally completing and rearranging the invisible or cross-image regions.
Linked together, these form a complete capability that can be called “active spatial mentalization“: instead of passively waiting for a question-to-answer mapping, the model actively plans the viewpoint, actively constructs a mental image, and then makes a judgment based on it. The value of this decomposition is that it turns the black-box “answer directly” into a white-box “plan first, imagine next, answer last,” where every stage can have its own supervision signal designed and its own errors located and fixed.
How to Build Trustworthy Training Data
Since the model’s free generation can’t be trusted, data construction must introduce external constraints so the reasoning chain has a solid basis. Here is a fairly general engineering approach used in the industry:
Prefer multi-view, multi-image, or 3D-annotated data. When the same object in a scene has multiple known camera poses, or carries ground-truth information such as depth and 3D coordinates, the relative positions from different viewpoints can be computed programmatically and deterministically, naturally sidestepping the hallucination problem.
Fill scarce stages with synthetic data. Through 3D scene rendering, you can deliberately set the camera position and orientation to precisely generate labels like “from a given viewpoint, object X is in direction Y.” Although this kind of data differs from the real distribution, it offers extremely high consistency and controllability when training the intermediate steps of perspective planning and perspective imagination.
Annotate the reasoning chain explicitly. Make training samples contain not only the question and answer but also a structured record of the intermediate steps. An illustrative data structure:
{"images": ["view_a.jpg", "view_b.jpg"],"question": "Treating the orientation of the person on the black chair as forward, in which direction is the computer located?","reasoning": {"step_1_localize": { "chair": "bottom-right area of image 1", "computer": "corner of the desk in image 2" },"step_2_plan_viewpoint": { "anchor": "black chair", "facing": "toward the window" },"step_3_imagine": "From this viewpoint, the computer falls to the front-left","answer": "front-left"}}
This way, every prediction from the model can be aligned to a specific reasoning step, which makes it easy to apply process supervision during training and easy to pinpoint exactly which link failed during evaluation.
Training and Evaluation: Keeping Every Step Under Control
For training strategy, there are two angles:
- Combine stepwise supervision with joint optimization. First train the two sub-abilities, perspective planning and perspective imagination, separately, so the model learns to choose the right viewpoint and build the correct picture; then, through joint training, connect the two into an end-to-end reasoning flow. The stepwise phase helps the model build a solid foundation, and the joint phase ensures the chain stays coherent.
- Introduce reward signals for correctness. For whether the answer is correct and whether the viewpoint plan matches the ground truth, you can design corresponding reward mechanisms to guide the model toward reasoning paths that align with real space rather than seemingly reasonable hallucinations.
For evaluation, it’s recommended to build a test set covering all three capabilities—perspective transformation, cross-image association, and occlusion inference—and to score the intermediate steps of the reasoning chain separately. Looking only at whether the final answer is right or wrong tends to mask problems; only by breaking things down to each step can you truly tell whether the model “planned the wrong viewpoint” or “imagined the wrong picture.”
Engineering Implementation: A Closed Loop from Data to Service
To actually run the above approach, you also need solid engineering support. Here you can build a closed loop using general cloud capabilities: massive image and annotation data can be managed in tiers on object storage; large-scale model training can scale elastically with container platforms and distributed training; the generation and cleaning of reasoning data suits batch-processing pipelines chained together; and both the training process and online performance can be continuously observed via log services and monitoring/alerting, so regressions of the reasoning chain on specific task types are caught in time. With mature AI development platforms, teams can focus their energy on the core difficulty of spatial reasoning instead of reinventing the wheel.
Conclusion
The spatial weakness of vision-language models is essentially not “can’t see clearly” but “can’t change viewpoint or rebuild a picture in the mind.” Existing data gives answers but not reasoning paths, leaving the model stuck at shallow matching; yet letting the model freely generate its reasoning process introduces hard-to-control hallucinations. The way out is to break the ability into two steps—perspective planning and perspective imagination—to fill in a trustworthy reasoning chain with truth-constrained multi-view and synthetic data, and then to use stepwise supervision, joint optimization, and fine-grained evaluation to make every step hold up. Following this path of active spatial mentalization, large models can truly complete the last missing piece of the spatial reasoning puzzle.
