Teaching Large Language Models to See from a Different Perspective: A Key Breakthrough in Spatial Reasoning

    Spatial reasoning requires more than recognizing objects. A model must identify the observer, determine the observer’s orientation, align information across images, and construct a new reference frame before answering questions about left, right, front, or behind. This article explains why vision-language models often fail at perspective-dependent tasks, how active spatial mental modeling can address the problem, and how structured representations, coordinate transformations, cross-image alignment, and process-aware evaluation can improve future multimodal systems, agents, and robots.

    Humans constantly perform a form of reasoning that is easy to overlook: we mentally move to another person’s position, observe the environment from that perspective, and then judge the relationships between objects.

    Imagine a person standing in the center of a living room and facing the television. If asked, “When the person standing in front of the television turns around, is the sofa on the person’s left or right?”, most people can answer quickly. There is no need to measure distances or draw a diagram. We simply simulate one turn in our minds.

    For today’s vision-language models, however, this type of question is far from simple. A model may correctly identify the sofa, television, and person, yet still give the wrong answer about their relative direction. Error rates often increase when a task involves a change in the observer’s position, information distributed across multiple images, or regions that are not directly visible in the current frame.

    This shows that object recognition is not the same as spatial understanding. The real challenge is whether a model can actively construct a new viewpoint and reorganize the scene information from that perspective.

    1. Spatial Recognition Is Not the Same as Spatial Reasoning

    Vision-language models generally have strong basic perception capabilities. They can identify people, furniture, roads, and buildings in an image, and they can answer direct questions such as “What is next to the table?” or “Is the door open?”

    These tasks share an important characteristic: the answer can usually be read directly from the current image. The model only needs to perform object detection, semantic matching, or a simple relationship judgment.

    Complex spatial reasoning involves a longer chain of operations, including at least the following steps:

    1. Identify the key entities mentioned in the question.
    2. Determine the implied position of the observer.
    3. Establish the direction the observer is facing.
    4. Align information from different images or image regions.
    5. Construct a new viewpoint internally.
    6. Recalculate relationships such as left, right, front, behind, and distance from that viewpoint.

    For example, consider the question: “When the person sitting on a particular chair faces forward, in which direction is the computer?” The model must do more than identify the chair and computer. It must understand that “forward” is defined from the perspective of the person sitting on the chair, not from the camera’s perspective. If the computer appears on the right side of the image but the observer is facing a different direction from the camera, its actual direction relative to the observer may not be “right.”

    The core of spatial reasoning is therefore not simply reading an answer from an image. It is building an observer-centered coordinate system.

    2. Why Models Commonly Reverse Left and Right

    Left-right relationships are strongly dependent on perspective.

    When someone looks into a room from the entrance, the sofa may be on the left. If that person walks to the other end of the room and turns around, the sofa may now be on the right. The object has not moved. What has changed is the observer’s orientation.

    Human cognition usually handles this automatically. We first determine where “I” am, then establish the direction “I” am facing, and finally judge the target object relative to that reference point.

    A model may instead adopt a simpler but unreliable strategy: treating the object’s visual position in the image as the answer. If the target object appears on the right side of the frame, the model is likely to answer “right.” This kind of image-position matching fails as soon as the task requires a change in viewpoint.

    The information received by a vision-language model is also often distributed across multiple images, image regions, or textual descriptions. The model must combine these local observations into a complete spatial structure. Without an explicit internal planning mechanism, it may identify the individual objects correctly without establishing a unified coordinate relationship among them.

    3. What “Active Spatial Mental Modeling” Solves

    Active spatial mental modeling can be understood as the ability to select a reasonable observation point based on the question and internally simulate the environment from that position.

    It consists of two closely related components.

    Viewpoint Planning

    Viewpoint planning answers the question: “Where should I stand, and which direction should I face to solve this problem?”

    This is not governed by a fixed rule. The question determines the appropriate viewpoint. For example:

    • To determine an object’s left-right position relative to a person, the person should be the reference point.
    • To determine what lies ahead on a road, the observation axis should follow the direction of the road.
    • To determine the relationship between two rooms, the model may need to consider entrances, doors, windows, and possible movement paths.
    • To compare the same object across two images, the model must first establish cross-image entity correspondence.

    Without viewpoint planning, a model may incorrectly use the camera angle, the image boundary, or the order of a textual description as its reference frame.

    Viewpoint Imagination

    Viewpoint imagination does not require generating a completely realistic new image. Instead, the model constructs a spatial representation that is sufficient for reasoning. For example, it may need to represent one object as slightly to the left and in front of the observation point, and another as behind it. It does not need to reconstruct every texture or visual detail if the directional relationship is preserved.

    This objective differs from image generation. Image generation emphasizes visual detail and appearance, while viewpoint imagination emphasizes geometric relationships, directional consistency, and information completeness. For spatial question answering, a detailed imagined image with incorrect directions is useless. A simplified scene representation with correct coordinates is more valuable.

    4. From Direct Answers to Spatial State Construction

    Improving spatial reasoning requires training objectives that cover intermediate steps, not just the question and final answer.

    A more useful training example could include:

    • The key entities in the scene.
    • The position of each entity.
    • The camera’s viewing direction.
    • The observer specified by the question.
    • How the observer’s orientation is determined.
    • Which information comes from the first image and which comes from the second.
    • What viewpoint transformation is required.
    • The target object’s direction after the transformation.
    • The final answer and the evidence supporting it.

    Process-based annotations can teach models how to reach an answer instead of merely memorizing common outputs for a class of questions.

    However, asking a model to generate its own reasoning process also introduces risks. It may invent doors, windows, or pieces of furniture that do not exist, or produce contradictory descriptions of direction. Training data therefore needs consistency checks.

    One general validation method is to convert spatial relationships into a structured representation. A scene can be abstracted into two-dimensional coordinates:

    1. Person: (0, 0)
    2. Chair: (0, 0)
    3. Computer: (2, 1)
    4. Orientation: North

    In this coordinate system, the computer is in front of and slightly to the right of the person. If the person turns around and faces south, the same computer may appear behind and slightly to the left in the new observer-centered coordinate system. Coordinate transformations can be used to verify whether a natural-language explanation is consistent with the underlying geometry.

    A production system does not necessarily need to expose every intermediate step to the user. Internally, however, it needs a spatial state that can be checked. Otherwise, direction errors can accumulate during long reasoning chains.

    5. Cross-Image Reasoning Requires a Shared Reference Frame

    Tasks involving multiple images are more difficult because each image may have a different camera position and orientation.

    If the first image shows a room entrance and the second shows the interior, a model cannot simply place the two images side by side. It needs to determine:

    • Whether the images depict the same scene.
    • Whether the images contain overlapping regions.
    • Which objects can serve as shared landmarks.
    • Whether the camera’s position or orientation has changed.
    • Whether a change in an object’s image position is caused by a viewpoint change or by actual movement.

    Shared landmarks are especially important. Stable structures such as door frames, fixed tabletops, columns, and windows can help the model establish correspondence between images. After alignment, local observations can be converted into a unified scene map.

    The same approach applies to video understanding. Adjacent video frames may show similar content, but camera movement can continuously change the positions of objects in the frame. If a model maintains relatively stable spatial memory, it can distinguish camera motion from object motion and improve its understanding of continuous events.

    6. Designing More Reliable Training and Evaluation

    Spatial reasoning should not be evaluated only by final answer accuracy. A model may occasionally produce the correct answer without actually learning how to transform viewpoints.

    A more comprehensive evaluation can cover several dimensions:

    1. Entity recognition accuracy
      Can the model locate the people, objects, and regions mentioned in the question?

    2. Reference-frame understanding
      Does the model understand who or what defines left, right, front, and behind?

    3. Viewpoint transformation
      Can the model update spatial relationships when the observer turns, moves, or changes position?

    4. Cross-image alignment
      Can the model identify the same entity across different images and build a unified scene?

    5. Inference about unseen regions
      Can the model infer reasonable relationships outside the visible portion of the image based on known structure?

    6. Reasoning consistency
      Are the model’s intermediate spatial descriptions consistent with its final answer?

    Evaluation data should also cover multiple levels of difficulty. Basic tasks may require only a single left-right judgment. Intermediate tasks can introduce changes in the observer’s orientation. More complex tasks may combine multiple images, occluded regions, and several successive viewpoint transformations.

    It is also important to identify systematic biases. Some models default to using the camera as the reference point. Others map the left and right sides of the image directly to the observer’s left and right, or ignore key conditions such as “facing” and “with their back to.” Targeted tests can reveal the specific stage at which the model fails.

    7. Why This Matters for Agents and Robots

    Active spatial mental modeling is not limited to visual question answering. It is a foundational capability for intelligent systems that interact with their environments.

    In robotics, a system must distinguish between “the object is to my left” and “the object is to the person’s left.” These statements may imply entirely different movement paths. During indoor navigation, a robot also needs to update its map according to its own orientation rather than relying only on the left and right positions in the camera feed.

    In assistant applications, a user might say, “Bring me the file behind me.” The system must build a user-centered spatial model and understand that “behind me” is not a fixed region of the image.

    Augmented reality and virtual environments present similar requirements. Different observers need to share consistent spatial relationships. An object may be in front and to the right of one user, but behind and to the left of another. Without viewpoint transformation, natural collaboration among multiple users is difficult.

    8. What Spatial Reasoning Reveals About the Future of Multimodal Models

    Spatial reasoning highlights an important fact: improving multimodal models cannot depend solely on larger parameter counts and more image-text data.

    Complex tasks require internal state management, coordinate transformations, relational memory, and process verification. Future multimodal systems may adopt a more explicit modular structure:

    • A perception module identifies entities and environmental cues.
    • A scene-modeling module maintains objects and their relationships.
    • A viewpoint-planning module selects the observation reference.
    • A spatial-reasoning module performs coordinate transformations.
    • A language module explains the result and communicates with the user.

    This does not mean every system must use the same fixed modules. It does show that “what is visible” and “how to understand it from a new perspective” should be treated as separate capabilities.

    From an engineering perspective, building spatial reasoning also requires structured data, simulation environments, geometric constraints, and verifiable evaluation. Models need repeated exposure to different viewpoints, scenes, and continuous interactions to reduce errors caused by superficial pattern matching.

    Ultimately, teaching a vision-language model to understand space is not about making it memorize more answers involving “left” and “right.” It is about enabling the model to move its observation point when necessary, establish a new reference frame, and reconstruct an internal scene representation that supports reliable judgment.

    This is a shift from passive recognition to active modeling—and a crucial step in moving multimodal AI from “understanding images” to understanding environments.