How Visual Primitives Reshape Localization and Reasoning in Multimodal Models

    This article examines the role of visual primitives in multimodal models, focusing on the gap between recognizing image content and reliably referring to specific objects. It explains what visual primitives are, why they may appear as special tokens in a tokenizer, and why they are not always visible in model outputs. It also discusses coordinate-based representations, multimodal placeholders, validation experiments, and practical engineering challenges. Readers will learn how to evaluate whether a model uses structured visual references internally, how to distinguish tokenizer artifacts from actual spatial reasoning ability, and what to consider when designing more reliable multimodal systems.

    1. From “Seeing an Image” to Accurately Referring to Objects

    The capabilities of multimodal models can usually be divided into two stages: first, extracting visual information from an image; second, using that information for understanding, reasoning, and expression.

    In the past, recognition failures were often attributed to insufficient image resolution, limited visual encoder capability, or objects that were too small. These factors can indeed cause perception errors. However, even when the model has correctly recognized the image content, another more subtle difficulty remains: the model knows what is in the image but cannot stably and precisely indicate where that specific object is.

    For example, when a model needs to answer “What is the second object from the left in the image?”, it must handle object recognition, spatial relations, order relations, and linguistic reference at the same time. If multiple similar objects appear in the image, natural-language descriptions such as “the cup on the left” or “the object near the corner of the table” can easily become ambiguous. This problem becomes more obvious in dense counting, object localization, region judgment, and multi-object relational reasoning.

    Therefore, multimodal systems face not only a “perception gap” but also a “reference gap”:

    • Perception gap: the model does not correctly see or recognize the image content.
    • Reference gap: the model recognizes the object but cannot represent it in a stable and unambiguous way.
    • Reasoning gap: the model can localize the object but cannot effectively incorporate spatial information into subsequent reasoning.

    Visual primitives mainly target the second problem, while also providing a new intermediate representation for the third.

    2. What Exactly Are Visual Primitives?

    Visual primitives can be understood as a set of structured symbols that connect continuous visual space with discrete language reasoning space. They are not ordinary natural-language words. Instead, they are special markers used to express object positions, region ranges, and geometric shapes in an image.

    Common visual primitives include the following:

    1. Points: used to represent key points, center points, or a precise location.
    2. Bounding boxes: used to represent the approximate rectangular range of an object.
    3. Polygons: used to represent the contours or segmentation regions of irregular objects.
    4. Reference markers: used to establish correspondences between image objects and text descriptions.
    5. Region spans: used to represent placeholder ranges for image features in a sequence.

    Formally, a visual primitive usually contains a start marker, coordinate data, and an end marker. For example, a system may internally use an abstract structure such as:

    1. <point> x, y </point>
    2. <box> x1, y1, x2, y2 </box>
    3. <polygon> x1, y1; x2, y2; ... </polygon>

    The key point here is not the specific marker names, but the structured expression role they play. Compared with “the object is in the upper-left area of the image,” coordinates or geometric regions are more deterministic and easier for programs to parse, compare, and reuse.

    3. Why Special Tokens Appear in the Tokenizer

    In models that support visual primitives, the relevant markers usually need to be registered as special tokens. The reason is that an ordinary tokenizer may split a structured marker into multiple fragments, making it difficult for the model to recognize its boundaries and semantics.

    For example, if a bounding-box marker is split into several ordinary characters, the model must additionally learn:

    • Which characters indicate the start of the structure;
    • Which characters indicate the end of the structure;
    • How coordinate fields are arranged;
    • How multiple coordinates are distinguished;
    • How structured content is associated with natural-language context.

    Once defined as special tokens, the model can directly recognize the structure at the sequence level, reducing the cost of learning formats and improving the consistency of training samples.

    Observing point, box, polygon, and reference-related markers in a token table usually suggests that the model designers have reserved structured visual expression capabilities. However, it is important to note that the existence of tokens does not mean that these tokens will necessarily be explicitly output during inference. It only shows that the model has vocabulary entry points for representing these structures. It cannot by itself prove that these markers are invoked in every generation.

    This distinction is important. A model’s vocabulary may contain many data structures used for training, alignment, internal sequence assembly, or special tasks, and they are not necessarily all exposed in the final answer.

    4. Why Visual Primitives May Be Invisible in Actual Outputs

    In real tests, a model’s reasoning process or final answer may show only ordinary coordinates, natural-language descriptions, or no visual primitive markers at all. This phenomenon does not necessarily contradict the design of visual primitives. It may be caused by the following factors.

    4.1 Reasoning and Display Are Not the Same

    A model may use structured representations internally, but convert the format when producing external output. For example, it may first generate an object box internally and then convert it into natural language such as “the object is located in the coordinate region…”. What users see is the result layer, not the intermediate layer.

    This is similar to a compiler using an abstract syntax tree internally while ultimately returning only the program’s runtime result. The existence of an internal structured representation does not mean that all intermediate data will be printed directly.

    4.2 Post-Training Changes Expression Habits

    The pretraining stage may require the model to learn explicit visual markers, while the instruction-tuning stage places more emphasis on natural and readable answers. After subsequent training, the model may learn to map special markers to ordinary coordinates or text descriptions, thereby reducing direct dependence on special formats.

    This is a common phenomenon in which “capability is retained while surface form changes”: structured capability may still exist, but external expression is repackaged.

    4.3 Visual Primitives May Be Used for Internal Supervision

    Visual primitives are not necessarily the final generation target. They may also be used for training labels, auxiliary losses, or cross-modal alignment. For example, training data may require the model to learn correspondences between image regions and text phrases, while only answers are output during deployment. In this case, special markers mainly serve as supervision signals rather than user-visible output.

    4.4 the Interface Layer May Perform Conversion

    A multimodal system usually consists of a visual encoder, a language model, reasoning templates, and an output parser. Even if the language model generates structured content, the outer interface may convert it into plain text or retain only coordinate values. Therefore, test outputs cannot directly represent the full internal representation of the model.

    5. Do Coordinate Descriptions Mean Visual Primitives Have Disappeared?

    It cannot be judged that simply. Coordinates themselves are a kind of visual primitive; they may simply not take the form of explicit special markers.

    Visual primitives can be divided into three implementation levels:

    • Explicit marker type: uses dedicated start and end symbols to wrap the structure.
    • Sequence-constrained type: expresses points, boxes, or regions through a fixed field order.
    • Semantically implicit type: the model outputs natural language, but it contains parseable coordinate information.

    Explicit markers are convenient for program parsing and format validation, while implicit expressions are more consistent with human reading habits. Engineering implementations often compromise between the two: structured information is retained inside the model, while a more natural textual form is used for users.

    To determine whether a model truly has visual primitive capability, it is not enough to check whether special symbols appear in the output. The following capabilities should be verified:

    1. Whether it can stably localize the same object;
    2. Whether it can distinguish multiple similar objects;
    3. Whether it can output a consistent coordinate format;
    4. Whether it can perform subsequent relational reasoning based on object boxes;
    5. Whether it can maintain spatial consistency in counting, ordering, and region-judgment tasks;
    6. Whether it can accurately map text descriptions back to image regions.

    6. Placeholder Mechanisms in Interleaved Image-Text Inputs

    In addition to point, box, and polygon markers, a vocabulary may also contain many multimodal placeholders with similar forms. Such markers are usually used to connect text sequences with visual feature sequences.

    An image is not simply inserted into a language model as a character. It must first be processed by a visual encoder and converted into a set of visual features. These features are then inserted into the text context through some placeholder structure, allowing the language model to process text and images within a unified sequence.

    A common misunderstanding can arise here: the number of placeholder indices does not necessarily equal the number of images, nor does it necessarily equal the number of visual tokens corresponding to a single image. It may represent:

    • Image-text slots at different positions;
    • Different resolutions or cropped regions;
    • Indices of visual feature segments;
    • Reserved spans in training templates;
    • Sequence separation units for multi-image inputs;
    • Some internal layout or alignment structure.

    Therefore, the maximum number of supported images cannot be inferred solely from the index range, nor can the number of placeholders be directly equated with the output length of the visual encoder. To understand their meaning accurately, it is necessary to analyze the model input template, visual encoding pipeline, sequence assembly method, and inference interface together.

    7. How to Design More Reliable Validation Experiments

    To determine whether visual primitives are actually used internally or merely exist in the vocabulary, layered experiments can be designed.

    Layer 1: Format Experiment

    Construct an image containing a single obvious object, ask the model to output the object position, and repeat the test multiple times. Observe whether the coordinate format is stable, whether the field order is consistent, and whether there is obvious format drift.

    Layer 2: Disambiguation Experiment

    Place multiple visually similar objects in the image and ask the model to localize one of them. Test whether the model can accurately refer to the target based on conditions such as color, numbering, and spatial relations.

    Layer 3: Compositional Reasoning Experiment

    Ask the model to first localize an object and then complete counting, distance comparison, area comparison, or relative-position judgment. If the model can only output coordinates but cannot use them to complete reasoning, this indicates that a gap remains between localization capability and reasoning capability.

    Layer 4: Format Perturbation Experiment

    Change the way the question is phrased. For example, separately ask for a natural-language position, coordinates, a bounding box, and a region description. If the results are highly consistent across different expression forms, this suggests that a relatively stable spatial representation may exist inside the model.

    Layer 5: Program Parsing Experiment

    Pass model outputs to a parsing program and check whether coordinates are out of bounds, whether the top-left corner of a box always precedes the bottom-right corner, whether polygons are closed, and whether coordinates are consistent across multiple rounds of answers.

    8. Remaining Engineering Challenges

    Visual primitives are not the endpoint for solving multimodal problems. Several difficulties remain in practical applications.

    First, the coordinate normalization method must be explicit. Different systems may use pixel coordinates, percentage coordinates, or normalized coordinates. Without information such as image width and height, coordinate origin, and axis direction, coordinates are difficult to reuse reliably.

    Second, the precision of visual primitives is related to image resolution. For small objects, occluded objects, and objects with blurred boundaries, relying only on rectangular boxes may be insufficient. Multiple representations such as points, boxes, and polygons need to work together.

    Third, structured output requires strict grammatical constraints. If brackets are missing, field order changes, or numeric types are incorrect, downstream programs may fail to parse the output. Therefore, production systems usually need to add format validation, automatic repair, and exception fallback mechanisms.

    Finally, models need to balance “machine-readable” and “human-readable” output. Using structured markers entirely makes program processing easier but harms readability; using natural language entirely may sacrifice precision. A more reasonable approach is to keep internal representations structured while selecting output formats according to the scenario.

    9. Conclusion

    The core value of visual primitives is not merely adding a few special tokens. Rather, they establish a stable spatial reference mechanism for multimodal models. They transform continuous regions in an image into discrete structures that the model can operate on, giving object localization, dense counting, relational reasoning, and image-text correspondence a clearer intermediate representation.

    When analyzing such models, three levels should be distinguished: whether the relevant markers are reserved in the vocabulary, whether these structures are used during training, and whether the inference interface explicitly exposes them. These three are not equivalent.

    Even if the final answer shows only ordinary coordinates, this cannot be used to conclude that visual primitives are not functioning. Likewise, merely finding special tokens cannot prove that the model already has reliable spatial reasoning capability. Truly valuable judgment should be based on repeatable experiments involving localization, disambiguation, compositional reasoning, and format consistency.

    In the long run, the development direction of multimodal models is not only “seeing more clearly,” but also “referring more accurately, remembering more stably, and reasoning more verifiably.” Visual primitives are an important technical link connecting perception, language, and spatial reasoning.