Visual Primitives: Bridging the Reference Gap in Multimodal Large Models

    This article examines why multimodal large models still struggle with dense counting, precise small-object localization, and spatial reasoning even when given high-resolution images. It distinguishes two layers of bottlenecks—the perception gap and the deeper reference gap—and explains how visual primitives (points, boxes, and polygons) act as unambiguous, computable pointers over continuous visual space. It covers how these primitives are registered as special tokens in the tokenizer, why models often drop the explicit syntax and emit coordinates directly during inference, how placeholder tokens align text with image features in mixed layouts, and the tradeoffs developers must weigh around token budgets and consistency checks. The goal is to help engineers build multimodal applications that not only see clearly but also refer accurately.

    1. The Starting Point: Why Models Still “Can’t See Clearly”

    A recurring pattern shows up in real-world deployments of multimodal large models: even when fed high-resolution images, they perform poorly on tasks like dense object counting, precise localization of small objects, and complex spatial-relationship reasoning. This raises a question worth digging into—where exactly is the bottleneck that limits model capability?

    The industry has gradually converged on a view: the bottleneck often lies not in the pixel layer but in the semantic layer. It helps to break a multimodal model’s capability into two stages:

    • Perception: whether the model can extract enough visual information from an image (resolution, feature representation, encoding quality);
    • Expression: whether the model can stably and unambiguously refer to objects in the image while “thinking” and producing output.

    Plenty of practice shows that once perception is adequately addressed (for example, by raising input resolution or improving the visual encoder), the remaining performance loss occurs mainly in the expression stage.

    2. Two Gaps: The Perception Gap and the Reference Gap

    Research on this problem has produced two progressively deeper judgments.

    2.1 The Perception Gap

    This is the most surface-level issue: the model “can’t see the image clearly.” Common causes include over-compressed input images, an insufficient number of visual tokens, and low encoder resolution. Such problems are relatively easy to mitigate through engineering means—for instance, increasing the effective visual-token budget or introducing multi-scale features.

    2.2 The Reference Gap

    This is the deeper and thornier problem. Even when the model “sees the image clearly,” it still relies on natural language to describe image content during reasoning. Natural language is inherently ambiguous:

    • When several similar objects appear in a scene, descriptions like “that cat” or “the person on the left” cannot uniquely identify a target;
    • In dense scenes, terms like “many” or “several” don’t constitute any computable quantity;
    • Once spatial relations get complicated, “the one near the bottom-right corner” easily leads to wrong binding.

    As a result, the model falls into an awkward loop: it uses vague, human-intuition natural language to point at something that should be an exact coordinate, so downstream logical reasoning (counting, comparison, localization) goes wrong at the very first step of “reference,” and errors are amplified level by level.

    Core insight: the key to closing the reference gap is to give the model an “unambiguous pointer” within continuous visual space, sidestepping the vagueness of natural language.

    3. The Core Idea of Visual Primitives

    Visual primitives are, in essence, a system of “precise pointers” designed for visual space. They replace vague natural-language references with a set of structured, computable symbols, letting the model “anchor” objects or regions in an image directly—much like passing arguments to a function call.

    A typical visual-primitive system usually contains the following basic forms:

    Primitive type Semantic meaning Role
    Point 2D coordinate (x, y) Points to a specific location or an object’s center
    Box Top-left/bottom-right or center + width/height Encloses a rectangular target region
    Polygon An ordered list of vertices Describes the precise outline of an irregular shape

    In terms of symbol design, these primitives are typically wrapped in paired “open/close” markers around their content, forming fragments that resemble a structured language. Represented schematically:

    1. <|point|>[x, y]<|/point|>
    2. <|box|>[x1, y1, x2, y2]<|/box|>
    3. <|polygon|>[[x1,y1],[x2,y2],...]<|/polygon|>

    Here, open and close markers appear in pairs, and the numbers inside are normalized coordinates. The value of this approach lies in three points:

    1. Eliminates ambiguity: a coordinate is unique, so there’s no “which cat” problem;
    2. Computable: coordinates can directly participate in arithmetic, naturally supporting reasoning such as counting, distance, and area;
    3. Trainable: by adding primitives to the vocabulary as special tokens, the model can learn “how to point at an image” much like it learns ordinary language.

    4. Engineering Implementation: How Special Tokens Enter the Vocabulary

    The first step in making a model understand visual primitives is getting the tokenizer to recognize these symbols. Such systems typically register a set of special tokens in the vocabulary as the grammatical skeleton for the primitives. Shown here as a schematic structure (in a real vocabulary, each entry usually also carries several boolean flags that control tokenization behavior, such as whether it stands alone as a word or is included in special handling):

    1. id token description
    2. 12001 <|point|> point primitive start
    3. 12002 <|/point|> point primitive end
    4. 12003 <|box|> box primitive start
    5. 12004 <|/box|> box primitive end
    6. 12005 <|polygon|> polygon primitive start
    7. 12006 <|/polygon|> polygon primitive end
    8. 12007 <|ref|> reference start
    9. 12008 <|/ref|> reference end

    Key design points here include:

    • Paired grammar: open/close markers are paired so the structure can be reliably parsed, similar to the boundaries in JSON;
    • Flag-bit control: the flags carried by each token (for example, whether to split, whether it’s special) determine its behavior during preprocessing and decoding, preventing coordinate digits from being chopped up by ordinary tokenization rules;
    • Reference binding: markers like <|ref|>...<|/ref|> further support “referential relationships,” which can express semantics such as “A and B are the same object” or “reference the Nth candidate in the image.”

    5. A Notable Phenomenon: From Explicit Primitives to Coordinate-Based Descriptions

    An intriguing observation is this: even though explicit visual-primitive tokens are registered in the vocabulary, during actual inference the model’s chain of thought rarely emits these paired markers directly. Instead, we frequently see the model give coordinates outright when describing objects—for example, “the second bird in the image is near (0.42, 0.67).”

    This points to a possible evolution path:

    • Early training: the model learns, via supervised data, to refer to objects using explicit structures like <|point|>[x,y]<|/point|>;
    • Later training: the model gradually internalizes the semantics of the primitives, no longer needing the explicit grammatical shell, and instead embeds coordinates directly into the natural-language stream in its output.

    Put differently, visual primitives may function more like a “training scaffold”—during training they help the model build the mapping between visual space and numeric coordinates, and once this capability is internalized, the explicit token shell “steps into the background,” with the model turning to more natural coordinate descriptions to accomplish reference.

    The takeaway for developers is this: to assess whether a multimodal model has truly mastered visual localization, don’t just check whether its vocabulary contains primitive tokens—look at whether it can stably and accurately output coordinates during free generation. The former is form; the latter is capability.

    6. The Placeholder Mechanism in Mixed Text-and-Image Layouts

    Another engineering challenge for multimodal models is mixed text-and-image layouts: when multiple images are interspersed within a passage of text, how do you get the model to correctly align the “surrounding text” with the “corresponding image tokens”?

    A common approach is to insert placeholder tokens into the sequence, marking “which image’s visual features should be embedded here.” These tokens usually appear as indexed placeholders, as shown schematically:

    1. ... please compare the two images below ... <placeholder_mm_span_0001> ... and ... <placeholder_mm_span_0002> ... and describe their differences ...

    The working mechanism is: the text sequence first “reserves space” with placeholders, and before the actual forward pass, the multimodal assembly logic “backfills” the encoded visual features of the corresponding images into the positions where the placeholders sit.

    There are several engineering issues here that deserve careful weighing:

    1. The correspondence between placeholder count and image count
      Intuitively, one image maps to one placeholder. But if an image is large and produces many visual tokens, engineering practice may require multiple placeholder spans to represent an image’s multiple feature fragments. At that point, the total number of placeholders no longer equals the total number of images, and you need to maintain a mapping table during preprocessing.

    2. Per-image visual-token budget
      How many visual tokens each image is encoded into is a key parameter that directly affects memory usage and inference speed. A higher budget yields richer detail but also higher compute and storage costs; too low a budget may reintroduce the “can’t see clearly” perception gap. The value of this budget calls for a fine-grained tradeoff between accuracy and cost, rather than simply maximizing it.

    3. Matching placeholder capacity to image capacity
      When the total number of available placeholder IDs, the number of spans per image, and the per-image token budget don’t line up with one another, you get the engineering confusion of things “not adding up.” The sensible approach is to make the constraints among these three quantities explicit at the system-design stage and to validate them strictly during preprocessing, avoiding sequence overflows or mapping errors.

    7. Practical Advice for Developers

    Drawing the above together, here are some recommendations for actually landing visual-localization capabilities:

    • Prioritize perception first: confirm whether the visual-token budget and resolution are sufficient, so you don’t misdiagnose “can’t describe it” as “can’t see it”;
    • Evaluate with coordinates: when designing evaluation sets, directly measure the accuracy of the coordinates a model outputs (such as IoU or hit-point error), rather than only looking at language fluency;
    • Use explicit structure to aid training: introducing paired structured primitives as a scaffold during training helps the model build the visual-to-numeric mapping faster;
    • Validate placeholders: in mixed text-and-image scenarios, always run consistency checks on placeholder counts, spans per image, and token budgets to prevent alignment errors;
    • Keep costs controllable: the visual-token budget directly affects inference cost, so consider tiered configurations aligned with business accuracy requirements.

    8. Summary

    The “visual challenge” of multimodal large models is essentially a relay from “perception” to “reference.” The core contribution of visual-primitive technology is that it provides the model with a system of unambiguous, computable, trainable pointers over continuous visual space, thereby sidestepping natural language’s inherent defects in reference. Meanwhile, the evolution from explicit special tokens to coordinate-based descriptions reveals how this kind of capability is internalized “from surface to depth.” For developers, understanding these two gaps—and grasping the design intent behind special tokens and placeholders—is what makes it possible to act with precision in engineering practice and build multimodal applications that truly “see clearly and refer accurately.”