Multimodal Visual Primitives: From Special Tokens to Real-World Testing

    This article examines whether a multimodal model that exposes special markers like `<|point|>`, `<|box|>`, and `<|polygon|>` in its tokenizer is actually equipped with visual primitives—precise, coordinate-based pointers that let a model reference exact locations in an image. It explains the perception gap and the reference gap that visual primitives are meant to close, introduces the three core primitives, and lays out a three-layer verification approach: static inspection of the tokenizer and chat template, structural parsing tests, and real visual-task testing across dense counting, small-target localization, irregular region segmentation, and spatial path planning. It also covers cross-entry consistency checks, a sample test-case schema and scoring script, and the role of the host tool (harness) in the last mile of visual capability, closing with an evaluation checklist and practical recommendations for teams assessing multimodal models.

    After an open-source multimodal model was released, the community spotted special markers such as <|polygon|>, <|point|>, and <|box|> in its tokenizer configuration. These markers map one-to-one to the definitions of polygons, points, and boxes used in “visual primitives” research. The clue quickly raised a key question: has the model actually been equipped with visual primitives?

    The basis for judgment can be gathered layer by layer: start with the tokenizer and chat template, then return to real tasks and test them. The key to verifying effectiveness is whether the model can work along the “annotate – re-observe – correct” visual reasoning loop. At the same time, whether the model’s host tool can feed images into the conversation history often directly determines whether visual capability can be brought into play. This article walks through the principles of visual primitives, a layered verification method, the role of the host tool, and practical implementation, so readers can build a reproducible approach to evaluating multimodal capabilities.

    1. Why Visual Primitives Deserve Repeated Scrutiny

    When evaluating multimodal models, “being able to see images” has long been a basic threshold. The reason the community is so sensitive to a few special markers in a tokenizer file is that they touch on a deeper question: after seeing clearly, can the model point accurately?

    1.1 Two Gaps: The Perception Gap and the Reference Gap

    Discussions of related problems are usually summarized as two gaps.

    • Perception Gap: Limited by resolution, tiling strategy, and visual encoding capability, the model only sees a limited amount of image detail. When targets are too small, text is too dense, or similar objects overlap, the model “can’t see it all.”
    • Reference Gap: Once the model takes a target into view, at the reasoning stage it can only rely on natural language to describe it. Saying “the person in red in the upper right” may still be workable, but in scenarios like crowd counting and small-target localization, textual reference becomes increasingly ambiguous.

    The perception gap determines how much can be seen, while the reference gap determines how precisely it can be used after being seen. The former relates to image resolution and tiling strategy—an input-side “seeing clearly” problem. The latter is a “describing clearly” problem at the reasoning stage. Once the reference becomes vague, the logical chain is distorted along with it.

    1.2 Two Weak Points of the Traditional Paradigm

    Following the classic “image captioning” paradigm, a model faces a visual task in roughly three steps: first perceive the whole image, then restate in text what it saw, and finally complete counting, comparison, and reasoning in text space. This flow seems natural, yet it has two weak points.

    • Coordinate expression is flattened: Dense targets require positions precise to individual objects, and a sentence of text often cannot uniquely pin down a referent.
    • The reasoning chain is hard to bring back to the image: Once the model needs to verify details, it can only regenerate the entire description. The existing context grows longer and longer, errors are continuously amplified, and adjacent targets end up merged or double-counted.

    Discrepancies between paper examples and actual behavior are a common product-side manifestation of these problems: the same set of example cases runs stably in a paper or a demo environment, but shows count deviations or planning failures when moved to a real entry point. The cause may come from pre- and post-processing, the chat template, the rendering pipeline, or a host tool that never wired up the “see – annotate – look again” loop.

    2. Visual Primitives: Letting Models “Think by Pointing at the Image”

    The concept of visual primitives comes from research on visual reasoning based on visual primitives. Its core is to turn “where” from a vague textual description into a precise coordinate expression—effectively providing an unambiguous pointer for an object within continuous visual space. It usually comprises three basic operations.

    Visual Primitive Expression Typical Use
    Point A pair of coordinates representing a position or center Counting, anchor localization, key-point annotation
    Box Two points—top-left and bottom-right—defining a rectangular region Framing targets, local zoom and cropping
    Polygon An ordered sequence of vertices forming a closed contour Irregular region segmentation, range constraints

    In practice, these primitives usually enter the conversation as coordinate strings wrapped in special markers, forming the model’s input together with image content blocks. Using a schematic coordinate format as an example:

    1. <|box|>[120, 80, 260, 240]<|/box|> # top-left and bottom-right
    2. <|point|>[512, 300]<|/point|> # a single position
    3. <|polygon|>[80, 300, 200, 260, 330, 320, 180, 400]<|/polygon|>

    Note: The coordinate format and value range (pixel or normalized coordinates) depend on the specific model’s convention. The above is only for illustrating the structure.

    The value of visual primitives is that they turn spatial reference in visual tasks into a “language” the model can operate step by step:

    1. Annotate: first mark candidate targets with points;
    2. Frame: when closer inspection is needed, use a box to circle the suspicious region and zoom in;
    3. Contour correction: when boundaries are irregular, use a polygon to correct the outline.

    The reasoning chain no longer contains only text, but also precise pointing into visual space. The model can “look” while it thinks, and the original reference gap is compressed into a matter of coordinate precision.

    This is two sides of the same coin as the interactive idea of “manipulating images during thinking.” The latter provides a “see – operate – re-observe” closed loop, such as cropping and zooming or adding annotations; visual primitives provide precise, parseable input for each step. Only when combined does the model truly gain a set of image-facing “actions.”

    3. Layered Verification: Has the Model Truly Equipped Visual Primitives?

    Back to the initial question: do special markers in the tokenizer prove that visual primitives are already in place? They can serve only as a clue, not a conclusion. A three-tier verification, progressing from static to dynamic, is recommended.

    3.1 Layer 1: Statically Checking the Tokenizer and Chat Template

    The tokenizer configuration is the “dictionary” for model input processing. If visual markers like polygon, point, and box are registered there, it means these markers were incorporated as independent symbols into the training or inference flow, rather than being random strings decomposed byte by byte.

    1. # Schematic code: check whether visual primitive markers are registered in the tokenizer
    2. VISUAL_MARKERS = [
    3. "<|polygon|>", "<|/polygon|>",
    4. "<|point|>", "<|/point|>",
    5. "<|box|>", "<|/box|>",
    6. ]
    7. def check_visual_markers(tokenizer):
    8. # get_added_tokens here is only for illustration;
    9. # the actual interface varies by tokenizer implementation. What matters is whether markers are recognized as independent tokens as a whole.
    10. known = tokenizer.get_added_tokens()
    11. registered = [m for m in VISUAL_MARKERS if m in known]
    12. print(f"Registered {len(registered)}/{len(VISUAL_MARKERS)} visual markers: {registered}")
    13. return len(registered) == len(VISUAL_MARKERS)

    There are two misconceptions to avoid at this step:

    • Treating “existence” as “effectiveness”: Files in a public repository are not necessarily fully consistent with the live service. Some markers may only be reserved placeholders so the tokenizer can parse special input, and the model may never have been trained on them specifically.
    • Treating “absence” as “unsupported”: Some approaches don’t rely on special markers and instead let the model output structured coordinates directly, such as point arrays parseable as JSON. So a missing static check cannot be grounds for a negative verdict.

    You should also check the chat template and coordinate conventions: in what content block the image is passed in, whether coordinates are pixel values or normalized values, whether the origin sits at the top-left corner or the center, and whether the canvas size must be declared explicitly. Unclear conventions are a common source of coordinate offsets during testing.

    3.2 Layer 2: Structural Parsing Test

    Construct input with known coordinate markers and observe whether the model’s interpretation is self-consistent.

    1. User: On the given image, I've annotated the following:
    2. <|point|>[512, 300]<|/point|> marks A's reference point,
    3. <|box|>[300, 200, 700, 500]<|/box|> marks B's bounding box.
    4. Please determine whether A is inside B, and explain the basis.
    • A model with primitive understanding: it compares the point coordinates against the box’s range numerically, usually adding a note on boundary determination.
    • A model guessing with natural language only: it often avoids specific numbers or gives a vague answer unrelated to the coordinates.

    On top of this, you can further ask “re-annotation” style tasks, such as “move A’s point 50 pixels to the left and re-output.” Coordinates that can be correctly rewritten are the “pointer” the model can truly operate.

    3.3 Layer 3: Real Visual-Task Testing

    Structural parsing only verifies whether the model can use primitives; real tasks verify whether they are used accurately. At minimum, cover the following four task types:

    Task Type Test Focus Pass Criteria (Example)
    Dense counting Many, small, partially occluded elements; require marking first, then giving the total Total-count error is controllable; check points land on targets, with low duplicate and miss rates
    Small-target localization Given an attribute combination (e.g., “the third red cone from the left”), output the center point Coordinates fall within the target range and stay stable across repeated runs
    Irregular region segmentation Output a polygon of the enclosing shape Overlap with the reference contour meets expectations
    Spatial path planning In an environment with obstacles, give a route from start to end Route is parseable, does not intersect obstacles, and has reasonable length

    These cases correspond exactly to the dense counting and maze navigation mentioned in public discussions. Note that paper examples often present only success cases with a limited distribution, which does not represent the capability’s lower bound. Testing should construct harder cases on your own.

    You can also design scenarios that require “zoom in first, then look”: the image contains densely packed small objects, a few of which carry subtle differences, such as a sticker pattern, a number, or a slight color shade.

    • Visual-primitives paradigm: the model should first frame the local area with a box, then zoom in before making a judgment.
    • Pure-text paradigm: the model is more likely to give a vague answer over the whole image, or to mistake adjacent objects for the same one.

    Cross-Entry Consistency Check

    The paper examples, official entry point, coding tools, and other entry points for the same model may use different pre- and post-processing and runtime configurations, so outputs are often inconsistent. A headcount that is accurate in the paper example may deviate when moved to the actual entry point, and a maze-navigation route that works in the demo may fail at the actual entry point—both are problems of this kind.

    It’s recommended to fix the case set, prompts, and parsing rules, then run the same tests entry by entry. Only when performance is consistent across entry points can the capability be considered truly delivered.

    Organizing Test Cases

    1. {
    2. "id": "counting-001",
    3. "task": "dense_counting",
    4. "image": "fixtures/crowd_001.png",
    5. "prompt": "Please first mark each person in the image, then give the total number.",
    6. "reference": { "count": 25 },
    7. "metrics": {
    8. "count_error_rate": 0.05,
    9. "marker_precision": 0.9,
    10. "marker_recall": 0.9
    11. }
    12. }

    Test scripts can be organized around a “parse – match – score” flow:

    1. import re
    2. PAIR = {"polygon": ("<|polygon|>", "<|/polygon|>"),
    3. "point": ("<|point|>", "<|/point|>"),
    4. "box": ("<|box|>", "<|/box|>")}
    5. def parse_primitives(text):
    6. """Extract visual primitives from model output (illustrative implementation)"""
    7. primitives = []
    8. for kind, (begin, end) in PAIR.items():
    9. pattern = re.escape(begin) + r"(.*?)" + re.escape(end)
    10. for raw in re.findall(pattern, text, flags=re.S):
    11. # Illustration: convert the coordinate string into a numeric list
    12. coords = [float(x) for x in re.findall(r"-?\d+(?:\.\d+)?", raw)]
    13. primitives.append({"type": kind, "coords": coords})
    14. return primitives
    15. def point_in_box(point, box):
    16. """Determine whether a point falls inside an axis-aligned box"""
    17. x, y = point
    18. x1, y1, x2, y2 = box
    19. return min(x1, x2) <= x <= max(x1, x2) and min(y1, y2) <= y <= max(y1, y2)
    20. def score_counting(pred_points, reference_count, gt_points, tol=20):
    21. """Compute count error and localization metrics by matching predicted points against ground truth"""
    22. used = set()
    23. matched = 0
    24. for p in pred_points:
    25. nearest, dist = None, float("inf")
    26. for i, g in enumerate(gt_points):
    27. if i in used:
    28. continue
    29. d = ((p[0] - g[0]) ** 2 + (p[1] - g[1]) ** 2) ** 0.5
    30. if d < dist:
    31. nearest, dist = i, d
    32. if nearest is not None and dist <= tol:
    33. used.add(nearest)
    34. matched += 1
    35. precision = matched / len(pred_points) if pred_points else 0.0
    36. recall = matched / len(gt_points) if gt_points else 0.0
    37. count_error = abs(pred_points.__len__() - reference_count) / reference_count
    38. return {"precision": precision, "recall": recall, "count_error_rate": count_error}

    For engineering deployment, three more points deserve attention:

    • Rendering consistency: when drawing detection boxes, confirm whether the image has been scaled, cropped, or rotated. If the output coordinate system is inconsistent with the rendering coordinate system, “misaligned boxes” seen by eye may not be the model’s fault.
    • Result stability: sample multiple times and measure the pass rate rather than drawing conclusions from a single run.
    • Persisted records: store cases, raw outputs, and scores uniformly in object storage; use log services to retain evaluation records, and track regression trends with monitoring and alerting.

    4. Don’t Overlook the Host Tool: The “Last Mile” of Visual Capability

    Beyond static checks and testing, there is another easily overlooked but equally important clue: the host tool (harness) the model runs in—the runtime framework outside the model that handles session organization, tool dispatch, and context management. Many practitioners have had a similar experience: while testing a new model with an official coding tool, they got stuck for a long time at the setup step. The official documentation assumes users will call from a third-party entry and explains clearly how to fill in the model identifier, yet says almost nothing about how the official tool itself switches runtime configuration.

    Behind this “can’t find the switch” experience lies the inherent positioning of the host tool: it is a runtime framework built for coding-agent and agent scenarios, not a graphical product polished for everyday users. This is unsurprising; instead, it points to an important fact: in the release chain of a new model, the host tool serves as a “capability channel.”

    4.1 How the Host Tool Affects Visual Primitives

    When a model works inside an agent, what it sees is not just a raw image, but also the state of images across different steps. The host tool’s influence on visual primitives shows up in four areas.

    • Image passing-in: whether images returned by tool calls (such as page screenshots or chart renders) can enter the next conversation turn as image content blocks. If they are passed only as text or file paths, visual capability is cut off at the source.
    • History retention: whether images in multi-turn conversations get dropped by the context-trimming strategy. Without them, the “see – annotate – verify” loop cannot hold.
    • Tool support: whether it provides image-manipulation tools such as cropping, zooming, and annotation drawing, turning visual primitives from “just talk” into executable actions.
    • Configuration loading: the host tool organizes model parameters and runtime configuration through structured configuration files (commonly JSON), including the model identifier, visual-marker switches, and coordinate conventions. A new model’s capabilities need matching runtime configuration to be carried.

    This also explains the relationship between two “route choices.” In interactive agent scenarios, tools can carry most of the “see – operate” responsibilities, and visual primitives exist more as internal conventions. A general web chat entry has no dedicated tools; to accomplish counting and navigation tasks, the “annotate – re-observe” capability must be built directly into the model itself.

    4.2 Old and New Configuration Compatibility

    As models iterate, older host tool configurations may not recognize the new model or new capabilities, so upgrading the tool and switching to the matching runtime configuration is often a prerequisite for using new capabilities. This is a common version-compatibility issue in engineering. It also illustrates, from another angle, that whether users can actually use a multimodal capability depends on three things being in sync: the model, the host tool, and the configuration.

    4.3 Documentation Completeness: An Underrated Engineering Metric

    In the same official documentation, the model-identifier settings for third-party integration are spelled out clearly, yet how to use the vendor’s own tool is left unexplained. For developers, this is a direct signal of documentation completeness.

    When evaluating multimodal solutions, it’s worth adding “is the usage path clear?” as a review item: do the release notes cover all entry points; how is runtime configuration declared; how do you troubleshoot when versions diverge. These details affect adoption efficiency as much as model accuracy does.

    5. Evaluation Checklist and Implementation Recommendations

    Consolidate the analysis above into a checklist to confirm item by item before integrating a new model.

    Evaluation Level Check Item Judgment Signal
    Static Whether the tokenizer registers visual primitive markers Markers parse as independent tokens as a whole
    Static Whether the chat template and coordinate conventions are clear Coordinate range, origin, and image-size conventions are explicit
    Structural Whether point-box relationships and coordinate rewriting are self-consistent Answers contain concrete numerical comparisons or rewrites
    Task Whether counting, localization, segmentation, and planning meet standards Case-set pass rate is stable and error is controllable
    Tool Whether the host tool supports image passing-in and history retention Screenshots and other tool outputs enter the next turn as images
    Consistency Whether performance across entry points is consistent The same case yields similar results across different entry points

    For real selection and adoption, keep the following recommendations in mind:

    1. Organize evaluation around tasks: first map out the scenarios in your business that rely most on spatial reference, such as document element localization, defect counting in industrial quality inspection, and chart data-point extraction, then design cases. General leaderboards can hardly replace business-specific cases.
    2. Define a degradation strategy: visual primitives improve reference precision but cannot compensate for insufficient perception resolution. For extremely small targets, traditional means such as high-resolution tiling and local zoom remain necessary—the two are complementary.
    3. Leverage general cloud capabilities: use container platforms to deploy inference services and evaluation jobs, use object storage to manage image and result data, and use log services and monitoring alerts to track production quality continuously.
    4. Leave room for model iteration: capability boundaries shift quickly with each version, so evaluation cases should be accumulated continuously and regressed regularly, rather than relying on a single conclusion for the long term.

    6. Conclusion

    Back to the initial question: markers like <|polygon|>, <|point|>, and <|/box|> in the tokenizer are strong evidence that the model takes the visual-primitives route, but not final proof that the capability is in place. Judging whether a multimodal model is truly “equipped” with visual primitives requires three layers of evidence: whether the markers and coordinate conventions are clear, whether structural parsing is self-consistent, and whether real tasks stably meet standards.

    This line of reasoning holds for every multimodal solution: demos and papers show potential, while testing and engineering determine value. From seeing clearly to pointing accurately, what visual primitives bridge is the reference gap. And from pointing accurately to using stably, you still need the joint support of host tools, configuration standards, and systematic evaluation.