Architecture and Practice of Native Multimodal Tool Calling

    This article explores the architecture and practice of native multimodal tool calling, addressing the limitations of text-only intermediation in AI models. It covers the costs of text conversion, core design principles like "image as parameter" and "result as context," and practical strategies for parameter organization, output processing, and engineering implementation to build a closed loop from perception to execution.

    In real-world applications, AI models often do more than generate text; they interpret images, read documents, retrieve information, and generate results to inform further decisions. Tool calling bridges “understanding” and “execution” by letting models decide which capabilities to invoke, what parameters to pass, and how to utilize the returned values. If this chain only supports pure text, visual inputs like images, video frames, scanned documents, and rendered results must first be transcribed into descriptions before proceeding. Transcription is inherently lossy—layouts, spatial relationships, colors, table lines, and fine annotations may disappear along the way. Engineers must also maintain complex logic for recognition, summarization, alignment, and retries.

    Native multimodal tool calling aims to eliminate this loss caused by “converting to text before understanding.” Its design principle is straightforward: visual content can serve directly as parameters, and tool results can remain part of the context. Instead of compressing a screenshot into an incomplete text description for downstream tools to guess from, the system retains the viewable visual object within the same reasoning chain, allowing newly generated charts, pages, or images to be read back when needed.

    Typical Costs of Text-Only Intermediation

    While text-based tool calling remains efficient for retrieval, calculation, and structured data queries, issues arise from modality mismatch. There are four typical costs:

    First, information loss. Packaging text, price tags, and promotional badges in a product image are difficult to describe while preserving both position and readability. For multi-column documents, full-text recognition often breaks headers, footnotes, cross-page tables, and stamps. Subsequent reasoning operates on a lossy summary rather than the original file.

    Second, error accumulation. Recognition errors enter parameters, leading to wrong tools or query conditions, and the results are then misinterpreted. Each intermediate conversion adds an unverified layer that cannot be directly checked against the original image.

    Third, engineering complexity. Systems must separately maintain visual preprocessing, text conversion, parameter templates, result parsing, and failure fallbacks. Inconsistent interfaces and scattered observability points make it difficult to pinpoint whether an online issue is due to “misreading” or “misinvoking.”

    Fourth, context disruption. If a tool returns a chart or screenshot with only a “chart generated” text note, the model cannot verify axes, outliers, or legends. The loop breaks after execution.

    Therefore, the need for native multimodal capabilities depends not on whether the model can “describe images,” but on whether the task requires visual evidence to persist through parameter construction, execution, and result verification.

    Design Principles: Image as Parameter, Result as Context

    Architecture can be constrained by two principles without binding to specific product interfaces.

    “Image as parameter” means tool inputs do not need to be fully serialized into natural language. Images, document pages, and local screenshots can be combined with text instructions and structured fields. Text expresses intent and constraints (e.g., “only look at the price area” or “align by table columns”), while visual objects provide irreplaceable evidence. The parameter schema should clearly distinguish text fields, visual references, and optional region hints, avoiding stuffing everything into a single string.

    “Result as context” means visual content in tool return values should retain a referable identity rather than just leaving log text. Statistical charts, rendered pages, and retrieved images must be observable in subsequent turns. The model can compare the original image with the result image to determine if the target was hit, if a retry with different conditions is needed, or if the visual conclusion should be written as the final answer.

    Together, these principles require that visual objects have stable references within the system, rather than repeatedly pasting untraceable binaries. The conversation state must preserve both text reasoning and “what is currently being viewed.”

    Organizing Parameters on the Input Side

    Multimodal input does not mean stuffing original images into every call unconditionally. A practical approach is layering:

    The Intent Layer uses short text for goals and constraints, such as comparison, extraction, verification, or recommendation.
    The Evidence Layer includes visual objects necessary for the current decision, which can be a full page or a cropped region.
    The Control Layer uses structured fields to describe the tool name, timeout, return format, and allowed operation scope.
    Separating these three makes auditing and replay much simpler.

    For document pages, prioritize referencing by page or logical block rather than concatenating into a super-long text. Tables, stamps, and handwritten annotations often only make sense on the original page. For UI screenshots, retain a resolution sufficient to discern text and control boundaries, and record the capture time to avoid treating an outdated UI as the current state. For multi-image tasks, assign stable IDs to each image and refer to them by ID in the text to reduce ambiguity like “the one on the left.”

    Parameter validation should occur before the call: check if visual references exist, if permissions are exceeded, if size and pages are within budget, and if text constraints match tool capabilities. Validation failures should return understandable errors rather than letting the model guess and auto-complete. A vendor-agnostic call structure might look like this:

    1. tool_call:
    2. name: extract_fields
    3. args:
    4. instruction: "Extract unit price, specifications, and whether there is a promotional mark"
    5. images:
    6. - ref: img_01
    7. hint: "Main image and price area"
    8. output: ["fields", "evidence_refs"]

    The key here is not the field names, but that visual references stand alongside text instructions, and the tool returns evidence references for the next round of verification.

    Re-entering Inference on the Output Side

    After a tool returns, the system should not treat the result as an endpoint. Return values that can be visually understood usually fall into three categories: data visualizations, UI/document renders, and externally retrieved images.

    For statistical charts, the model needs to read titles, units, legends, and obvious anomalies, rather than relying solely on accompanying numerical summaries. Summaries can accelerate processing, but when they conflict with the chart, verifiable visual evidence should trigger clarification or recalculation. For rendered screenshots, focus on whether the layout is complete, key copy is visible, and error prompts appear. For retrieved images, verify consistency with the query object rather than relying solely on title text.

    The return protocol should ideally include: machine-readable structured results, optional text summaries, and one or more visual object references. Structured results facilitate programmatic processing, while visual references enable the model to make secondary judgments. Returning only one means losing either automation or verification capability.

    The Closed Loop from Perception to Execution

    A complete chain can be divided into four steps: perception, planning, execution, and review, allowing a return to planning if the review fails.

    The Perception phase determines current visual objects and their task relevance, cropping or paginating as needed, without prematurely rewriting all content into a single truth. The Planning phase selects tools, constructs parameters, and declares expected evidence. The Execution phase handles retrieval, calculation, rendering, or business queries, with permissions and side effects controlled at this layer. The Review phase compares new results with the original goal: are fields complete, do images correspond, are values self-consistent. If it passes, generate the user-facing conclusion; if not, adjust parameters or switch tools, limiting retries.

    State management should retain at least: the goal, visual references used, parameter summaries for each call, return references, and failure reasons. This prevents multi-turn conversations from losing “what the previous image yielded” and facilitates manual review. For operations that alter external systems, separate visual understanding from actual writing: first present a structured action pending confirmation, then execute after explicit authorization.

    Scenarios and Boundaries

    Mixed text-image output suits explanatory tasks. The model can invoke drawing or rendering capabilities, read the generated image, check if annotations obscure content or if conclusions match the graphic, and then organize text. The value lies in reducing “text describing a non-existent or incorrect image.”

    Tasks like product identification and price comparison depend on packaging appearance, specification text, and price displays holding true simultaneously. Native calling keeps the main image, detail screenshots, and retrieved result images in the same context. The model can point out which image the evidence comes from, rather than outputting an untraceable recommendation. Such scenarios should also follow business rules: prices and inventory are subject to authoritative data sources, and images only assist in identification and explanation; promotional text on the screen cannot be directly treated as the transaction price.

    In assistant agent scenarios, the model decides the next query, form-filling suggestion, or to-do generation based on UI or documents. This requires permission boundaries, operation whitelists, and manual confirmation. Visual understanding improves the accuracy of locating UI elements and reading documents, but cannot replace authorization, nor should it allow the model to execute irreversible operations by default.

    Video and long documents significantly increase cost and latency. A safer approach is to locate key frames or pages first, then make fine-grained calls on candidate segments; full re-reading is usually uneconomical and prone to exceeding context budgets.

    Engineering Implementation Essentials

    Observability must answer three questions: what did the model see, what did it call, and based on which return did it make the decision. Logs should record reference IDs, tool names, parameter summaries, latency, and error codes, avoiding unnecessary long-term storage of raw sensitive images. For visual materials containing personal privacy or trade secrets, references should have access controls and expiration policies.

    Evaluation should not just look at whether the final text is fluent. It should examine item by item: whether parameters referenced correct images, whether tool selection was reasonable, whether result review found obvious contradictions, and whether it stopped instead of hallucinating upon failure. Regression testing can use fixed text-image samples covering blurry screenshots, multi-column layouts, low-contrast text, and empty results.

    Cost control relies on budgets: limit the number of images per call, maximum resolution, and retries. If a structured query can solve the problem, do not force an image-reading approach. For security, distinguish read-only tools from those with side effects; require explicit confirmation for the latter, and validate parameter sources to prevent text in untrusted images from being treated as system instructions (prompt injection).

    Prompts should be short and stable, placing formats and permissions in the system-side schema rather than relying on long prompts to compensate for protocol flaws. The model handles judgment and organization, tools handle deterministic computation and external access, and visual objects provide evidence. Mixing responsibilities often leads to irreproducible processes that “look like they can read images.”

    Conclusion

    The core of native multimodal tool calling is not simply adding an input format, but keeping visual evidence referable and verifiable across parameters, execution results, and subsequent reasoning. Text-only intermediation remains suitable for clearly structured tasks; when spatial relationships, layout, and appearance constitute decision criteria, premature irreversible transcription should be minimized. Treat images as parameters, put visual results back into context, and constrain the loop within an operable range using permissions, budgets, and reviews. By designing states and interfaces around perception, planning, execution, and review, teams can support implementation with standard object storage, logging, and monitoring capabilities, without relying on whether a single text description happens to be complete.