Multimodal Native Tool Calling: Closing the Loop from Visual Perception to Action
Multimodal agents are moving from describing images to acting on them. This article examines multimodal native tool calling, where visual content is modeled within the same context as tool calls, so images can serve directly as tool arguments and returned images become evidence for further reasoning. It explains where the visual-to-action loop breaks under traditional function calling, the four principles behind the new paradigm, and the engineering practices needed to make it reliable, including context budgeting, visual grounding, tool-contract design, and security and cost control. It also covers cloud-versus-device model selection, evaluation and observability, and real-world scenarios such as mixed content generation, product recognition and price comparison, complex document extraction, and assistant agents.
Multimodal agents are moving from “looking at images and talking about them” to “looking at images and getting things done.” As tasks expand from question answering, summarization, and description to screen operation, product price comparison, and structured document extraction, models need more than text generation—they must call external tools to perform real actions. Whether this chain stays intact determines whether multimodal agents can actually ship. This article focuses on multimodal native tool calling: what it is, the key implementation points, and the engineering practices behind it.
1. Background: where the loop breaks
Traditional function calling was born in the text era: the model outputs structured arguments, the tool returns a text result, and the model continues reasoning from there. This paradigm is efficient and mature for pure text tasks. But once the input becomes images, screenshots, scanned documents, or video frames, three structural gaps appear.
1. Visual distortion on the input side
To make a tool “understand” an image, you usually first convert it into a text description—for example, “there’s a blue button in the top-right corner of the screenshot, labeled with the price 299”—and then pass that description as an argument. The description is a second-order abstraction of the visual content. Information that is critical for downstream decisions—position, color, font size, control hierarchy, chart trends—is often lost in transcription. Worse, the model’s visual reasoning and its tool call are forcibly split into two stages, with no inspectable “intermediate state” in between.
2. Visual blind spots on the output side
Tool return values are mostly structured data: price lists, search results, rendered page screenshots, statistical charts. When a return value is an image, the model can only see a text summary and cannot perform a second visual check on the result. For instance, after calling a search tool and getting a list of candidate products, the model needs to confirm “whether the main image matches the title” before writing an accurate product description. Without visual understanding, the model can only rely on metadata for text matching.
3. Pipeline bloat on the engineering side
Each modality conversion maps to extra services and protocols: image upload, OCR, coordinate conversion, image reinjection, and cross-system authentication. The longer the chain, the higher the end-to-end latency, the more failure points, and the harder debugging and replay become. As a result, multimodal inputs are often “dimensionally reduced” into text and handed to a pure text agent.
Visual capability shows up only in the first link; everything after—reasoning, execution, and verification—falls back to text logic. That’s where the loop breaks. Fixing it requires making visual content a “first-class citizen” of tool calling: usable directly as a tool argument, and usable again by the model as a return value.
2. The paradigm shift: multimodal native tool calling
Multimodal native tool calling means modeling visual content and tool calls within the same unified context: an image can serve directly as a tool input argument, and an image returned by a tool can serve as evidence for subsequent reasoning. Recently released next-generation multimodal models from several teams in the industry generally push their training context window to the hundreds-of-thousands-of-tokens range (128k is a common tier) and perform strongly on general visual understanding benchmarks at comparable parameter scales. This provides the compute and capacity foundation for such loops. The design can be summarized in four principles.
1. Image as argument: no more “translation” on the input side
Tool arguments can contain images or image references directly (file addresses, page screenshots), and the model can output pixel coordinates, bounding boxes, or element IDs that point to the locations it “sees.” The key is no longer just whether the text JSON is valid, but whether the visual references the model provides land in the right place—that is, whether visual grounding is accurate.
2. Result as context: the output side can be “seen” again
Images returned by a tool—retrieved product images, rendered page screenshots, generated statistical charts—are written back into the context in full. The model performs another visual understanding pass over the returned result and decides the next step accordingly: retry with different arguments, make an additional call, or output a conclusion directly. Visual evidence and the reasoning process therefore sit on the same timeline, making decisions traceable and replayable.
3. Long context: carrying real multi-turn, multi-image conversations
Multi-page documents, video frame extraction, and multi-round “call—observe—call again” loops all accumulate large numbers of images in the context. A hundreds-of-thousands-of-tokens window lets the model reference multiple images, multiple documents, and multiple rounds of tool results within a single session, reducing information loss from cross-turn summarization. This also introduces new budget-management problems, covered in Section 3.
4. Large-and-small model collaboration: one model family spanning cloud and device
A common industry practice is for the same generation of models to ship in two forms:
- A large-parameter version for cloud clusters (often using a sparsely activated MoE architecture to carry strong capability at a lower per-token compute cost);
- A lightweight version for local deployment and low-latency scenarios.
Through distillation, data-mix alignment, and prompt compatibility, the two can share the same tool-calling protocol and system prompt. Upper-layer applications route by task difficulty to balance cost and quality.
Example: a tool call that includes an image
The code below is illustrative; it only expresses the message structure and is not tied to any specific vendor API. call_model and parse_calls must be implemented according to the documentation of the model service platform you use; platforms differ in how well they support multimodal content in tool results, and in some cases you still need to convert visual results into structured fields at the tool layer.
# Example user message{"role": "user","content": [{"type": "text", "text": "Analyze this product screenshot: find the price text and compare prices online"},{"type": "image_url", "image_url": {"url": "https://example.com/item.png"}},],}# Example tool call returned by the assistant{"role": "assistant","tool_calls": [{"id": "call_01","function": {"name": "web_search","arguments": '{"query": "Model X flagship version final price"}',},}],}# Example tool result: text summary + visual result (price trend screenshot){"role": "tool","tool_call_id": "call_01","content": [{"type": "text", "text": "Found prices from 3 platforms; the lowest final price is 399 yuan"},{"type": "image_url", "image_url": {"url": "https://example.com/price_trend.png"}},],}
import jsondef run_vision_agent(task_text, image_url, call_model, parse_calls, tools, max_steps=8):messages = [{"role": "system","content": "You are an assistant with visual understanding and tool-calling abilities. ""When parameters are uncertain, call a tool to verify first. Treat images ""returned by tools as evidence, and cite your sources in the conclusion.",}, {"role": "user","content": [{"type": "text", "text": task_text},{"type": "image_url", "image_url": {"url": image_url}},],}]for _ in range(max_steps):# Multimodal responses use the multimodal_content field to carry both text and imagesresp = call_model(messages=messages, tools=tools)assistant = {"role": "assistant","content": resp.get("content", ""),"multimodal_content": resp.get("multimodal_content", []),}if resp.get("tool_calls"):assistant["tool_calls"] = resp["tool_calls"]messages.append(assistant)calls = parse_calls(resp)if not calls:return messages[-1] # The model has given its final replyfor call in calls:try:args = json.loads(call["function"]["arguments"] or "{}")result = TOOL_REGISTRY[call["function"]["name"]](**args)except Exception as exc: # Argument error or tool exception: return a structured response so the model can self-correctresult = {"error": str(exc)}messages.append({"role": "tool","tool_call_id": call["id"],"content": result.get("summary", ""),# The image returned by the tool enters the context as new evidence"multimodal_content": result.get("multimodal_content", []),})raise RuntimeError("Reached the maximum number of tool-call steps without completing the task")
This illustrative logic has three key actions:
- Don’t terminate the loop on argument parsing failure: return the error in a structured way so the model can correct its arguments and retry.
- Images returned by tools enter the context: the model can see price trends, product images, or page screenshots and decide the next step accordingly.
- A step-count limit as a fallback: prevents the model from looping endlessly between tools.
3. Implementation points: making the loop robust
1. Context budget: how many tokens does an image actually consume
Most visual encoders split an image into patches (or tiles) and then map them to visual tokens, and the main cost in long-context or multi-image scenarios comes from here. For engineering, capacity planning should use a coarse “approximate by area” approach:
def estimate_image_tokens(width, height, max_side=1024):"""Estimate visual tokens from area after scaling to the max-side limit; for capacity planning only"""scale = min(1.0, max_side / max(width, height))tokens = (width * scale) * (height * scale) / 750 # ~750 pixels ≈ 1 tokenreturn max(1, int(tokens))def plan_budget(history, max_new_tokens, context_window, reserve_ratio=0.2):"""Compute the budget for retained history; the remaining allowance is reserved for output and tool calls"""budget = int(context_window * (1 - reserve_ratio)) - max_new_tokensused, kept = 0, []for msg in reversed(history): # Accumulate from the most recent message backward; truncate once the budget is exceededcost = estimate_message_tokens(msg)if used + cost > budget:breakused += costkept.append(msg)return list(reversed(kept)), used
estimate_message_tokens can simply count text with the tokenizer and estimate images with the formula above. The conversion coefficient needs to be adjusted to the actual model; the core is to make capacity planning quantifiable and reproducible.
Truncation needs a strategy: keep the system prompt and the original task, and prefer to drop older tool-returned images first, or replace them with summaries. This preserves the task goal while reserving budget for the visual evidence most relevant to the current decision.
2. Visual grounding: making the model “point accurately”
Screen operation, document field location, and chart-region cropping all depend on visual grounding. The output format should be agreed in advance:
- Coordinate system: standardize resolution and origin position;
- Integer pixels: avoid floating-point error;
- Element references: for UI tasks, prefer having the model reference numbered elements in a list (accessibility tree or DOM structure) rather than raw coordinates.
TOOLS = [{"type": "function","function": {"name": "click_element","description": "Click the specified element in the current screenshot","parameters": {"type": "object","properties": {"element_id": {"type": "string","description": "ID of the target element in the element list, e.g. btn_12",}},"required": ["element_id"],},},}]def click_element(element_id):# Map the element ID back to coordinates and perform the click; illustrative return structurebox = ELEMENT_REGISTRY[element_id]perform_click(*box["center"])return {"summary": f"Clicked element {element_id}", "multimodal_content": [{"type": "image_url", "image_url": {"url": capture_screen()}}]}
Returning a fresh screenshot after execution creates an “observe—act—observe again” loop, so the model can immediately verify whether the action took effect.
3. Tool contracts: multimodal arguments need constraints too
Visual tasks carry higher uncertainty, so tool design needs to be “narrow and clear”:
- Semantic naming and descriptions: tool names and descriptions directly determine whether the model picks the right tool;
- Strongly typed schema: spell out dimensions, enumerations, and coordinate formats, and set valid ranges;
- Idempotency first: prefer queries over writes, and avoid duplicate execution;
- Structured errors: error messages should explain “what’s wrong and how to fix it” so the model can self-correct.
{"name": "crop_region","description": "Crop a rectangular region from the given image and return the cropped image","parameters": {"type": "object","properties": {"image": {"type": "string", "description": "Image URL, e.g. https:// or data:image/..."},"box": {"type": "array","items": {"type": "integer", "minimum": 0},"minItems": 4,"maxItems": 4,"description": "Pixel coordinates [x1, y1, x2, y2] with origin at top-left; require x2>x1 and y2>y1"}},"required": ["image", "box"]}}
4. Security and cost control
Visual agents often touch real systems, so constraints must come first:
- Least privilege: grant tools only the permissions needed to complete the task, and use domain allowlists for network access;
- Action tiering: query actions can run automatically, while high-risk actions like writes, payments, and deletions require human confirmation or re-authentication;
- Prompt-injection defense: text in web pages, documents, and screenshots may carry malicious instructions, so wrap them uniformly in delimiters and label them “untrusted data”—treat them as material, not as instructions;
- Cost control: compress or chunk images at the entry point; cache repeated visual analyses (such as field extraction from the same image); and use logging, monitoring, and alerting services on cloud platforms such as Baidu AI Cloud to continuously track token usage and tool error rates, alerting promptly on abnormal consumption.
4. Deployment scenarios
1. Mixed image-text content generation
Inputs like papers, research reports, and slide decks are often layout files mixing images, text, tables, and formulas. Native tool calling brings two changes:
- Complex layout understanding: the model reads the layout and semantic relationships of an entire page in one pass and accurately extracts structured information such as sections, tables, captions, and formulas;
- On-demand figure generation: while generating each section, the model automatically calls a retrieval tool to find candidate images, or crops key charts from the source, and then performs another visual check on the candidates.
For implementation, a “two-stage” approach is recommended:
- First do layout parsing and structured extraction;
- Then do segmented writing and figure assembly.
Record the source and rationale for each figure decision to ease human review and fact-checking.
2. Product recognition and good-price recommendations
Typical flow: the user uploads a product screenshot → the model recognizes the model and key attributes → calls a search-and-compare-price tool → the tool returns prices and images → the model compares main images to confirm it’s the same product → combines this with the price trend to give a purchase recommendation. Throughout, the model can “see” the visual evidence, avoiding a wrong same-product judgment based on the title alone.
3. Complex document information extraction
Tables in contract terms, financial reports, and bidding documents often span pages, use merged cells, and nest multiple header layers. The model can crop and zoom into local regions on demand, then return to the global view to cross-check—equivalent to how a person reads: “first the big picture, then the details.”
4. Assistant agents
In internal operations back offices or office software, the model understands the interface from screen screenshots and then calls tools like click, type, and scroll to perform actions, and can judge from post-action screenshots whether an action succeeded or an error dialog popped up. Such scenarios should default to “human-in-the-loop”: show an action plan before key actions, and keep an auditable trace of the execution.
5. Model selection: how cloud and device cooperate
| Dimension | Cloud-side large-parameter version | Device-side lightweight version |
|---|---|---|
| Typical tasks | Multi-page document parsing, complex layout understanding, multi-round tool orchestration, video-level long context | Screen understanding, form filling, local file Q&A, device control |
| Latency and cost | Higher per-request latency, suited to async or near-real-time tasks | Low first-token latency, data stays local |
| Data compliance | Requires evaluating data-egress requirements | Suited to privacy-sensitive scenarios |
| Operations | Call the model service via API, scale by call volume | Local quantized deployment; watch memory and inference-framework compatibility |
In practice, a “routing + cascade” strategy works well:
- Use a lightweight model or classifier to judge task difficulty;
- Let simple tasks be completed directly by the device-side model;
- Escalate complex tasks to the cloud-side model, and if necessary hand the result back to the lightweight model for formatted output.
This both lowers average cost and preserves quality on hard tasks.
6. Evaluation and observability
The multimodal tool-calling chain is long with many stages, and looking at model leaderboards alone cannot reflect real usability. The evaluation system should cover at least four dimensions:
| Dimension | Key metrics | Measurement method |
|---|---|---|
| Visual understanding | Field-extraction accuracy, OCR accuracy, visual-grounding hit rate (IoU between predicted and ground-truth boxes) | Annotated datasets |
| Tool calling | Tool-selection accuracy, argument validity rate, average number of calls | End-to-end replay |
| Task completion | First-pass rate, human-correction rate, P95 latency, per-task cost | Offline evaluation + online canary |
| Security and robustness | Prompt-injection interception rate, number of unauthorized actions | Adversarial test sets |
For online observability, it is recommended to log a summary of the input and output of each reasoning round, tool names and arguments (with sensitive fields masked), tool latency and error codes, and image token consumption. When problems occur, you can quickly locate them via the call chain and determine whether it is a visual understanding error, a tool-selection error, or an argument-construction error. After locating the cause, feed the failed samples back into training or prompt iteration to form a closed-loop optimization.
7. Conclusion
The competitive focus of multimodal agents is shifting from “can the model understand images” to “can the model get things done by looking at images.” Multimodal native tool calling—with images as arguments and results as context—puts perception, reasoning, and execution back into the same context. But model capability is only the starting point; what truly determines deployment outcomes are engineering details like context-budget management, visual-grounding accuracy, tool-contract design, and security boundaries. By turning the input and output of each stage into measurable, replayable, and optimizable objects, visual intelligence can reliably translate into business value.
