Multimodal LLMs: Visual Understanding and Tool Calling
Recent multimodal models emphasize real-task reliability and deployability, not just parameter count. This article covers visual-precision techniques, evaluation caveats, long-context trade-offs, and how function calling turns image understanding into executable actions.
Background: Capability and Deployment Together
Several Chinese labs recently released multimodal models at two scales: a cloud-oriented high-performance version and a lightweight version for local or edge devices. Industry attention has shifted. Comparisons once focused on parameter size; they now focus on whether a model can complete real work reliably and whether it can be deployed under an organization’s data and budget constraints.
Earlier multimodal models had three common weaknesses: unstable fine-grained visual recognition, inefficient image–text alignment, and limited ability to call external systems on their own. Newer models keep baseline skills such as image classification, object detection, and image–text retrieval, while putting extra weight on visual precision and tool calling.
The cloud version uses a mixture-of-experts (MoE) design with about 106 billion total parameters and about 12 billion activated per inference, which helps control compute cost while preserving inference quality. The lightweight version has about 9 billion parameters and is intended for resource-limited edge devices. These figures come from the original materials; confirm them against the publisher’s official documentation before selecting a model.
Visual Understanding: Gains and Limits
Three ways to raise visual precision
Gains usually come from several modules working together, not from more parameters alone.
First, fine-grained visual encoding. The model splits images into smaller patches and uses learnable spatial position encodings. That makes it easier to keep edges, textures, and small text, which helps document OCR and industrial inspection.
Second, cross-modal alignment. Instead of simply concatenating image and text features, multi-layer cross-attention lets each text token attend to relevant image regions. Contrastive losses train the model to separate matching and non-matching image–text pairs. This structure helps with questions that need spatial grounding, such as “what color shoes is the third person wearing?”
Third, multi-scale feature fusion. Features are extracted at several resolutions, then low-level detail is combined with high-level semantics. That reduces information loss for small objects and distant scenes.
How to measure precision gains
Source materials claim a clear improvement in localization accuracy versus a prior generation, but they do not name the test set, baseline, or statistical definition. Evaluate on your own business samples. Track precision, miss rate, and false-positive rate, and keep failure cases. The size of any gain depends on data and task; do not reuse marketing numbers as-is.
Applicability limits
Visual models still hallucinate—describing objects or text that are not in the image. In high-risk settings such as medical imaging, legal documents, and industrial inspection, model output is assistive only; critical conclusions need human review. Errors are more common on tiny text, low-contrast images, and blurry scans; test those cases separately.
Long Context: Value and Cost
Training context is described as up to 128k tokens, on the order of 100,000 English words. Long PDFs, large drawing sets, and frame samples from longer videos can fit in one request. Treat the window size as subject to official docs; usable length also depends on how many tokens images consume.
The main cost is that attention compute grows quickly with sequence length. Grouped-query attention (GQA) and rotary position embedding (RoPE) extrapolation are used to ease memory pressure. Developers still need to watch:
- Longer inputs usually mean longer latency.
- Recall of key facts in long inputs is not always stable; restate important points in the instruction and ask the model to cite source locations.
- More images consume context quickly.
Tool Calling: From Seeing Images to Doing Work
How function calling works
Function calling means the model, given a user instruction and tool descriptions, decides whether a tool is needed and emits well-formed arguments. Most mainstream models already support this. Newer models mainly improve multi-tool selection, argument generation, and handling of vague instructions. Training data includes tool-use examples so the model learns both when to call a tool and when to ask the user first.
Output is typically structured JSON. The application must parse it, validate parameters, then issue the real API request.
