Multimodal LLM Engineering in Practice: Architecture, Inference, and Deployment
This article provides a practical engineering guide to multimodal large language models (LLMs) that process both visual and textual input. It covers the three-stage architecture (vision encoder, adapter, language decoder), the cost of visual tokens, reasoning chains for transparent intermediate steps, native function calling based on image content, interleaved text-and-image generation, long-context handling, evaluation design, deployment sizing and cost estimation, typical use cases, and risk and compliance considerations. The discussion is aimed at developers integrating multimodal models into production systems.
1. Background: From Image Recognition to Executable Visual Intelligence
Early vision models were mostly designed for a single task, such as image classification or object detection, producing labels or bounding boxes. As large language models matured, the industry began connecting visual information to language models, enabling models to understand images, charts, UI screenshots, and scanned documents, and to answer questions, reason, and act through natural language. These systems are commonly called vision-language models or multimodal large models.
For developers, the challenge is not simply “can the model see an image,” but rather three questions:
- Can the model reliably extract information from visual input, especially small text, dense tables, and complex charts?
- Can the model organize its reasoning into inspectable steps, rather than directly returning conclusions that cannot be traced?
- Can the model convert results into structured output or calls to external systems, integrating smoothly into existing business workflows?
The rest of this article addresses these three questions and provides a complete overview of architecture, capabilities, evaluation, and engineering deployment.
2. Base Architecture: A Three-Stage Design
Most multimodal models consist of three main modules.
2.1 Vision Encoder
The vision encoder converts pixels into feature representations. A common approach is to divide an image into fixed-size patches and encode the patches into a feature sequence using a structure similar to a vision transformer. The higher the input resolution, the more detail is preserved, but the sequence length increases, and the attention computation cost rises accordingly. Therefore, the choice of vision encoder often involves a trade-off between accuracy and latency.
2.2 Adapter (Projection Layer)
The dimension of visual features does not match the word embedding space of the language model. The adapter’s role is to map and align these features. In engineering practice, a multi-layer perceptron (MLP) is commonly used to project visual features into the language model’s embedding space, so that image regions appear as “visual tokens” within the sequence. The adapter is structurally simple, has low training cost, and is the preferred choice in most approaches.
2.3 Language Decoder
The language decoder receives the aligned visual tokens and text instruction tokens, and generates output tokens one by one in an autoregressive manner. It handles tasks such as question answering, reasoning, summarization, and structured output, determining the model’s language expression and logical organization capabilities.
The data flow can be summarized as follows:
Image -> patch splitting and normalization -> vision encoder -> visual feature sequenceVisual feature sequence -> MLP adapter -> aligned visual tokens[Visual tokens] + [Text instruction tokens] -> language decoder -> output text
2.4 The Cost of Visual Tokens
The number of visual tokens directly affects GPU memory usage and response latency. A high-resolution screenshot can generate thousands of visual tokens, far exceeding the length of an ordinary prompt. Common cost-reduction methods include:
- Dynamic resolution: Decide the splitting granularity based on image content and task requirements; large blank areas are split less.
- Feature pooling: Apply average or attention-based compression to neighboring image patches before alignment.
- Region cropping: First locate regions relevant to the question, then process those regions at high resolution.
Each method has its trade-offs. Over-compression loses details, so it is necessary to verify on a business evaluation set whether the accuracy drop is acceptable.
The following pseudocode illustrates the data flow and is not tied to any specific framework’s interface:
function multimodal_generate(image, prompt, model):# 1. Vision encodingpatches = split_image(image, patch_size=14)vis_feat = model.vision_encoder(patches)# 2. Modal alignmentvis_tokens = model.projector(vis_feat)# 3. Concatenate input sequencetxt_tokens = model.tokenize(prompt)sequence = concat(vis_tokens, txt_tokens)# 4. Autoregressive decodingreturn model.decoder.generate(sequence)
3. Reasoning Capability: Making Intermediate Steps Inspectable
On complex problems, direct answers are prone to errors. Reasoning mode allows the model to first output analysis steps and then provide a conclusion—this approach follows the chain-of-thought method. For visual tasks, reasoning typically includes: locating key regions, reading relevant numbers or text, establishing correspondences, performing calculations, and verifying the conclusion.
The cost of reasoning mode is a significant increase in output tokens, leading to higher latency and cost. In engineering, it is recommended to route tasks based on difficulty:
| Task Type | Recommended Mode | Reason |
|---|---|---|
| Image description, simple QA | Direct output | Few steps, latency-sensitive |
| Chart calculations, multi-step visual reasoning | Reasoning mode | Intermediate steps are inspectable, higher accuracy |
| Batch information extraction | Direct output with format constraints | Throughput priority |
Reasoning output should also be curated. For end users, only the conclusion and key evidence can be shown; for audit or debugging, the full process should be retained to locate errors.
4. Native Multimodal Function Calling
Function calling shifts the model from “answering questions” to “driving systems.” Native multimodal function calling means the model can directly decide which tool to invoke based on image content and generate a request that conforms to parameter specifications. For example, a user uploads a screenshot of an order and asks about shipping status; the model can identify the order number and generate a query request.
The typical flow has five steps:
- The user submits a screenshot and a natural language question.
- The model combines visual information with tool descriptions to decide whether an external capability is needed.
- The model outputs structured call parameters.
- The external system validates the parameters and executes the call, returning results.
- The model integrates the visual information and returned results to generate the final reply.
Call parameters are usually output as JSON, for example:
{"name": "query_order_status","arguments": {"order_id": "2026110912345","channel": "app"}}
Important engineering considerations:
- Tool descriptions must be clear, including purpose, parameter types, value ranges, and required fields, to reduce mis-calls.
- The external system must validate model-generated parameters; it should not trust model output directly.
- Call permissions should follow the principle of least privilege; write operations should require a second confirmation.
- Text in images may contain malicious instructions; the model should treat it as data to be processed, not as operational commands.
5. Interleaved Text and Image Generation
Interleaved text and image generation means that in a single output, the model inserts both text paragraphs and image-related content—for example, step-by-step instructions with illustrative screenshots, or charts referenced in a report. This capability is useful for tutorials, product documentation, and data reports.
In implementation, a common approach is for the model to output structured text with placeholder tokens, and a post-processing module handles image retrieval, generation, or insertion. For example:
## StepsStep 1: Open the settings page.[Image placeholder: screenshot of settings, id=step1]Step 2: Select the “Export data” option.[Image placeholder: screenshot of export option, id=step2]
The post-processing module looks up materials based on the placeholder tokens and generates the final document format. The advantage is that text and materials are decoupled, making it easy to replace materials and verify references.
6. Long-Context Handling
Multi-page documents, long reports, and extended conversations require the model to process a long context. Long-context windows introduce new engineering challenges:
- GPU memory pressure: Key-value caches grow linearly with context length, quickly filling GPU memory for long sequences.
- Attention dilution: When key information is located in the middle of a long text, the model may ignore or downweight it.
- Cost escalation: The longer the input, the higher the cost per call.
Mitigation strategies include: summarizing sections first and then aggregating, sending only pages relevant to the question into the model, and rolling compression of conversation history. For visual documents, a lightweight prescreening step can identify relevant pages, which are then passed to the large model for close reading.
7. Evaluation System Design
Evaluation is the only reliable way to determine whether a model fits a business need. Public evaluation sets provide a cross-reference, but they cannot replace validation on business-specific scenarios.
7.1 Capability Dimensions
It is recommended to split evaluation into the following dimensions, each with its own metrics:
- General visual question answering: Understanding common objects, scenes, and relationships.
- Chart and document understanding: Table reading, numeric comparison, and layout comprehension.
- OCR and detail recognition: Accuracy on small text, handwriting, and low-contrast text.
- Visual math and logical reasoning: Multi-step calculation and rule-based judgment.
- Hallucination detection: Whether the model fabricates content not present in the image.
- Format compliance: Whether the output meets JSON structure or field constraints.
7.2 Metrics and Samples
Common metrics include accuracy, hallucination rate, format compliance rate, time-to-first-token, and total latency. Samples must cover real distribution, including blurry images, low-quality scans, and screenshots with distracting information. Each dimension should accumulate enough samples, and human review results should be preserved to identify failure patterns.
7.3 Common Evaluation Biases
- Data contamination: Samples from public evaluation sets may have appeared in training data, inflating scores.
- Prompt sensitivity: Different phrasings of the same question can lead to different results; evaluation should fix prompt templates.
- Single-metric bias: Focusing only on accuracy can hide hallucination and formatting errors; multiple metrics should be combined.
8. Deployment Selection and Cost Estimation
Deployment selection needs to consider accuracy, latency, throughput, and cost simultaneously. Here is a rough GPU memory estimation method.
The memory occupied by model weights is approximately the number of parameters multiplied by the bytes per parameter. With 16-bit floating point, each parameter takes 2 bytes, so a 9-billion-parameter model requires about 18 GB of GPU memory; a 100-billion-parameter dense model requires about 200 GB. Actual usage also includes key-value caches, activations, and runtime overhead, so a buffer should be reserved on top of the estimate.
Quantization is a common way to reduce memory usage. 8-bit or 4-bit quantization can significantly reduce weight footprint but may cause accuracy loss, especially in mathematical reasoning and fine-grained recognition tasks. Before deployment, results with and without quantization should be compared on business evaluation sets.
Selection guidelines:
- For edge-side or high-concurrency simple recognition tasks, a small parameter model is suitable.
- For complex reasoning, multi-image analysis, and long-document tasks, a large parameter model is suitable.
- For latency-sensitive scenarios, evaluate both time-to-first-token and the extra output overhead of reasoning mode.
- For cost-sensitive scenarios, use tiered routing: simple requests go to a small model, and complex requests escalate to a large model.
9. Typical Application Scenarios
- Customer support: A user uploads a fault screenshot; the system identifies the error message and provides troubleshooting steps.
- Financial analysis: Extract numbers from report images, verify cross-relationships, and flag anomalies.
- Education and tutoring: Parse problem images and provide step-by-step solution explanations.
- Industrial inspection: Read instrument readings and equipment states, and generate inspection records.
- E-commerce operations: Extract attribute information from product images and generate structured product data.
These scenarios share common traits: input includes visual information, output must be structured or executable, and the business has a clear tolerance for errors.
10. Risk and Compliance Considerations
- Hallucination risk: The model may fabricate numbers or text not present in the image. For high-risk content such as financial amounts, medical information, or legal content, a human review step should be retained, and the model should be asked to indicate the evidence region.
- Privacy risk: Screenshots often contain names, phone numbers, ID numbers, and other sensitive information. Data should be de-identified before upload, and raw images should not be retained in logs.
- Prompt injection risk: Text in images may contain instructions attempting to alter the model’s behavior. The system should treat image content as data and restrict executable operations through permission controls.
- Small text and low-quality input: Blurry, skewed, or heavily compressed images reduce recognition accuracy; quality checks should be performed on the front end, and users should be prompted to re-upload.
- Interpreting evaluation results: Public leaderboard rankings should not be directly equated with business performance; always validate on the actual task distribution and business metrics.
