# Vision-Language Models (VLMs): Architecture, Capabilities, Evaluation, and Deployment

    A practical guide to vision-language models: how the vision encoder – projector – language decoder architecture works, how dense and sparse (MoE) model tiers trade off quality against cost, and what capabilities matter in production, including native multimodal function calling, interleaved image-and-text generation, reasoning modes, and long-context video understanding. It covers evaluation dimensions from perception to safety, a selection framework for choosing between lightweight and flagship models, a reference inference service architecture with continuous batching and streaming, and example Python code for calling a VLM API and running a multi-step tool-use loop. Includes architecture details, forward-pass pseudocode, benchmark discussion, and engineering checklists for anyone building multimodal applications.

    1. Introduction: From Text-Only Intelligence to Omni-Modal Intelligence

    Over the past two years, the boundaries of large language models have expanded rapidly from pure text toward images, video, and audio. Vision-language models (VLMs) are the primary vehicle for this shift: they can identify objects, text, charts, and scene relationships inside an image, and combine that understanding with natural language to answer questions, reason, generate code, and call tools.

    Take enterprise document review as an example. The model has to read the layout, tables, and body text of a scanned document at the same time, then produce a structured conclusion according to business rules. Requirements like these go far beyond the scope of traditional OCR, which only recognizes characters.

    Recently, leading model teams have shipped a new generation of vision understanding models: some target low-cost on-device deployment, others use a sparse architecture with roughly ten-billion total parameters and roughly ten-billion activated parameters, and nearly all of them add function calling, interleaved image-and-text generation, and long-context support.

    Setting aside specific product claims, this article examines the technical lineage and the key implementation points of vision-language models across four layers: architecture, capability evolution, evaluation, and engineering practice.

    2. Core Architecture: Vision Encoder – Projector – Language Decoder

    Most mainstream VLMs use a three-stage architecture. The design goal is to keep the reasoning ability of a language model while adding visual information with as little structural change as possible.

    2.1 Vision encoder: from pixels to visual features

    The vision encoder is typically an image encoder pretrained on large-scale image-text alignment data, such as a CLIP-style contrastive model or a SigLIP-style model. It splits the input image into patches and encodes them into a fixed-dimension sequence of visual features.

    The reason for choosing this kind of encoder is that its training objective emphasizes image-text semantic alignment. The features it extracts therefore already live in a space close to language, which significantly lowers the difficulty of aligning them later.

    For visual resolution, the industry generally adopts a dynamic tiling strategy: a preprocessing model first decides whether the original image needs to be preserved at full detail, then a high-resolution image is split into several sub-images and encoded separately, while a downscaled thumbnail of the full image is kept to preserve global information. This retains the ability to recognize fine detail while avoiding the information loss caused by a fixed low resolution.

    2.2 Projector: the adapter that aligns modalities

    The feature dimension produced by the vision encoder usually does not match the embedding dimension of the language model, so an adapter layer (commonly called a projector or connector) does two jobs:

    • Dimension mapping: a multi-layer perceptron (MLP) or cross-attention structure projects visual features into the language model’s embedding space.
    • Sequence compression: visual tokens are pooled or downsampled to control how many tokens are injected into the language model. Some designs use convolutional downsampling or a Perceiver Resampler–style structure to compress the sequence length while keeping the important information.

    The projector accounts for only a very small fraction of the total parameter count, yet it has a large impact on multimodal alignment quality. In practice, a two-stage training strategy is common: freeze both ends and train only the projector, then unfreeze part of the model for joint fine-tuning.

    2.3 Language decoder: unified autoregressive generation

    The language decoder performs the final understanding and generation. The input sequence is built by concatenating text tokens and visual tokens according to a specific template: the text is converted to vectors through word embeddings, the visual part comes from the projector output, and the two are concatenated or inserted at placeholder positions before being fed to the decoder. The decoder then generates the answer autoregressively, token by token, so multimodal understanding is formally a sequence modeling problem — describing an image in language.

    An illustrative forward pass:

    1. import torch
    2. import torch.nn as nn
    3. class VisionLanguageModel(nn.Module):
    4. """Minimal skeleton: vision encoder + projector + language decoder"""
    5. def __init__(self, vision_encoder, projector, decoder):
    6. super().__init__()
    7. self.vision_encoder = vision_encoder
    8. self.projector = projector # dimension mapping and visual token compression
    9. self.decoder = decoder
    10. def encode_image(self, images):
    11. # images: [B, 3, H, W], after preprocessing (resize, tiling, normalization)
    12. feats = self.vision_encoder(images) # [B, N, Dv]
    13. feats = self.projector(feats) # [B, M, Dt], Dt = language model dimension
    14. return feats
    15. def forward(self, input_ids, attention_mask, images=None):
    16. embed = self.decoder.get_input_embeddings()
    17. text_embeds = embed(input_ids) # [B, L, Dt]
    18. if images is not None:
    19. vis_embeds = self.encode_image(images) # [B, M, Dt]
    20. # merge/insert visual tokens at the image placeholder positions defined by the chat template
    21. inputs_embeds = self.merge(text_embeds, vis_embeds, attention_mask)
    22. else:
    23. inputs_embeds = text_embeds
    24. # visual positions in the labels must be set to -100 so the language loss is not computed on visual tokens
    25. return self.decoder(inputs_embeds=inputs_embeds, attention_mask=attention_mask)

    It is worth emphasizing that visual tokens also participate in the key/value computation of self-attention. The sharper and higher-resolution the image, the more visual tokens it produces, and the more GPU memory and compute the model consumes during inference.

    3. Parameter Scale and Sparse Activation: Balancing Quality Against Cost

    The evolution of recent models shows a clear two-tier route:

    Form Total parameters Activated parameters Typical positioning
    Dense small model ~9B ~9B On-device and private deployment; high-concurrency, low-cost scenarios
    Sparse large model ~100B ~10B Complex visual reasoning and multi-step tasks

    Taking a recent model family as an example, its flagship has roughly 106B total parameters and 120B… activated around 12B, while the lightweight version has about 9B total parameters; both are built on a base language model in the ten-billion range.

    The engineering value of this sparse activation design comes from three things:

    1. Inference cost close to a small model. A mixture-of-experts (MoE) structure activates only part of the expert networks per forward pass. So even though the total parameter count is large, the actual compute (FLOPs) and memory-bandwidth pressure are far lower than for a dense model with the same total size.
    2. Knowledge capacity decoupled from inference efficiency. More total parameters means stronger knowledge storage and generalization, while fewer activated parameters keep response times short.
    3. More flexible tiered deployment. The lightweight version suits cost-sensitive, latency-critical scenarios; the sparse flagship suits tasks that need deep visual reasoning. Enterprises can route by task complexity and balance cost against quality.

    4. Capability Evolution: From Seeing to Doing

    4.1 Native multimodal function calling

    Early multimodal models mainly solved image-and-text question answering. The new generation supports native multimodal function calling, meaning tools can be invoked directly on top of visual understanding. The typical flow is:

    1. The user submits an image plus a natural language instruction.
    2. The model infers the task intent from the image content and selects a suitable tool.
    3. The model outputs the call arguments as structured JSON.
    4. The execution environment parses the arguments, calls the tool, and returns the result to the model.
    5. The model combines the visual information with the tool result and produces the final answer.

    Compared with the traditional “OCR first, then call a text model” approach, the native design avoids loss when information is converted between modalities. In scenarios such as extracting fields from invoices, querying chart data, and comparing prices after product recognition, the chain is shorter and more accurate.

    4.2 Interleaved image-and-text generation

    Another notable capability is interleaved image-and-text generation: within one coherent response, the model can insert images, charts, or code blocks on demand to produce structured, mixed-modality output. This usually relies on a special output markup protocol:

    • While generating text, the model switches to an “image generation” or “chart rendering” sub-flow when it encounters a specific marker.
    • The generated result is filled back into the text stream as a quote block or placeholder.
    • A rendering layer finally assembles the text and visual fragments into a complete page.

    This is directly useful for generating course materials, drafting data reports automatically, and producing product detail pages.

    4.3 Chain-of-thought enhancement and reasoning modes

    Some models (for example, “Thinking” variants) are trained with long chain-of-thought (long CoT) data, so that on complex visual reasoning tasks such as geometry proofs or scientific chart analysis, they output a step-by-step reasoning process before the conclusion. Practical experience:

    • Simple tasks such as text recognition or object counting do not need deep reasoning; enabling it only adds latency.
    • Complex tasks such as multi-image comparison or mathematical derivation see a clear accuracy gain with reasoning enabled.
    • Reasoning depth can be controlled through the prompt or API parameters, giving you thinking on demand.

    4.4 Long context and video understanding

    The new generation generally extends the context window to the 128K level, which makes long documents, long videos, and multi-turn multi-image conversations possible. The engineering side usually involves:

    • Position encoding extrapolation: extending the effective range of rotary position embeddings via interpolation or methods such as YaRN.
    • Visual token compression: uniform frame sampling for video and dynamic resolution adaptation for long images, to keep the total number of visual tokens under control.
    • Attention optimization: mechanisms such as sliding windows and grouped-query attention to reduce the computational complexity of long sequences.

    5. Evaluation: How to Measure Multimodal Model Capability

    Evaluating a vision-language model is far more complex than evaluating a text-only model. The industry usually builds an evaluation matrix from the following dimensions.

    5.1 Basic perception

    General visual question answering, object recognition, scene description, and text recognition (OCR). Common metrics include accuracy, CIDEr, and BLEU.

    5.2 Logical reasoning

    Math and chart reasoning, visual commonsense reasoning, and multi-step visual QA. These tasks require the model not only to see but also to reason.

    5.3 Document and structured understanding

    Document QA, table understanding, field extraction from invoices, and layout analysis — all of which test how well the model parses complex page layouts.

    5.4 Generation and interaction

    Interleaved image-and-text generation, function-calling accuracy, and consistency across multi-turn dialogue. These are the key indicators of whether a model is actually usable.

    5.5 Robustness and safety

    Adversarial testing, hallucination-rate measurement, and harmful content filtering.

    According to public information, the recent GLM-4.5V series reached state-of-the-art results among peer open-source models on 41 public vision-multimodal benchmarks, and the GLM-4.6V series showed clear improvements over the previous generation on more than 30 mainstream benchmarks including MMBench, MathVista, and OCRBench. It is worth noting that leaderboard scores reflect performance on specific test sets; real business results still need to be validated against your own data.

    6. Engineering Practice: From Model Selection to Service Deployment

    6.1 Selection framework

    Enterprises should weigh the following when selecting a model:

    1. Task complexity: simple recognition tasks can use a lightweight model; complex reasoning tasks are better served by the sparse flagship.
    2. Deployment constraints: on-device or private environments require attention to model size, quantization support, and hardware compatibility.
    3. Cost budget: evaluate GPU rental or purchase cost, inference latency, and concurrent throughput together.
    4. Ecosystem support: prefer models that support standard APIs (such as an OpenAI-compatible interface) and have an active community with solid documentation.

    6.2 Inference service architecture

    A typical VLM inference service consists of the following components:

    1. Client
    2. → API gateway (authentication, rate limiting, routing)
    3. → Request preprocessing (image validation, resizing, Base64/URL parsing)
    4. → Inference engine (batching, KV cache management, streaming output)
    5. → Response post-processing (JSON parsing, sensitive content filtering)
    6. → Logging and monitoring (request volume, latency, token consumption, error rate)

    Key engineering points:

    • Push image preprocessing down: validate image format and enforce size limits at the gateway layer, so malformed input never reaches the inference engine.
    • Continuous batching: the inference engine merges the dynamic generation processes of different requests into static batches, which noticeably improves GPU utilization.
    • Streaming output: for long answers, push incremental tokens over SSE or WebSocket to improve the user experience.
    • Caching: cache results for high-frequency repeated requests (for example, OCR of the same image) to reduce inference load.
    • Observability: collect metrics such as time to first token (TTFT), generation speed (TPS), and GPU utilization to inform capacity planning.

    6.3 Example code

    The code below shows a complete “image understanding + tool calling” flow through a standard API.

    1. import json
    2. import requests
    3. def call_vlm(image_url, prompt, tools=None):
    4. """Call a vision-language model with optional function calling"""
    5. payload = {
    6. "model": "vl-model", # replace with the actual deployed model name
    7. "messages": [
    8. {
    9. "role": "user",
    10. "content": [
    11. {"type": "image_url", "image_url": {"url": image_url}},
    12. {"type": "text", "text": prompt},
    13. ],
    14. }
    15. ],
    16. "max_tokens": 2048,
    17. "temperature": 0.7,
    18. }
    19. if tools:
    20. payload["tools"] = tools
    21. resp = requests.post(
    22. "https://api.example.com/v1/chat/completions", # replace with your inference service endpoint
    23. json=payload,
    24. headers={"Authorization": "Bearer YOUR_API_KEY"},
    25. timeout=60,
    26. )
    27. resp.raise_for_status()
    28. return resp.json()
    29. # Example: extract product information from an image
    30. result = call_vlm(
    31. image_url="https://example.com/receipt.jpg",
    32. prompt="Extract the product names, prices, and the total from this image, and output them as JSON.",
    33. )
    34. print(json.dumps(result["choices"][0]["message"]["content"], ensure_ascii=False, indent=2))

    For scenarios that require multi-step tool calls, you can implement a simple agent loop on the client side:

    1. def agent_loop(image_url, user_query, max_iterations=5):
    2. messages = [{"role": "user", "content": [
    3. {"type": "image_url", "image_url": {"url": image_url}},
    4. {"type": "text", "text": user_query},
    5. ]}]
    6. tools = [
    7. {
    8. "type": "function",
    9. "function": {
    10. "name": "search_product",
    11. "description": "Look up price information for a product by name",
    12. "parameters": {
    13. "type": "object",
    14. "properties": {
    15. "keyword": {"type": "string", "description": "product keyword"}
    16. },
    17. "required": ["keyword"],
    18. },
    19. },
    20. }
    21. ]
    22. for _ in range(max_iterations):
    23. resp = requests.post(
    24. "https://api.example.com/v1/chat/completions",
    25. json={"model": "vl-model", "messages": messages, "tools": tools},
    26. headers={"Authorization": "Bearer YOUR_API_KEY"},
    27. ).json()
    28. msg = resp["choices"][0]["message"]
    29. messages.append(msg)
    30. if msg.get("tool_calls"):
    31. for call in msg["tool_calls"]:
    32. args = json.loads(call["function"]["arguments"])
    33. # replace with your actual tool execution logic
    34. tool_result = search_product_impl(**args)
    35. messages.append({
    36. "role": "tool",
    37. "tool_call_id": call["id"],
    38. "content": json.dumps(tool_result, ensure_ascii=False),
    39. })
    40. else:
    41. return msg["content"]
    42. return "Maximum number of iterations reached"

    7. Cloud-Enabled Delivery: Shipping Multimodal Capabilities at Scale

    In real deployments, enterprises often face limited compute, high operational complexity, and strict data-security requirements. General-purpose cloud services can lower the barrier here:

    • Container platforms: deploy the inference engine in containers and use elastic scaling to absorb traffic spikes.
    • Object storage: manage training data, image assets, and model weight files centrally.
    • Logging, monitoring, and alerting: collect runtime metrics from the inference service, spot anomalies early, and trigger alerts.
    • Message queues: smooth out peaks and fill in troughs in asynchronous workflows (such as batch document review) to improve system stability.

    Baidu AI Cloud, for example, offers general-purpose cloud products such as container services, object storage, logging services, and monitoring and alerting. Enterprises can combine these building blocks as needed to support a vision-language model inference service.

    8. Outlook

    Looking ahead, vision-language multimodal models are likely to move in four directions:

    1. Native omni-modal fusion: vision, speech, and video modeled uniformly inside one model, reducing conversion loss between modalities.
    2. On-device and cloud collaboration: lightweight models handle real-time processing on device, while complex tasks go to the cloud flagship model, forming a tiered inference system.
    3. Agentization: shifting from passive question answering to proactive planning, where the model decomposes tasks, calls tools, and executes in multiple steps on its own.
    4. Standardized evaluation: as capability boundaries keep expanding, the industry needs benchmarks that are more comprehensive and closer to real business scenarios.

    For developers and enterprise users alike, three things form the foundation for embracing the era of multimodal intelligence: understanding how the models work architecturally, mastering evaluation methods, and building engineering deployment capability.