Lightweight Flagship Multimodal Models: Inside the Native Vision Integration Architecture, Visual Token Budget, and 24-Layer ViT

    A practical breakdown of how today's mainstream lightweight flagship multimodal models deliver near-flagship GUI understanding and related capabilities while spending less than 1% of flagship-class parameters on the vision encoder. The article explains the native vision integration approach in detail: image preprocessing and the visual token budget (why 28×28 pixels maps to one token), the engineering tradeoffs behind a 24-layer ViT with roughly 0.56B parameters fixed at BF16, and the variable-length attention forward pass. It also covers axial RoPE, 2×2 spatial merging, SwiGLU projection, the visual feedback loop that makes native integration valuable, how to evaluate these models beyond public leaderboards, and deployment guidance for choosing between precision, latency, and cost.

    Introduction: The “Asymmetric Design” of Lightweight Flagships

    Over the past year, a clear trend has emerged in the multimodal model space: the rapid growth of so-called “lightweight flagship” models. These models are noticeably smaller than full flagships in parameter count, yet they aim to match flagship-level performance on high-frequency tasks such as image understanding, document parsing, and code generation.

    They typically adopt an asymmetric parameter allocation: the ViT backbone responsible for visual encoding accounts for less than 1% of the model’s total parameters, with the overwhelming majority reserved for the language model. For example, publicly available information on a class of lightweight flagship multimodal models indicates a vision encoder of roughly 0.56B parameters, run at a fixed BF16 precision during inference. Behind this ratio lies a repeated trade-off between the cost and the payoff of the visual stage.

    Fewer parameters, however, does not mean weaker capability. This article answers three questions:

    • On what kind of architecture does the visual capability of lightweight flagships rest?
    • What are the key implementation details and trade-offs?
    • What should developers watch for when selecting and deploying these models?

    To avoid vague discussion, this article takes publicly documented multimodal inference implementations as a reference, abstracts the “native vision integration” route commonly adopted by lightweight multimodal models, and dissects it piece by piece.

    I. From “Seeing Images” to Native Vision Integration

    1.1 What Vision Coding Really Means

    Vision coding is not about having the model attach a description to an image. It means feeding visual signals directly into the inference and generation process, making them objects that can be “written, modified, and verified,” including:

    • Turning UI screenshots into code;
    • Reconstructing layouts from design mockups;
    • Reading chart data and generating analysis scripts;
    • Observing execution results within an agent loop and self-correcting.

    This extends the boundaries of “programming.” Code tasks that once handled only structured inputs can now take in graphical interfaces, visualization reports, and scanned documents. Under this paradigm, “seeing accurately” is only the prerequisite. More importantly, the model must:

    1. Know when to look;
    2. Know where to look;
    3. Turn what it sees into the correct action.

    1.2 Bolt-On Pipelines vs. Native Integration

    Traditional multimodal systems mostly rely on serial, bolt-on pipelines, which differ from native integration across several dimensions:

    Dimension Bolt-On Pipeline Native Integration
    Typical form A standalone vision model recognizes or transcribes, then hands results to the language model Vision and text are jointly modeled within the same sequence
    Timing of observation Fixed: processed once before the pipeline begins Dynamic: decides on demand whether to look
    Information fidelity Reduced to text; layout, color, and other details are easily lost Preserves spatial layout and visual detail
    Feedback loop Hard to correct mid-process Can iterate based on new observations
    Training and deployment Multiple chained components; long pipeline End-to-end training; simple deployment unit

    This is exactly the point emphasized in some official technical documentation: vision coding is not merely about processing images; vision must be natively integrated into the model so that it can decide for itself when to look and use visual feedback to guide subsequent actions. For developers, this means vision shifts from a one-time input to an information source that can be invoked during inference.

    II. Overall Architecture: Preprocessing → Vision Encoder → Projection

    The visual side of mainstream lightweight flagship multimodal models breaks down into roughly four stages:

    1. Preprocessing: normalizing inputs of any size to a fixed granularity;
    2. Visual encoding: compressing images or video into visual tokens;
    3. Spatial merging: reducing the token count to decouple resolution from the context budget;
    4. Projection alignment: mapping to a representation space the language model can understand.
    1. Arbitrary-resolution image/video
    2. │
    3. ▼
    4. ┌──────────────────────┐
    5. │ Preprocessing │ Split into 14×14 patches
    6. │ (resize + zero-pad) │
    7. └────────┬─────────────┘
    8. ▼
    9. ┌──────────────────────┐
    10. │ Vision encoder ViT │ 24 layers · ~0.56B params
    11. └────────┬─────────────┘
    12. ▼
    13. ┌──────────────────────┐
    14. │ Spatial merge + proj. │ 2×2 merge · map to LM hidden dim
    15. └────────┬─────────────┘
    16. ▼
    17. Visual tokens → Language model

    The value of this layering lies in clear separation of responsibilities: preprocessing is responsible for resolution, the encoder for visual semantics, and merging for cost. Each stage can be tuned and evaluated independently, and it is easy to reuse across different model versions.

    Based on public information, the implementations of such encoders often trace a lineage across generations of models—from general visual understanding to OCR-specific designs to lightweight flagships. This code evolution path is stable, signaling that visual encoding is becoming a “standard component,” and suggesting that a mature implementation is more reliable than one designed from scratch.

    III. Preprocessing: Where the Number 28 Comes From

    3.1 Three Steps

    Before entering the model, an image goes through a standardized processor in roughly three steps:

    1. Resize and zero-padding: scale proportionally, then pad the remainder with zeros up to a multiple of 28;
    2. Split into patches: cut into 14×14-pixel patches;
    3. 2×2 merge: combine each adjacent group of 2×2 (four) patches into one.

    28 = 14 (patch side length) × 2 (merge factor), so each 28×28-pixel region ultimately corresponds to one visual token. You can think of 28 pixels as the minimum “photosensitive unit” on the visual side. Aligning to a multiple of 28 rather than 14 guarantees that every image can be fully divided by 2×2 in both height and width, avoiding boundary problems where a merge block is split in half.

    3.2 Controlling the Token Budget

    The number of visual tokens for a single image is roughly “image width ÷ 28” times “image height ÷ 28,” bounded by lower and upper limits (configuration items are often named min_image_tokens / max_image_tokens, with a common industry default range of 16 to 8000). This mechanism serves three purposes:

    • Decoupling resolution from context: a tall or wide image need not generate a massive number of tokens at its native size;
    • Predictable resources: capacity planning can be done directly from token counts;
    • Adjustable quality: different tasks can set different levels of detail.
    1. # Illustrative code: preprocessing and the visual token budget (pseudocode)
    2. PATCH, MERGE = 14, 2
    3. ALIGN = PATCH * MERGE # 28, the pixel side length per visual token
    4. MIN_TOKENS, MAX_TOKENS = 16, 8000
    5. def preprocess(image, dynamic=True, fixed_size=None):
    6. """Returns an image aligned to a multiple of 28;
    7. token count = (h' // 28) * (w' // 28)"""
    8. w, h = image.size
    9. if dynamic:
    10. # Treat 28×28 pixels as the "photosensitive unit";
    11. # determine the scaled target size from the token budget
    12. max_area = MAX_TOKENS * ALIGN * ALIGN
    13. if w * h > max_area:
    14. scale = (max_area / (w * h)) ** 0.5
    15. w, h = round(w * scale), round(h * scale)
    16. # Common range for the target token count;
    17. # actual scaling and fine-tuning depend on the implementation
    18. target = min(max(round(w * h / (ALIGN * ALIGN)), MIN_TOKENS), MAX_TOKENS)
    19. w, h = grid_for_budget(w, h, target) # Map to multiples of 28 by aspect ratio
    20. else:
    21. w, h = fixed_size
    22. return resize_pad(image, w, h) # Resize + zero-pad; w and h are multiples of 28

    3.3 A Set of Reference Conversions

    Estimating one token per 784 (28×28) pixels:

    Input Pixel count Estimated tokens Exceeds the 8000 limit?
    Phone screenshot (1170×2532) ~2.96M ~3776 No
    1080P (1920×1080) ~2.07M ~2645 No
    4K (3840×2160) ~8.29M ~10580 Yes
    Single A4 page (150 dpi scan) ~1.24M ~1585 No

    At runtime, the processor applies fine scaling or adjustments based on the aspect ratio so that width and height come close to multiples of 28 and the final token count stays within the limit. The estimates in the table serve as an order-of-magnitude reference for capacity planning.

    3.4 Practical Recommendations

    • Set budget tiers per task. For example, tasks such as invoice and contract recognition and UI reconstruction need a higher budget, while rough visual retrieval can use a lower budget to reduce latency;
    • Watch how the lower bound affects fine detail. Small targets such as icons and corner markers are easily lost at a low budget, so evaluate them separately;
    • Tile large images on the engineering side. Long screenshots or long documents can be split, encoded separately, and then summarized by the language model, avoiding a single oversized image budget that squeezes out text context.

    IV. The Vision Encoder: Engineering Rationality in the Parameter Trade-offs

    The typical parameter configuration of this class of vision encoder is as follows:

    Item Value
    Depth (layers) 24
    Hidden size 1024
    Attention heads 16
    Per-head dimension 64 (1024 / 16, no GQA)
    Patch size / merge 14 / 2×2
    Output dimension 4096 (aligned with the language model hidden dim)
    Backbone parameters ~0.56B
    Inference precision Fixed BF16, no FP8 quantization

    Behind these numbers is a judgment about cost and payoff:

    • Per-head dimension of 64: a common alignment convention in attention implementations, convenient for efficient execution;
    • No GQA: grouped-query attention (GQA) is mainly used to compress the KV cache during autoregressive decoding. Visual encoding is a one-shot forward pass with no pressure to reuse a KV cache, so GQA offers limited benefit and may weaken the discriminative power of fine-grained visual features;
    • Output dimension of 4096: it must align with the language model’s hidden dimension; otherwise the projection layer can only add an extra linear transform, needlessly increasing parameters and error;
    • Less than 1%: for a model with total parameters in the hundreds of billions, a 0.56B visual backbone accounts for only about 0.5%. A “lean” visual side paired with a “thick” language side is the common form of lightweight flagships.

    Why Fix BF16 Instead of Moving to FP8

    There are two main reasons. First, the vision encoder has few parameters, so the memory and bandwidth gains from quantization are inherently limited. Second, visual tasks are sensitive to numerical error: fine strokes, light backgrounds, and small font sizes tend to blur under low precision. Trading for stability with BF16 is a reasonable, conservative choice.

    That said, if memory is tight, you can evaluate precision strategies per module. The impact of the vision encoder’s precision should be validated on your own task set rather than copied from others’ conclusions.

    V. Dissecting the Inference Flow: Six Steps in the Forward Function

    The illustrative code below shows the core forward flow of the vision encoder:

    1. # Illustrative code: forward pass of a lightweight vision encoder (pseudocode)
    2. def vision_forward(pixel_values, grid_twh, cfg):
    3. # grid_twh: patch grid per image/video segment, e.g., [1, H/14, W/14]
    4. position_ids = get_vision_position_ids(grid_twh, cfg.merge_size)
    5. seq_lens, max_seq_len = get_vision_seq_lens(grid_twh, cfg)
    6. # seq_lens: boundaries between images; max_seq_len: patches of a single image
    7. x = patch_embed(pixel_values) # (1) Conv3d patching: 1 patch -> 1 vector
    8. x = apply_axial_rope(x, position_ids) # (2) Axial RoPE positional encoding
    9. for blk in blocks: # (3) 24 × Block (variable-length attention + SwiGLU MLP)
    10. x = blk(x, seq_lens=seq_lens, max_seq_len=max_seq_len)
    11. x = post_layernorm(x) # (4) Final normalization
    12. x = x.reshape(-1, 2, 2, cfg.hidden) # (5) Concatenate adjacent 2×2 patches into the channel dim
    13. x = x.transpose(0, 3, 1, 2) # -> [N, C*4, 2, 2]
    14. x = downsample(x).reshape(-1, cfg.out_hidden) # Conv2d downsampling + dimension reduction
    15. return merger(x) # (6) SwiGLU projection to the language model hidden dim

    5.1 Conv3d Patching

    The vision encoder uses a 3D convolution to cut patches instead of the more common 2D convolution, so that images and video share the same code path: for an image the number of “temporal patches” is 1, while for video it is grouped by duration.

    For images, both the kernel and stride are (1, 14, 14), equivalent to “one patch per 14×14 pixels,” then a linear projection produces a 1024-dimensional vector. The benefit of the 3D implementation is that it can process two adjacent frames as a group, compressing video tokens at the source.

    5.2 Axial RoPE

    Axial rotary position encoding decomposes high-dimensional positions into different subspaces, encoding the 2D spatial position separately (with the temporal dimension added on top for video). The benefits are:

    • The same set of rotation parameters is reused across both dimensions, which is parameter-efficient;
    • Positional information enters attention as a relative rotation, providing a degree of extrapolation capability.

    5.3 The 24-Layer Block and Variable-Length Attention

    Blocks typically use a “normalize → attention → residual → normalize → feed-forward → residual” structure. The key is that the attention layer receives two quantities:

    • seq_lens (cu_seqlens in the code): the cumulative lengths of the patch segments corresponding to each image; for example, if two images have 2645 and 1585 patches respectively, the sequence is [0, 2645, 4230];
    • max_seq_len: the patch count of the longest single image, used by the attention kernel to determine blocking and scheduling parameters.

    This yields two benefits:

    1. Multiple images can be concatenated into one sequence and sent together with the text in a single pass, rather than being invoked one by one;
    2. Attention is strictly confined within a single image, so patches from image A never “see” patches from image B.

    What to watch out for is length consistency. The padded patch count must stay consistent with the grid metadata, or the boundaries will misalign. In batched inference, when multiple images have different grids, the usual approach is to split by image and run the visual forward pass separately, then concatenate the outputs.

    5.4 Final Normalization and 2×2 Merging

    post_layernorm unifies the output scale across samples, making it easier for the subsequent projection and cross-task reuse. Then reshape + transpose concatenates the four patch vectors within each 2×2 neighborhood into the channel dimension, changing the channel count to 4×1024 = 4096 and the sequence length to one-quarter of the original.

    The order here cannot be swapped arbitrarily. You must first group by spatial adjacency and then concatenate; otherwise the merged tokens will mix patches that are spatially unrelated.

    5.5 Downsampling and SwiGLU Projection

    downsample is a 2D convolution with stride 2, taking 4096 input channels and outputting the language model’s hidden dimension. After completing the merge that raises and then reduces the dimension, the result finally passes through a SwiGLU feed-forward layer (the merger in the code) to produce the output.

    Using SwiGLU instead of a plain linear layer is because the merged channel dimension carries information from four neighboring patches, which needs a nonlinear transform to reorganize. The overall output is a visual representation of “one token per 28×28 pixels.”

    VI. The Visual Feedback Loop: The Real Value of Native Integration

    Architecture is only the foundation; the value of native integration shows up in the feedback loop. Take “reconstructing a UI from a screenshot” as an example:

    1. # Illustrative code: visual feedback loop
    2. reference = capture(page) # Reference screenshot (provided externally or from the first round)
    3. plan = llm(image=reference, task="reconstruct this page") # First generation
    4. result = render(plan) # Render execution
    5. diff = compare(result, reference) # Visual comparison (structured diff + screenshot diff)
    6. while not accept(diff) and budget_left:
    7. # The model decides "where to look next": overall thumbnail + zoomed diff regions
    8. view = llm.choose_view(diff, budget=image_tokens)
    9. plan = llm.revise(plan, image=view, diff=diff)
    10. result, diff = render(plan), compare(render(plan), reference)
    11. return plan

    This loop has three engineering points:

    1. Give the decision to the model: whether to keep observing and which region to look at are judged by the model based on task context, rather than a hard-coded fixed number of rounds;
    2. Budget constraint: the visual tokens consumed by each observation should be counted against the context budget, prioritizing the regions with the largest differences;
    3. State management: keep only summaries of past observations and re-fetch the original screenshots on demand, avoiding context bloat.

    This also explains why resolution and the token budget matter so much. The finer the observation, the more accurate the feedback; and the more economical the budget, the more rounds the loop can sustain.

    VII. How to Evaluate: Don’t Just Look at Public Leaderboards

    What is most easily misread about lightweight flagships is “performance.” Public leaderboards reflect average capability, whereas the long tail of visual tasks is pronounced. It is advisable to build your own task set and evaluate across the following dimensions:

    • Recognition: small font sizes, low contrast, tilted text;
    • Charts: reading axes, unit conversion, dual-axis charts;
    • Interfaces: layout reconstruction, control localization, state determination;
    • Documents: cross-page tables, headers and footers, obscured stamps;
    • Video: temporal understanding, keyframe localization.

    At the same time, track three metrics: task accuracy under different token budgets, which reflects the capability ceiling of the visual side; time-to-first-token latency, mainly driven by visual encoding time; and visual tokens per request, which directly determines cost.

    1. # Illustrative code: an experiment on the trade-off between token budget and capability
    2. for n_tokens in (256, 512, 1024, 2048, 4096):
    3. acc = evaluate(image_set, max_tokens=n_tokens)
    4. cost = estimate_cost(image_set, max_tokens=n_tokens)
    5. print(n_tokens, acc, cost) # Find the "diminishing-returns point": where more tokens add little

    VIII. Engineering Deployment Recommendations

    1. Precision strategy: the vision encoder can stay at BF16; evaluate low-precision inference for the language model depending on resources, and run regression tests on your own task set.
    2. Scheduling strategy: split visual encoding per image and execute with variable lengths; preprocessing (decoding, resizing, color-space conversion) is mostly CPU-intensive, so run it as a separate, asynchronous service to avoid blocking the GPU.
    3. Observability: log the visual token count per request, per-stage latency, and memory usage to catch abnormally large images or misconfigured budgets promptly.
    4. Cloud deployment: whether on Baidu AI Cloud or another cloud platform, you can use object storage to hold samples and intermediate results, container services to host the stateless preprocessing service, log services to record the request chain, and monitoring and alerting to track latency and memory watermarks—forming a complete “preprocess → infer → observe” loop.

    IX. Conclusion

    The core logic of lightweight flagship multimodal models is to support the overall visual capability with a minimal vision encoder:

    • A 24-layer ViT with 0.56B parameters, through 14×14 patches and 2×2 merging, unifies inputs of arbitrary resolution into “one token per 28×28 pixels”;
    • It uses min/max_image_tokens to control the per-image cost, decoupling resolution from the context budget;
    • Through axial RoPE, variable-length attention, and SwiGLU projection, it natively wires the visual signal into the language model.

    The real difference lies not in the number of parameters, but in whether vision becomes an information source the model can actively invoke and use to guide its actions. For developers, understanding this pipeline helps strike a balance among precision, latency, and cost, and truly brings multimodal capability into business scenarios.