Multi-Size On-Device Multimodal Models: Architecture Breakdown and Selection Guide

Not long ago, visual understanding and ultra-long context were almost exclusively the domain of cloud-based large models: with limited on-device compute and memory, even running a text-only chat model was a challenge. But the release cadence of the open-source community over the past year makes it clear this landscape is being rewritten. From sub-1B ultra-compact sizes to medium sizes around 9B, this new wave of on-device models natively supports image-text understanding across the entire lineup, and most push context windows up to the 256K level. For developers, what truly matters is not “yet another release,” but what these models got right architecturally, and how to choose among different sizes.

1. Small Models Are Collectively Catching Up on “Eyes” and “Memory”

This shift is driven by three factors:

• Hardware: Mobile AI accelerator compute and memory bandwidth improve year over year, INT4/INT8 quantized inference pipelines have matured, and “large models on phones” has moved from demos to products;
• Architecture: New structures such as hybrid attention have compressed the memory and compute overhead of long contexts to a level that on-device hardware can bear;
• Demand: Scenarios like offline translation, screenshot Q&A, document summarization, smart cockpits, and local code assistants inherently require models that can “understand images, read long documents, and work offline.”

2. One Architecture Across All Tiers: Multi-Size Scaling Under a Unified Design

Taking a representative multimodal small-model family from the recent open-source community as a reference, all tiers share the exact same conditional generation backbone structure and the same model implementation, with systematic scaling applied only to a few dimensions such as language layer count, hidden dimension, and vision encoder configuration. The general pattern is as follows:

Size Tier	Language Layers	Hidden Dim	Vision Layers	Vision Hidden Dim	Context
Sub-1B	24	~1K	12	~768	256K
~2B	24	~2K	24	~1K	256K
~4B	32	~2.5K	24	~1K	256K
~9B	32	~4K	27	~1.2K	256K

Three layers of insight can be read from this scaling curve:

1. Small sizes go “wide but not deep”; mid-to-large sizes go “deep and wide together.” Sub-1B and ~2B both use 24 layers, with only the hidden dimension doubled. At the 4B/9B tier, layers deepen to 32 and hidden dimensions expand in tandem. Why do small models prioritize width? Because wider dimensions improve hardware utilization for matrix multiplication, yielding immediate throughput gains. Increasing layer count amplifies sequential dependencies and operator invocation overhead, which is cost-ineffective for small-parameter models. Only when the parameter budget is large enough do the hierarchical representation and reasoning benefits of depth clearly outweigh the cost.

2. The “eyes” also scale by tier. Vision layers increase from 12 to 27, and vision hidden dimensions grow progressively, meaning larger tiers have stronger perception of high-resolution, dense text (documents, tables, charts). If your workload is primarily OCR-dense documents, prioritize tiers with higher vision configurations.

3. A unified architecture delivers real engineering dividends. All tiers share the same tokenizer, the same inference code and quantization pipeline, and the same fine-tuning and distillation recipes: on the training side, data curricula can be reused to distill capabilities from large tiers to small ones; on the deployment side, one adaptation covers all device models. This “model family” approach is becoming the mainstream strategy in the open-source community.

3. Native Multimodal: “Grown Together” From Pre-Training

Many early multimodal approaches were “post-hoc stitching”: train a text model first, then attach a vision adapter to “translate” image features to the text side. This yields low parameter efficiency and shallow image-text alignment, with small sizes faring especially poorly. The new generation of on-device models generally shifts to a unified vision-language foundation route: from the pre-training stage onward, images and text are jointly trained in the same representation space, with image tokens and text tokens participating in attention computation on equal footing.

For on-device developers, the direct benefits are threefold: first, OCR, visual Q&A, and chart understanding no longer require separate “OCR model + text model” services, halving memory and latency budgets; second, small sizes now have usable image-text capabilities, with visual information deeply participating in reasoning rather than shallow concatenation; third, it aligns with cloud-side large-model paradigms, so prompts and interaction protocols can be reused in cloud-edge collaboration.

4. Hybrid Attention: The Decisive Factor for On-Device Long Context

Why was 256K context previously unimaginable on-device? The bottleneck is the KV cache of full attention. A rough estimation formula:

KV cache ≈ 2 × num_layers × seq_length × KV_channels × precision_bytes

Take a 32-layer model with a hidden dimension of about 4K: even with grouped-query attention compressing KV head count, at FP16 precision every 1,000 tokens adds over a hundred MB of cache. At full 256K context, the cache alone could reach tens of GB — no phone can handle that.

Linear attention takes an entirely different approach: it rewrites attention as a fixed-size recurrent state, performing only one state update per token processed. Compute grows linearly with sequence length, and state size is independent of context length. Gated delta-rule-style mechanisms — which use error-corrective updates that “first remove historical components conflicting with new input, then write in new information” — clearly outperform naive linear attention on precise retrieval and are a currently prominent technical route. However, purely linear structures still have shortcomings for “needle-in-a-haystack” precise recall, so new models generally adopt hybrid attention: the vast majority of layers use linear attention for long-sequence compression, while a small number of full-attention layers are retained for precise retrieval and key-position anchoring.

The implications for on-device deployment can be summarized in three points: Controllable memory — cache no longer grows unboundedly with context, making 256K truly deployable; Faster prefill — pasting a long contract no longer means waiting half a minute; Better NPU alignment — regular matrix operations suit on-device accelerators. One engineering detail to note: hybrid attention is a non-standard structure. Before shipping, verify that your inference framework supports this structure and its quantized operators; otherwise it may fall back to CPU execution and run slower, not faster.

5. How to Actually Use 256K Long Context On-Device

The long-context scenarios worth bringing on-device are quite specific: local knowledge Q&A (pairing with retrieval to inject relevant snippets, usable offline or in intranet environments), long document processing (contract comparison, financial report skimming, meeting transcription summarization), multi-turn long conversation memory, and stuffing key files from a mid-sized project into context for local code review.

Two rational reminders: first, a nominal context length does not equal lossless understanding length — long contexts commonly exhibit “lost in the middle,” so place critical information at the beginning or end; second, the more realistic on-device architecture is a combination of “long context + structured retrieval,” rather than treating 256K as a universal container to cram everything into.

6. On-Device Benchmarking Methodology: Don’t Just Look at Leaderboards, Run Your Own Cases

After getting a new model, evaluate it with “three device tiers + four metric categories + one set of your own use cases”:

for case in eval_set:              # one set each for image-text, long text, instruction following
    t0    = now()
    first = model.generate(prompt, max_new_tokens=1)
    ttft  = now() - t0              # time to first token: determines "instant response" feel
    full  = model.generate(prompt, max_new_tokens=512)
    tps   = 512 / (now() - t0)      # generation throughput: determines long-output efficiency
    mem   = peak_memory()           # peak memory: determines device coverage
    log(power, temperature)         # power draw and thermal: determines sustained experience

For device coverage, pick three tiers: entry-level (about 4–6GB available memory), flagship (8–12GB), and edge workstation (16GB+). Real-device benchmarking is time-consuming, so you can use AI coding tools to quickly generate evaluation scaffolding, containerize the tasks, and hook them into a cloud CI pipeline. For example, orchestrating evaluation tasks on a container platform on Baidu AI Cloud, persisting evaluation artifacts for each version in object storage, and pairing with logging and monitoring alerts to catch metric degradation — this is a common and low-effort combination.

Three key points for use-case design: image-text cases should cover “dense-text OCR / natural-scene Q&A / chart understanding”; long-text cases should include both “needle retrieval” and “global summarization” groups; then add a set of business-specific instruction-following cases. Leaderboard scores can only serve as an initial filter — final selection must be based on real measurements with your own cases.

7. Selection Advice: Two-Dimensional Decision by “Device Tier × Task Complexity”

Deployment Target	Memory Budget	Recommended Tier	Typical Tasks
Wearable/IoT terminal	<2GB	Sub-1B	Intent recognition, simple Q&A, single-image tagging
Mid-range phone/tablet	4–6GB	~2B	Chat assistant, translation, summarization, lightweight visual Q&A
Flagship phone/thin laptop	8GB+	~4B	Screenshot Q&A, document understanding, lightweight Agent
Edge box/workstation	16GB+	~9B	Local code assistant, multi-image reasoning, Agent orchestration

Three supplementary considerations: Memory is the first constraint — under INT4 quantization, weights take roughly half the parameter count in bytes; a 9B model’s weights are about 4.5–5GB, and adding the vision tower, activations, and KV cache, resident memory should be budgeted at 6–8GB. Don’t blindly go bigger — if a task can be reliably solved by 2B, moving to 4B only brings worse concurrency and heat. Cloud-edge collaboration is the endgame — the device handles low-latency, privacy-sensitive, and offline paths, while complex tasks go to the cloud. Platforms like Baidu AI Cloud with service orchestration and unified monitoring capabilities can help you build this upgrade path relatively quickly.

8. Pre-Deployment Checklist

1. Quantize before shipping: run accuracy regression on the target tier before and after INT4/INT8 quantization, focusing on the long tail of image-text tasks;
2. Confirm the inference framework’s support for hybrid attention structures, and verify in practice that it does not fall back to CPU;
3. Build an evaluation baseline using your own business data and incorporate it into continuous regression;
4. Design a degradation strategy: automatically fall back to a smaller tier when the larger model fails or memory runs tight, ensuring service availability;
5. The iteration cycle for on-device small models has shortened to monthly intervals — selection conclusions need periodic review.

Overall, the technical throughline of these on-device multimodal models is very clear: unified architecture for engineering efficiency, native multimodality for capability ceiling, and hybrid attention for on-device long-context feasibility. For small teams, rather than chasing parameter-count rankings, it’s better to invest effort in three things — “device coverage × task matching × empirical regression.” The on-device AI dividend period has only just begun.