Vision Transformer Encoder Design for Joint Training from Scratch in Multimodal LLMs
This article examines the design of a vision encoder for multimodal large language models that is trained jointly with the language model from random initialization. It covers a 27-layer bias-free vision Transformer with about 401M parameters, 2×2 pixel shuffle for visual token reduction, shared image-video parameters, fixed-ratio data mixing, gradient control, quantization-aware training, and high-resolution long-context deployment, along with evaluation considerations.
1. Technical Background: Why Vision Encoders Need Redesign
In multimodal large language models, the vision encoder converts non-text inputs such as images and videos into continuous vector sequences. The encoded visual vectors are typically fed into the language model together with text tokens, enabling tasks such as image understanding, chart analysis, visual question answering, video captioning, and complex reasoning.
Traditional approaches often train a visual representation model separately and then connect it to a language model for cross-modal adaptation. This approach has advantages such as relatively mature engineering implementation and stable basic visual capabilities, but it also has clear limitations:
- Visual objectives and language objectives diverge. Independent visual pretraining usually focuses on image classification, image-text alignment, or contrastive learning, while a multimodal language model ultimately needs to perform continuous text generation and next-token prediction. Representations learned by the vision encoder are not necessarily well suited for use by the language model.
- Multi-stage training can create optimization discontinuities. When a pretrained vision module is jointly trained with a language model, the two modules may differ in parameter distribution, gradient scale, and learning pace, which can lead to loss fluctuations or gradient shocks during training.
- Visual token counts can grow rapidly. High-resolution images provide more detail, but they also produce many visual tokens, directly increasing attention computation and context usage.
- Images and videos often require different processing paths. Designing separate encoders for images and videos increases parameter count and can also create inconsistent representation spaces between the two types of visual input.
To address these issues, a new class of vision encoder designs emphasizes starting from random initialization, training jointly with the language model directly, and controlling visual sequence length through more efficient spatial downsampling.
2. Overall Architecture: A 27-Layer Bias-Free Vision Transformer
This design uses a 27-layer vision Transformer encoder with approximately 401 million parameters. Unlike some traditional Transformers, its network layers use a bias-free design, meaning linear mappings and related projection layers do not introduce additional bias terms.
A bias-free structure has several potential benefits:
- Reducing a small amount of redundant parameters. Although bias terms account for a small proportion of total parameters, they can still simplify parameter management in large-scale networks.
- Reducing distribution shifts between modules. With normalization, linear projections, and residual connections working together, a bias-free design helps make the network structure more regular.
- Facilitating low-precision training and inference. A unified matrix computation path is better suited to hardware acceleration and also helps with numerical control in quantization-aware training.
The basic flow of the vision Transformer can be summarized as:
Input image or video frames↓Patch partitioning and local pixel rearrangement↓Visual token embedding↓Multi-layer Transformer encoding↓Visual vector sequence↓Joint modeling with text tokens
Each Transformer layer typically includes multi-head self-attention, normalization, a feed-forward network, and residual connections. For video input, multiple frames can be treated as a temporally ordered sequence of visual tokens. Under shared encoder parameters, spatial structure and inter-frame relationships can be distinguished through positional or temporal information.
3. 2×2 Pixel Shuffle: Reducing Visual Token Count
High-resolution input is an important foundation for improving visual understanding, but higher resolution also leads to longer visual sequences. Suppose an image is initially partitioned into a visual grid of height H and width W. The number of visual tokens is roughly proportional to H×W. For high-resolution images, the token count can easily reach thousands or more.
This design introduces a 2×2 pixel shuffle operation for spatial compression. Its core idea is not to simply discard pixels, but to rearrange adjacent 2×2 local regions into the channel dimension and then complete information fusion through subsequent projections. This preserves local detail while compressing the spatial size to one quarter of the original.
The process can be illustrated as follows:
Original local region:[a b][c d]After pixel shuffle:[a, b, c, d] → reorganized along the channel dimension
Compared with direct pooling, this method retains more local information. Compared with no downsampling at all, it significantly reduces the number of tokens that the subsequent Transformer must process. The benefits are mainly reflected in three areas:
- Lower attention computation. Self-attention computation usually grows quadratically with sequence length, so reducing visual tokens can clearly lower computational cost.
- More effective context space. Under a fixed context window, fewer visual tokens leave more positions for text, reasoning processes, and output content.
- Support for higher-resolution input. When input reaches the level of thousands of pixels, reasonable spatial compression is necessary for practical deployment.
It is important to note that the downsampling ratio cannot be increased indefinitely. Excessive compression can lose key details such as small fonts, thin lines, formula symbols, and chart markers. Engineering practice therefore needs to balance token efficiency against local information fidelity.
4. Shared Parameters for Images and Videos
The vision encoder uses one set of parameters to process both images and videos, rather than maintaining separate image and video models. Direct benefits of parameter sharing include more controllable model size and a more unified representation space for image and video representations.
For image input, the encoder mainly learns:
- Object and scene structure;
- Layout of text, tables, and formulas;
- Spatial relationships between local regions;
- Color, texture, and geometric information.
For video input, the model also needs to handle:
- Inter-frame changes;
- Action and event order;
- Object persistence;
- Temporal causality;
- Scene transitions and motion patterns.
Shared parameters do not mean that images and videos use exactly the same input organization. Practical systems usually still need to add spatial position, frame order, or other temporal identifiers to tokens so the model can distinguish “spatial relationships within the same frame” from “temporal relationships between different frames.”
This unified encoding approach helps reduce duplicate training and makes it easier for the model to transfer static visual knowledge to dynamic scenes. For example, the model can first use image capabilities to recognize objects, then combine multi-frame changes to determine object actions and event development.
5. Training from Scratch: Making Visual Representations Serve Language Generation
A key design choice is that the vision encoder does not rely on external visual pretraining weights. Instead, it starts from random initialization and is jointly trained with the language model. The training objective centers mainly on next-token prediction.
From an optimization perspective, the model learns:
Given text history and visual contextpredict the next text token
This means the vision encoder is no longer separately optimized for a particular visual classification metric. Instead, it is directly required to provide visual information that helps the language model generate correct answers.
This approach has the following advantages:
5.1 Direct Objective Alignment
If a user asks the model to read a table, explain a mathematical figure, or describe video events, the vision encoder should extract features relevant to language expression. Joint training allows the encoder to adjust its representations according to the final generation task rather than remaining limited to independent visual objectives.
5.2 Reduced Module Interface Mismatch
In traditional multi-stage approaches, the vector distribution output by the vision encoder may be inconsistent with the input space expected by the language model, requiring additional projection layers or adaptation modules for compensation. Joint training from scratch allows both modules to form a more consistent representation space early in training.
5.3 Less Reliance on External Initialization
External pretrained weights can provide strong initial visual capabilities, but they may also carry representation preferences that do not fully match the current language task. Fully joint training gives the model greater freedom to adapt to the task.
6. The Importance of Fixed-Ratio Mixed Image-Text Data
Joint pretraining is not simply random mixing of text data and visual data. To prevent one type of data from dominating training, a relatively stable data ratio needs to be set, and loss changes across different tasks should be continuously monitored.
If the visual data ratio is too low, the vision encoder may receive insufficient gradient signals, ultimately resulting in inadequate image understanding. If the visual data ratio is too high, it may weaken the model’s pure-text modeling ability or cause the language model to over-adapt to visual input scenarios.
In large-scale training, data mixing strategies usually need to consider:
- Text length distribution;
- Image resolution distribution;
- Number of video frames and time span;
- Proportion of special data such as charts, formulas, and documents;
- Completeness and accuracy of supervised answers;
- Whether image-text pairs contain semantic noise.
Joint training data can reach the trillion-token-level mixed token scale. At this scale, data quality, sampling ratios, and deduplication strategies are often as important as network architecture. A stable data distribution allows the vision encoder to continuously receive effective gradients and reduces sudden fluctuations during training.
7. Training Stability and Gradient Control
One difficulty of multimodal joint training is that gradient statistics differ considerably across modalities. Text token count, visual token count, image resolution, and video frame count all affect the computational scale and gradient magnitude of a single training step.
When designed properly, joint training from scratch can reduce gradient shocks caused by suddenly connecting external modules. However, this does not mean training is naturally stable. The following engineering measures are still needed:
- Reasonable learning rate warmup. Gradually increase the learning rate in the early stage of training to avoid overly strong updates to a randomly initialized network.
- Gradient clipping. Limit abnormally large gradients to prevent a single high-resolution sample from affecting the entire batch.
- Modality-balanced sampling. Prevent text or visual data from suppressing the learning signal of the other modality over long periods.
- Monitor modality-specific losses. In addition to total loss, record text, image, and video-related metrics separately.
- Control visual sequence length. Reduce the impact of extreme samples through pixel shuffle, frame sampling, and dynamic resolution strategies.
Training stability should not be evaluated only by loss curves. Attention should also be paid to whether visual question answering, chart parsing, formula understanding, video captioning, and pure-text tasks improve together. If a certain type of metric drops significantly, it usually indicates that data mixing, learning rate, or visual token budget needs to be adjusted.
8. Quantization-Aware Training and Deployment Efficiency
During deployment, this type of encoder can use low-precision weight and activation representations, such as lower-bit-width weight formats and slightly higher-bit-width activation formats, while simulating quantization error in advance during training.
The basic idea of quantization-aware training is:
Floating-point training parameters↓Simulated quantization and dequantization↓Forward computation↓Update parameters with approximate gradients
Compared with quantizing directly after training, quantization-aware training allows the model to adapt to precision loss in advance. It is particularly suitable for attention projections, feed-forward networks, and large-scale matrix multiplication in vision Transformers.
However, low-precision deployment requires close attention to:
- Numerical stability of normalization layers;
- Whether attention scores exhibit abnormal overflow;
- Whether recognition of small objects and fine-grained text degrades;
- Error variation under different input resolutions;
- Whether image and video tasks show inconsistent quantization sensitivity.
Therefore, quantization should not be judged only by average accuracy. Dedicated validation should also be performed on high-resolution images, complex charts, and long videos.
9. Coordination of High Resolution and Million-Level Context
This vision encoder supports image inputs up to approximately 3584×3584 and can incorporate visual information into million-level context scenarios. The key is not only “being able to accept large images,” but also controlling visual token usage, organizing long context, and avoiding interference from irrelevant regions during reasoning.
In practical applications, a layered strategy can be used:
- First use lower-cost global visual encoding to obtain the overall layout;
- Apply local high-resolution processing to regions suspected of containing key content;
- Sample and compress videos by time segment;
- Retain the visual segments most relevant to the question before text generation.
For documents, tables, and mathematical images, global scaling alone cannot be relied upon. Excessive shrinking can cause characters, superscripts and subscripts, and thin-line structures to be lost. Therefore, the visual token budget should be allocated dynamically according to the task, balancing overall understanding and local detailed reading.
10. Applicable Scenarios and Evaluation Methods
This type of vision encoder is suitable for the following tasks:
- High-definition document and complex page understanding;
- Table structure recognition and content question answering;
- Analysis of mathematical formulas, geometric figures, and scientific diagrams;
- Multi-frame video content summarization;
- Mixed image-text reasoning;
- Analysis of multi-image and multi-page materials in long contexts.
Evaluation should cover three dimensions at the same time: capability, efficiency, and stability.
Capability metrics include visual question answering accuracy, text recognition performance, chart parsing quality, video event understanding, and complex reasoning performance.
Efficiency metrics include the number of visual tokens produced per image, memory usage, inference latency, and long-context throughput.
Stability metrics include performance variation across resolutions, degradation across video lengths, differences before and after quantization, and whether abnormal peaks appear in training loss.
11. Conclusion
For vision encoders in multimodal large language models, the focus is shifting from “acquiring visual capabilities separately” to “directly serving language modeling.” Based on a 27-layer bias-free vision Transformer, this design reduces visual token count through 2×2 pixel shuffle, shares parameters between images and videos, and adopts joint image-text pretraining from scratch. This creates a tighter connection among model scale, visual detail, context efficiency, and training objectives.
The core value of this type of design is not only parameter count or input resolution, but the optimization of visual encoding, language prediction, context management, and low-precision deployment as an integrated system. For multimodal systems that need to handle high-resolution images, complex documents, video content, and long-context information, this end-to-end, objective-aligned architectural approach offers strong engineering reference value.
