Open-Source LLM Architecture Evolution and Multimodal Model Construction Methods
Open-source large language models are moving from decoder-only text generation toward multimodal perception and reasoning. This article reviews the architectural trends behind that shift—including efficient attention, Mixture-of-Experts sparsity, long-context mechanisms, KV-cache compression, and speculative decoding—and then compares the two dominant approaches to multimodal construction: unified embedding decoders and cross-modal attention architectures. It also examines tokenizer design, visual projection, visual position encoding, staged training, distributed deployment trade-offs, and emerging directions such as native multimodal representation learning and inference-time computation. Rather than promoting a specific model or product, the article provides a technical decision framework for teams balancing capability, latency, cost, data requirements, and engineering complexity when selecting an architecture.
1. Introduction: From Unimodal to Multimodal Architecture Shifts
Over the past two years, the capability boundaries of large language models have expanded steadily—from plain text processing to image-text understanding, audio-video perception, and more complex cross-modal reasoning. Model architecture is the structural foundation that carries these capabilities, and its design philosophy has undergone rapid iteration: from early Encoder-Decoder frameworks, to the establishment of Decoder-only dominance, and now to the pressure that multimodal input requirements place on architecture innovation. The industry is at a key point of paradigm transition.
For developers, understanding the mainstream architectural categories of current open-source large models and the technical paths for multimodal extension is not only a prerequisite for technology selection. It is also necessary knowledge for optimizing model quality, controlling inference cost, and designing data pipelines in real business scenarios. This article sets aside product-specific marketing and takes a purely technical perspective. It systematically reviews the architectural characteristics of representative models in the current open-source ecosystem and analyzes two mainstream methodologies for building multimodal models, with the goal of providing technical teams with a practical reference for architecture decisions.
2. Mainstream Classification and Evolution of Open-Source LLM Architectures
Although current open-source large models differ in their architectural details, a few clear lines of evolution emerge once surface differences are set aside.
2.1 Decoder-Only Dominance and Continuous Optimization
Since the GPT series validated the feasibility of scaling and in-context learning, the Decoder-only architecture has become the de facto standard for modern large models. Its core logic is to predict the next token autoregressively and unify all tasks—dialogue, translation, code generation, and others—as sequence generation problems.
Within the pure Decoder-only framework, different models emphasize different optimizations. Some focus on attention efficiency improvements, such as grouped-query attention or sliding-window attention, which preserve long-text modeling capability while significantly reducing the memory footprint of the KV cache. Others focus on activation functions and normalization layers, for example by adopting smoother activation functions to stabilize training in deep networks, or by adjusting LayerNorm variants to reduce early-stage gradient oscillations.
One notable trend is the large-scale return of Mixture-of-Experts models. With total parameter count held constant, a sparse activation mechanism splits the model into multiple expert subnetworks and activates only a subset during each inference pass. This preserves model capacity expansion while controlling actual computation. Among current third-generation open-source models, the share of models using MoE architectures has risen significantly, and expert routing strategies have become more refined—from coarse layer-level routing to token-level dynamic routing, with even domain-specific expert configurations beginning to appear.
2.2 Architectural Breakthroughs for Long Context
Long-text processing capability has become an important benchmark for a new generation of large models. From initial context lengths of 2K and 4K to current mainstream lengths of 32K and 128K, and on to attempts at the 1M level, this is not merely a matter of stacking more GPU memory. It requires systematic architectural changes to position encoding, attention computation, and cache management.
In position encoding, the industry has reached a broad consensus: absolute position encoding is gradually being replaced by relative position encoding or rotary position encoding. The latter injects relative position information directly into attention score computation through rotation matrices. It can naturally extrapolate to sequences beyond the training length and does not require additional parameters. Some frontier models also attempt length interpolation during fine-tuning, allowing the model to learn generalization for ultra-long sequences from training data of limited length.
However, longer context is not simply “better.” It hides the classic “Lost in the Middle” problem: when processing very long text, a model often perceives key information in the middle more weakly than information at the beginning and end. To address this, some architectures introduce explicit retrieval or focusing mechanisms, such as adding a lightweight information-localization network before the attention layer, or splitting long text into chunks and building a block-level index. This allows the model to actively “consult” relevant segments during generation instead of being forced to replay attention over every historical token.
2.3 Inference-Oriented Architecture Pruning and Simplification
As model deployment scales, inference cost gradually exceeds training cost and becomes the largest expense in real business operations. As a result, the industry has begun to pursue maximum output per parameter from the architecture level, producing a series of inference-optimized designs.
KV cache compression is one of the most active directions. During autoregressive generation, the key-value cache grows linearly with sequence length and becomes a memory bottleneck in long-dialogue scenarios. Mainstream approaches include sparse pruning based on importance scores, which evicts unimportant historical tokens from the cache, and cross-layer KV sharing, which allows adjacent layers to reuse the same cache computation results and thereby compress cache usage to a fraction of its original size.
At the same time, speculative sampling and parallel decoding have received more native support at the architecture level. Traditional autoregressive decoding generates tokens one at a time in serial, with speed limited by memory bandwidth. Speculative sampling uses a small draft model to generate candidate token sequences first, and then a large model verifies and corrects them in a single pass. Some new architectures are beginning to integrate this internal draft-verify dual-channel design, increasing inference throughput several-fold without sacrificing generation quality.
3. Two Main Technical Paths for Building Multimodal Models
If the preceding discussion describes the evolution of unimodal text models, multimodal model construction is the area of architecture design with the greatest challenge and innovative energy today. In the current open-source ecosystem, there are two main methodological paths for building multimodal large models. Each has its own strengths and weaknesses and suits different business scenarios and levels of technical readiness.
3.1 Method A: Unified Embedding Decoder Architecture
The core idea of this path is to “translate” non-text modalities such as images and audio at the model entrance into vector representations with the same dimensionality as text tokens, then feed them into the same decoder for unified sequence modeling.
The architecture flow can be broken down as follows:
- A vision encoder, such as ViT, splits an input image into a fixed-size patch grid and extracts a feature vector for each patch.
- A projection module maps visual feature vectors into the high-dimensional semantic space where text embeddings reside.
- After mapping, these visual embeddings are concatenated with text token embeddings along the sequence dimension to form the decoder’s input context.
The design of the projection module deserves closer discussion. Early multimodal models used a simple linear layer for mapping. The approach is intuitive but has limited expressive capacity: there is a large semantic gap between local image details and abstract text symbols, and a linear transformation alone cannot easily bridge it. More advanced implementations therefore introduce learnable query mechanisms such as Q-Former or Perceiver Resampler. The idea is to predefine a fixed number of learnable query vectors and use cross-attention layers to repeatedly “query” visual features. This compresses a variable-length visual feature sequence into a fixed-length representation with greater semantic distillation. The compressed vectors are then sent into the decoder, which not only shortens the input sequence to save computation but also helps the model focus on the most important semantic components of the visual information.
The advantage of this architecture is that its engineering implementation is relatively simple. The encoder and decoder are decoupled through an alignment layer and can be pretrained and fine-tuned independently. The decoder can reuse an existing strong text model; multimodal data pairs need to be constructed mainly for targeted training of the projection layer and vision encoder. Its disadvantages have also become apparent. First, visual information is compressed once at the entrance and merged into the text stream; if compression is too aggressive, fine-grained spatial or color features can be lost. Second, when handling longer visual inputs, such as multiple frames in a video, the unified embedding approach significantly lengthens the computation graph and introduces inference latency challenges.
3.2 Method B: Cross-Modal Attention Architecture
Unlike Method A, in which a single decoder runs throughout, Method B introduces an independent vision encoder branch that runs in parallel with the text decoder. The two exchange information through dedicated cross-modal attention layers.
In this architecture, the vision encoder not only extracts raw image features but also maintains an independent representation of visual features. In each decoder layer, text-token attention computation is split into two channels: one is self-attention based on the text sequence itself, and the other is cross-attention between text tokens and visual features. This means that when generating each text token, the model follows the internal logic and grammatical rules of language while also being able to “look back” at specific regions of the image in real time and dynamically extract the visual information it needs.
This design brings significant advantages. Because visual information is not compressed and merged too early, the model can repeatedly reference image details at multiple layers. This is especially important for tasks such as visual question answering and image captioning, which need to cite precise visual evidence. In addition, the method naturally supports multi-image input and interleaved image-text input: features from different images are encoded independently in the visual branch and are “called” only as needed during cross-attention. This is more flexible than concatenating multiple images into an ultra-long input sequence as in Method A.
However, Method B is also much more complex than the former. Adding cross-attention layers incurs considerable computation, especially as model depth increases, causing training and inference costs to grow multiplicatively. At the same time, the vision encoder and text decoder need joint training to achieve the best cross-modal synergy, and joint training places higher demands on data mixture, learning-rate strategy, and parallel partitioning. In engineering implementation, load imbalance between the visual branch and text branch in distributed training is another difficult issue. The vision encoder usually has fewer parameters and less computation than the text decoder, and reasonable device resource allocation is needed to avoid idle waiting.
3.3 Comparison and Selection Guide
| Dimension | Method A: Unified Embedding Decoder | Method B: Cross-Modal Attention Architecture |
|---|---|---|
| Degree of visual information retention | Compressed at the entrance; risk of information loss | Multi-layer cross-reference; richer retention of details |
| Computational efficiency | Shorter input sequence; relatively lower inference cost | More cross-attention layers; higher training and inference cost |
| Implementation complexity | Relatively simple; can reuse pretrained decoder weights | Larger architectural changes; difficult joint training |
| Multi-image and video extension | Requires truncation or compression of visual tokens | Naturally supports parallel multi-feature branches |
| Typical use cases | Image-text retrieval, visual classification, lightweight image-text understanding | Fine-grained visual question answering, image-text generation, multimodal dialogue |
4. Engineering Details of Tokenizers and Projection Modules
Regardless of which multimodal architecture is used, tokenizer design and the implementation of the projection module are key engineering details that affect final model quality. This section discusses them in more detail.
4.1 Text Tokenizer: Balancing Compression Rate and Semantic Integrity
On the text side, byte pair encoding and its variants remain the mainstream approach. One detail worth noting is the balance between vocabulary size and text compression rate. If the vocabulary is too small, each token carries too little semantic information, sequence length becomes longer, and computational cost rises. If the vocabulary is too large, it not only increases the parameter scale of the embedding matrix but also introduces sparse training data—many long-tail tokens with extremely low frequency are difficult to train sufficiently. In industry practice, vocabulary size is usually kept in the range of 32K to 100K, supplemented by BPE merge strategies to dynamically expand the vocabulary with new words.
More advanced exploration also includes multi-granularity tokenizer design: using BPE at the lexical level and introducing phrase-level token segmentation at the semantic level. The purpose is to give the model more flexible “granularity control” during generation—it can output precise words and also apply common sentence templates as whole segments, thereby reducing decoding pressure during generation while maintaining semantic coherence.
4.2 Image Projection Module: From Linear Mapping to Learnable Queries
Returning to the projection module: in Method A, the specific implementation of the projection module directly determines how faithfully visual semantics are “transported” into text space.
- Linear projection layer: The simplest and most basic approach. It only applies a linear transformation to map visual feature dimensions to the same dimensions as text embeddings. Its advantages are very low additional parameter overhead and fast inference. Its disadvantage is that a linear transformation cannot capture nonlinear relationships, so its expressive power is insufficient for complex visual concepts such as spatial relationships between objects or implicit semantics in changes of light and shadow.
- Multilayer perceptron mapping: Adds one or two hidden layers in between and introduces nonlinear activation functions. This has become the standard configuration for current entry-level multimodal models and can significantly improve alignment quality with almost no effect on efficiency.
- Learnable query pooling: The Q-Former idea mentioned earlier. Through cross-attention between a fixed number of query vectors and the visual feature sequence, it compresses variable-length input into fixed-length representation. This not only reduces the subsequent sequence length but, more importantly, allows the query vectors to dynamically extract attention weights according to different task requirements—whether global scene understanding or local object recognition—giving the method a degree of adaptivity.
- Full attention layer-wise fusion: Some new architectures no longer set up a separate projection module. Instead, visual features participate in attention computation at every layer of the decoder, which has in fact crossed into the boundary of Method B. This “projection-free” approach achieves the best alignment effect, but it also has the highest computational complexity.
4.3 Position Encoding and Visual Order Information
After a visual patch sequence is projected into text space, the relative position information of visual tokens is often overlooked. Text tokens naturally carry word order, but if image patches do not have position encoding added, the model cannot perceive the spatial adjacency relationships between patches—which is critical for understanding object shape, scale, and boundaries. Therefore, adding visual position encoding before or after the projection module is essential. Common schemes include continuing to use two-dimensional sinusoidal position encoding, or directly reusing a set of learnable position embeddings that use the same functional form as text position encoding but have independent parameters.
5. Training Methodology for Multimodal Models
Once the architecture is selected, the training strategy is the other half that determines the upper limit of model capability. Multimodal model training usually follows a staged, progressive paradigm.
5.1 Stage 1: Modality Alignment Pretraining
With the text decoder parameters frozen, only the vision encoder and projection module are trained. The composition of training data is somewhat special: complete instruction pairs are not required. Instead, large numbers of “image-caption text” pairs are used. At this stage, the model has only one objective: to map visual features to positions in semantic space that are close to the corresponding text descriptions. The industry often uses contrastive learning losses or simple language modeling losses to drive this alignment process.
5.2 Stage 2: Cross-Modal Joint Fine-Tuning
Some or all decoder layers are unlocked, and instruction fine-tuning data containing images and interleaved text input is used to jointly optimize the entire network. This stage is the “skill acquisition” phase. The model needs to learn how to reason based on image content and how to organize multimodal knowledge to answer questions. Data mixture is critical at this stage: the proportion of pure text data should remain between 40% and 50% to prevent “language capability degradation” after training on large numbers of multimodal samples.
5.3 Stage 3: Human Feedback Alignment
Through reinforcement learning or direct preference optimization, the model’s outputs are aligned more closely with human preferences for image-text answers. Collecting preference data in multimodal scenarios is more difficult than in pure text scenarios because judgment criteria must consider not only language fluency but also the accuracy of visual information references and the reasonableness of temporal order. This requires building more detailed scoring dimensions and stricter annotation specifications.
6. Key Engineering Considerations and Trend Assessment
In real production environments, architecture choice is not a matter of pursuing performance metrics alone. The following engineering factors also play a decisive role.
6.1 Constraints from Deployment Cost and Inference Latency
Method A, the unified embedding approach, usually requires only one forward pass to complete image-text understanding, whereas Method B, the cross-modal attention approach, needs to interact with visual features during every decoder layer, making computation significantly more expensive. For online scenarios requiring high concurrency and low latency, Method A and its variants are often the safer choice. For offline analysis, expert-domain decision support, and other latency-insensitive scenarios, it is possible to further exploit Method B’s advantages in detailed understanding.
6.2 Hardware Adaptation and Distributed Parallel Strategies
The introduction of cross-modal architectures makes tensor parallelism and pipeline parallelism partitioning more complex in distributed training. The vision encoder and text decoder have different computational time profiles; simply dividing layers across devices can easily cause severe load imbalance. In practice, a heterogeneous parallelism strategy is often used: the smaller visual branch is placed entirely on a subset of devices, pipeline bubble filling techniques are added, and the gaps in visual branch computation are used to interleave text-branch micro-batches.
6.3 Trends in Context Extension and Multimodal Fusion
Looking ahead, open-source large-model architecture evolution is converging along two linked directions. The first is “native multimodal” unified representation learning—no longer relying on an independent vision encoder, but instead mixing and encoding all modalities, including text, images, and audio, under a unified tokenization framework, completely eliminating “translation error” between modalities. The second is architectural support for inference-time computation—the model no longer generates a complete answer in one pass, but has internal chain-of-thought planning and self-reflection mechanisms. It can call visual information multiple times during inference for verification and correction. This trend will redefine the interaction frequency and layer distribution of cross-modal attention in multimodal architectures.
7. Conclusion
The architectural evolution of open-source large models is far from reaching its endpoint. Whether it is the simplicity and efficiency of unified embedding decoders or the depth and precision of cross-modal attention architectures, each approach has found a reasonable ecological niche under specific scenarios and resource constraints. For developers, the most important thing is not to blindly follow whichever architecture label is considered “most advanced,” but to clearly understand the costs and benefits behind each technical route and make rational choices based on business scenarios, data resources, and hardware resources. Architecture determines the upper bound; data and engineering determine the lower bound. In the rapidly iterating open-source wave, pragmatic teams are the ones that can progress steadily and go far.
