Inference Architecture Evolution in a New-Generation Lightweight Multimodal Model
This article examines the first natively multimodal lightweight release in a leading large model team's flagship series. It covers the extremely sparse MoE design (320B total and about 18B active parameters), the 3:1 mix of linear and sparse attention, the IndexPool compression technique, manifold-constrained hyper-connections, native support for image and video input with 1M context, and what the cost structure means for developers and enterprises.
1. Background: An Architecture-Level Update Worth Watching
Recently, a leading large model team released the first natively multimodal lightweight version of its flagship series. Rather than a routine version update, the release works as a public test of an architecture designed for the next generation of models. The model uses an extremely sparse MoE structure with 320B total parameters and about 18B active parameters. Combined with a newly trained base model, it performs close to the top tier of closed-source flagship models on public benchmarks, while its inference cost is roughly one-tenth that of the previous generation.
For developers, lightweight releases of this kind often matter more for engineering than parameter races among flagship models. They answer a practical question: under limited compute and budget, how can architectural innovation, rather than simply adding parameters, deliver better performance?
2. Architecture Breakdown: Why a Shallow Model Can Be Strong
1. Extremely Sparse MoE: 320B Total Parameters, 18B Activated per Token
The model has 45 layers and 288 routed experts. Each token activates only 8 experts, and the first three layers use a dense structure. The reasoning behind this design is as follows:
- Decoupling capacity from computation: Total parameters set the upper limit of knowledge capacity, and active parameters determine the compute cost of each inference step. A low activation ratio (about 5.6%) keeps a large knowledge base while holding FLOPs at roughly the level of a mid-sized dense model.
- Why the first three layers are dense: Lower-layer features tend to be highly general. Shared computation there helps stabilize training and prevents the router from making low-quality assignments in shallow layers.
2. Hybrid Attention: A 3:1 Interleaving of Linear and Sparse Attention
This is the core cost optimization of the design. About 34 of the 45 layers use linear attention, and 11 layers use sparse attention. The two types alternate periodically at roughly a 3:1 ratio:
Layer layout (illustrative):[Linear] [Linear] [Linear] [Sparse] [Linear] [Linear] [Linear] [Sparse] ...
The two attention types have distinct roles:
- Linear attention handles continuous context: For most tokens, semantic understanding depends on nearby tokens. The compute cost of linear attention scales linearly with sequence length, so long-context processing is far cheaper than with standard quadratic attention.
- Sparse attention handles precise retrieval over long distances: Inserting a sparse attention layer every few layers establishes exact connections for long-range dependencies. This compensates for the weaker exact recall of linear attention.
The core idea is to handle most computation cheaply and to use the costlier but sparse method only where it is necessary. Compared with full dense attention, attention compute drops to about one-third, and KV cache memory shrinks to about one-quarter. For long-context inference, this is a substantial improvement in memory use and throughput.
3. IndexPool: Further Compression for Sparse Attention
The sparse attention layers use an index pool mechanism. Index keys are compressed at about a 4:1 ratio before they take part in retrieval. Before the model decides which historical information to attend to, the candidate set is condensed, which reduces the number of keys involved in exact matching. This adds representation-level compression on top of algorithmic sparsity, a typical approach to reducing cost end to end.
4. mHC: A Key Complement for Shallow Architectures
The model also introduces Manifold-Constrained Hyper-Connections (mHC). Instead of simple residual addition from layer to layer, mHC mixes residual information in a more complex way. This is easy to overlook, but it matters especially for a shallow model with only 45 layers:
- With fewer layers, there are fewer paths for information to flow across layers, which limits the model’s ability to assemble deep semantics.
- Enhanced residual mixing provides richer side paths for information flow without adding layers. This gives the shallow network representational capacity closer to that of a deeper network.
Overall, although each token activates only about 18B parameters, the base model’s capability approaches that of the previous generation model with about 40B active parameters. It also performs better on coding tasks.
3. Native Multimodality and Ultra-Long Context
The model was trained from scratch on about 30T of multimodal pretraining data. It natively supports 1M context along with image and video input. The word “native” deserves emphasis:
- Compared with bolted-on multimodality: Many models gain multimodal ability by attaching a vision encoder to a pure language model and fine-tuning it. This limits alignment quality across modalities.
- The advantage of a native design: Because text, images, and video share one representation space from pretraining onward, long context and multimodality are unified at the architecture level. This lays groundwork for complex agent tasks such as long video understanding and cross-modal code generation.
Practical use of 1M context still depends on how well the attention mechanism recalls information across very long sequences. This explains why the architecture keeps sparse attention layers. Linear attention handles fast reading, and sparse attention handles accurate retrieval.
4. Cost Structure: What the Flash Positioning Really Means
Public evaluations show that the model achieves a competitive score on the composite intelligence index, while the cost per individual task stays very low, at a few cents. This result does not come from common cost-cutting methods such as quantization or distillation. It comes from redesigning the inference structure itself.
| Optimization | Effect |
|---|---|
| Extremely sparse MoE | Reduces compute per token |
| Mainly linear attention | Makes long-context compute scale linearly |
| Periodic sparse attention | Preserves precise retrieval ability |
| IndexPool | Reduces KV cache by about 4x |
| mHC | Maintains representation capacity in a shallow structure |
This shows that low cost and low capability are not inherently linked. When architectural efficiency is high enough, cost can fall while performance rises. For enterprises that deploy agents at scale or process code in bulk, this shift in the price-performance curve matters more than leaderboard scores.
5. Takeaways for Developers and Enterprises
- Look at the ratio of active to total parameters, not total parameters alone: When evaluating MoE models, the number of activated parameters per token, the attention structure, and KV cache overhead are the key indicators of inference cost.
- Hybrid attention is becoming a mainstream direction for long context: Pure linear attention is weak at exact recall, and pure dense attention is too costly. The interleaved linear and sparse layout offers a reusable design pattern.
- Validate on agent-oriented benchmarks first: Coding and agent benchmarks, such as repository-level tasks and tool-calling chains, reflect a model’s value in real development workflows better than general Q&A. When selecting a model, teams should build evaluation sets around their own tasks.
- Managed services add more value: For most teams, self-hosting a model of this kind is costly to operate. Calling a mature model serving platform, such as Baidu AI Cloud Qianfan, lets teams focus on business logic and agent orchestration.
6. Summary
The real significance of this release is that it validates a complete set of next-generation architecture choices: extremely sparse MoE, a shallow Transformer, hybrid linear and sparse attention, index compression, enhanced residual mixing, and native multimodality. It shows that systematic architectural innovation can approach flagship capability at about one-tenth the cost. This efficiency-first direction will likely become a standard pattern for future lightweight models. For developers, now is a good time to reassess inference cost models and adjust technology selection strategies.
