How Leading Open-Source LLMs Work: MoE Architecture, Attention Mechanisms, and Training Optimization Strategies
This article breaks down the shared technical foundation behind today's leading open-source large language models — the combination of Mixture-of-Experts (MoE), Multi-head Latent Attention (MLA), and multi-stage training. It examines key MoE Transformer parameters such as the split between total and active parameters, expert count and sparsity, and attention-head tradeoffs, then looks at differentiated training strategies including optimizer choices like Muon and the growing importance of token efficiency. Finally, it offers practical guidance for developers on model selection and deployment, emphasizing active parameters over total size, inference-framework compatibility, parameter-efficient fine-tuning, and technical reports over leaderboard scores.
Why the Top Open-Source Models Look So Similar
If you’ve been following the open-source LLM ecosystem over the past year, you’ll notice something interesting: many newly released trillion-scale open models explicitly state in their technical reports that their architecture is “highly similar” to a certain benchmark generation. This is not a coincidence, but an inevitable outcome of technical evolution.
Once a given open model proved the effectiveness of the “MoE + MLA + multi-stage training” combination with extremely low inference cost and strong performance, later entrants tend to build incremental innovations on the same technical foundation — adding more experts, adjusting the number of attention heads, swapping optimizers, expanding the training corpus. As a result, at the macro-architecture level these models genuinely look like “siblings from the same source”; but when you break down each design choice, you can see each team’s distinctive engineering tradeoffs.
This article uses a typical trillion-parameter MoE model as its subject, breaks down the core components of this technical foundation, and discusses the differentiated design approaches that different teams take.
Core Architecture: Key Parameters of the MoE Transformer
Separating Total Parameters from Active Parameters
The most common parameter structure in modern MoE models is “a huge total parameter count, with only a small fraction activated per forward pass.” Taking a typical trillion-scale model as an example:
- Total parameters around 1T, coming from hundreds of expert networks (Experts);
- Only about 30B active parameters, meaning each token is routed to just a few experts.
The essence of this design is “trading storage for compute”: the model’s knowledge capacity grows linearly with total parameters, but inference-time compute cost depends only on the active parameters. For enterprise users, this means achieving knowledge reserves far beyond a 30B model at inference costs close to a 30B dense model.
Number of Experts and Sparsity
Scaling Law analyses show that continuously increasing sparsity yields significant performance improvements. Based on this, the new generation of models generally expands the number of experts from 256 to 384 or more. The more experts there are, the larger the routing space, and the finer the model’s partitioning of specialized knowledge.
But this introduces an engineering tradeoff: more experts mean greater overhead for routing computation and memory management, especially higher demands on memory fragmentation and batch scheduling during inference. This is why different teams make different choices on the number of experts — there is no absolute optimum, only a balance point that matches their own inference infrastructure.
The Attention Mechanism Tradeoff
Another typical tradeoff is the number of attention heads. Some models reduce attention heads from 128 to 64 to lower inference-time compute overhead; combined with Multi-head Latent Attention (MLA) to compress the KV cache, the memory footprint of long-context inference drops substantially.
For teams that need to deploy long-context services in production, KV cache size directly determines how many concurrent requests a single card can handle. By projecting Key and Value into a low-dimensional latent space, MLA compresses cache size by an order of magnitude — this is the core reason the mechanism is now widely adopted across open models.
Schematically, the forward logic of an MoE layer can be simplified as:
# Simplified MoE layer pseudocodegate_logits = router(x) # Router scoringtopk_idx, topk_weight = topk(gate_logits, k=8)output = sum(w * expert_i(x) for i, w in zip(topk_idx, topk_weight))
Differentiated Training Strategies: Optimizers and Data Efficiency
Optimizer Choice
Traditional large-model training almost defaults to AdamW-class optimizers, but their second-order momentum caches consume a large amount of memory. Muon-class optimizers that have emerged in recent years, through orthogonalizing parameter updates, have shown higher sample efficiency and lower memory usage in trillion-parameter-scale training, and have been adopted by some teams as the default choice for pretraining.
This points to a trend: optimizers are once again becoming an active area of innovation in large-model training, rather than a “default infrastructure” that is taken for granted.
Token Efficiency as a Key Variable
As the availability of high-quality human data becomes increasingly limited, “token efficiency” is replacing “data scale” as the key scaling factor. Teams generally adopt two types of strategies:
- On the data side: building large-scale high-quality corpora (on the order of 15T tokens), and improving the learning value per token through cleaning, deduplication, and mixture optimization;
- On the training side: using multi-stage curriculum learning — first establishing foundational capabilities with general corpora, then gradually introducing code, math, and instruction data to shape capabilities.
Practical Implications for Developers
Understanding these technical commonalities, developers and enterprises can draw a few pragmatic conclusions when selecting and deploying:
- Don’t over-index on parameter count: active parameters are what determine inference cost, so when selecting a model, focus on the active scale and context capability;
- Pay attention to inference framework compatibility: designs like MLA and MoE routing impose specific requirements on the inference engine, so confirm your framework’s support before deployment;
- Fine-tuning costs stay manageable: with mainstream parameter-efficient fine-tuning methods, enterprises can build on a general open-source model and use cloud infrastructure such as object storage, container platforms, and GPU resource pools to complete domain customization, without pretraining from scratch;
- Focus on technical reports rather than leaderboards: architectural parameters, training recipes, and sparsity design reflect a model’s true capability boundaries better than aggregate benchmark scores.
Summary
Today’s leading open-source LLMs are highly convergent at the macro-architecture level — mixture-of-experts, low-rank attention compression, and multi-stage training form a common technical foundation; the real differences lie in micro engineering decisions such as expert granularity, attention-head tradeoffs, optimizer choice, and data strategy. For developers, understanding these common architectures and differentiated tradeoffs is far more valuable in the long run than chasing any specific model. Once you master this technical analysis framework, you can quickly assess a model’s design intent and use cases no matter how models iterate in the future.
