Open-Source LLM Architecture Upgrades: A Deep Dive into Key Techniques

    This article analyzes five key architecture techniques revealed in a recent open-source large model technical report: sparse Mixture of Experts, low-rank KV compression attention, per-head orthogonalized optimization, multi-token prediction, and sparse attention. It explains what problems these designs solve, why they are used, and what they mean for engineering and deployment, offering practical guidance for developers who want to understand how limited compute can be turned into deployable model capability.

    Over the past year, open-source large models have rapidly approached the capability boundary of leading closed-source models. Industry evaluations have repeatedly reached the same conclusion: top closed-source models are still regarded as the “smartest,” but the gap between them and the strongest open-source models has narrowed significantly.

    When a technical report for a new generation of open-source large model was released, the community quickly treated it as a new benchmark for the open-source ecosystem. For developers, the more important question than “which one is stronger” is: what architectural techniques did these models use to turn limited compute into deployable capability?

    Based on the technical route disclosed in this report, this article abstracts five key design choices and explains each one: what problem it solves, why it is designed this way, and what it means in engineering.

    1. Sparse Expert Architecture: Separating Total Parameters from Activated Parameters

    This generation continues the sparse Mixture of Experts (MoE) architecture. The core idea is that the model has a large total parameter count, but each token activates only a small subset of experts during inference. This decouples the knowledge capacity brought by parameter scale from the compute cost of a single inference pass.

    According to the publicly disclosed fundamentals, the model has approximately 744B total parameters and activates about 40B parameters per inference pass, with 256 experts and 80 layers. Compared with the previous generation—approximately 355B total parameters and 32B activated parameters—the total parameter count has nearly doubled, while activated parameters have increased only modestly. The pre-training corpus has also expanded from about 23T tokens to about 28.5T tokens, of which about 27T are used for pre-training and about 1.5T for mid-training.

    The value of this design can be understood through the following conceptual logic:

    1. # Conceptual illustration: forward pass of a MoE layer
    2. def moe_forward(x, experts, router, top_k=8):
    3. # Router scores each token and selects the top_k experts
    4. scores = router(x) # [batch*seq, num_experts]
    5. topk_idx, topk_w = topk(scores, k=top_k)
    6. out = 0
    7. for slot in range(top_k):
    8. idx = topk_idx[:, slot]
    9. w = topk_w[:, slot].unsqueeze(-1)
    10. out = out + w * experts[idx](x) # Invoke only selected experts
    11. return out

    The key trade-off is that activated parameters determine compute and latency during inference, while total parameters determine the upper limit of model capacity. Making the total parameter count large while keeping activated parameters constrained is a clear trade-off between knowledge capacity and serving cost. This also helps explain why such models can achieve high scores in capability evaluations while keeping deployment costs manageable.

    2. Low-Rank Latent Attention: Compressing the KV Cache

    Standard multi-head attention faces a major bottleneck during long-sequence inference: the KV cache grows linearly with sequence length, becoming a primary constraint on memory and throughput. The attention mechanism used by this class of models projects keys and values into a low-dimensional latent space, caches only this compressed low-rank representation, and projects it back when needed.

    A conceptual illustration is as follows:

    1. # Conceptual illustration: low-rank latent attention (cache only compressed latent vectors)
    2. c = W_down @ x # Dimensionality reduction: x -> low-dimensional latent vector c (cached during inference)
    3. k = W_up_k @ c # Project latent vector back to Key when needed
    4. v = W_up_v @ c # Project latent vector back to Value when needed

    Replacing the original KV pairs with low-dimensional latent vectors can significantly reduce KV cache usage while largely preserving expressive power. This makes long-context inference more memory-efficient and higher-throughput. It is another careful balance between quality and efficiency.

    3. Orthogonalized Optimizers and Per-Head Separation

    During training, the team introduced an optimizer based on matrix orthogonalization. It constrains weight updates to directions that are closer to orthogonal, making parameter distributions more uniform and training more stable. However, when training with compressed attention, the team found that orthogonalizing the entire projection matrix as a single block did not perform well.

    Instead, they refined the granularity of orthogonalization: rather than treating the whole projection matrix as one unit, they handled each attention head separately.

    1. Whole-matrix orthogonalization: Orthogonalize( W_all ) # All heads are coupled together
    2. Per-head separation: for h in heads: Orthogonalize(W_h) # Each head is constrained independently

    The reason is that multiple attention heads often carry different subspace semantics. Forcing them into the same matrix for uniform orthogonalization can cause interference. After separating per head, each head can remain well-conditioned within its own subspace, producing cleaner training signals and more stable convergence. Although this change appears to be only a granularity adjustment, it is a key detail refined under the combination of compressed attention and orthogonalization.

    4. Multi-Token Prediction: Increasing Training Signal Density

    Traditional autoregressive training predicts only the next token at each step, so the supervision signal is relatively sparse. Multi-Token Prediction allows the model to predict multiple future tokens at each position, effectively increasing the density of supervision signals during training.

    1. # Conceptual illustration: predict n future tokens in a single forward pass
    2. hidden = backbone(x)
    3. for offset in range(1, n + 1):
    4. logits = head_offset[offset](hidden) # Prediction head for step offset
    5. loss += cross_entropy(logits, tokens_shifted(offset))

    This brings two benefits. First, each position provides a stronger learning signal, improving sample efficiency. Second, this forward-looking training method helps the model plan more coherently about subsequent content during inference, which is especially helpful for strongly structured tasks such as code generation.

    5. Sparse Attention: Further Reducing Long-Context Overhead

    In addition to latent compression, sparse attention is another path to reducing long-sequence overhead. The core idea is that each token does not need to attend to all historical tokens. Instead, through learnable or rule-based mechanisms, it interacts only with a subset of key tokens. This reduces attention computation from quadratic growth with sequence length to nearly linear growth.

    Sparse attention can be combined with latent compressed attention: the former reduces “which tokens to attend to,” while the latter reduces “how much to store for each object.” Together, they make ultra-long contexts more feasible in terms of compute and memory.

    Conclusion: Practical Takeaways for Developers

    Taken together, these five points reveal a clear technical thread in this generation of open-source models: under limited compute, capability is pushed close to the level of leading closed-source models through five techniques—expanding capacity with sparse experts, compressing the KV cache, refining the optimizer, increasing training signal density, and sparsifying attention.

    For developers, there are three practical takeaways:

    • When selecting a model, do not look only at total parameters. Activated parameters and inference cost are equally important because they directly determine the unit cost after serving.
    • For long-context scenarios, pay attention to KV cache and attention complexity. Compressed attention and sparse attention are two combinable cost-reduction paths.
    • When reproducing or fine-tuning open-source models, training details such as optimizer granularity and multi-token prediction often matter more than simply increasing model size.

    If enterprises plan to deploy such open-source models in their own environments or on cloud platforms such as Baidu AI Cloud, they typically still rely on general capabilities such as GPU compute, container platforms, object storage, logging services, and monitoring alerts to achieve elastic scaling and stable operations. Only by understanding these architectural trade-offs can compute be truly converted into usable product capabilities.