Open-Source LLM Architecture Explained: MoE and Three Key Innovations

    This article breaks down a new-generation open-source large language model built around an efficiency-first design—large capacity, low activation, and long context. It explains how the MoE (Mixture of Experts) architecture separates total parameter capacity from per-request compute cost, reviews how the model performs against leading closed-source models on public benchmarks, and dissects three key architectural changes: MLA combined with Muon Split, multi-token prediction, and DSA sparse attention. It closes with practical takeaways for evaluating and deploying such models in production.

    Introduction: An Architecture Evolution Centered on Efficiency

    As open-source LLMs iterate rapidly, simply stacking more parameters is no longer the only goal. Taking a recently released open-source model as its subject, this article uses its publicly available technical details to unpack a design philosophy built around “large capacity, low activation, long context.” Note that any benchmark scores and rankings mentioned below come from third-party public composite intelligence indices and open-source model leaderboards; this article borrows only their overall trends and does not endorse any specific product.

    1. Model Fundamentals: Separating Parameter Scale from Activation Cost

    The first key to understanding this class of models is the MoE (Mixture of Experts) architecture.

    In a traditional dense model, every token passes through all parameters, so parameter count and compute are tightly coupled. MoE instead splits the feed-forward layers into many parallel “experts” and uses a routing/gating network to dynamically select a small subset of experts for each token (top-k routing). The core benefit: total parameter capacity can be very large, but only a small fraction of parameters actually take part in each inference pass.

    For the model analyzed here, the key figures are:

    • Total parameters: 744B
    • Activated parameters per inference: 40B
    • Number of experts: 256
    • Network layers: 80

    Compared with the previous generation (355B total, 32B activated), a clear signal emerges: total parameters nearly doubled, yet activated parameters grew by only about 25%. This shows the designers shifted their focus from “scaling size” to “increasing effective capacity within a controlled compute budget.” Pretraining data also grew from 23T tokens in the previous generation to 28.5T tokens (27T pretraining, 1.5T mid-training); the mid-training stage typically handles capability strengthening and domain alignment.

    Takeaway for developers: When evaluating an MoE model, don’t look at total parameters alone—examine activated parameters and the expert routing strategy. Activated parameters are the key metric that determines the compute cost of a single request.

    2. Benchmark Performance: Open-Source Models Closing in on Top Closed-Source Models

    On a third-party composite intelligence index, the model ranks among the leading open-source models and takes first place among open-source entries on several text and coding leaderboards, placing it in the same tier as top closed-source models.

    It’s worth being objective here: composite indices and leaderboard rankings reflect “human preference” and “overall task performance,” which are not fully equivalent to real-world performance in a specific business scenario. When selecting a model for production, you should combine your own evaluation sets, latency budgets, and deployment costs rather than relying solely on public leaderboards.

    3. Three Architectural Innovations, Broken Down

    Unlike the previous generation, this one has three key architectural changes: MLA + Muon Split, multi-token prediction, and DSA sparse attention. Together they target the same goal—balancing memory usage, training stability, and inference speed in long-context scenarios.

    3.1 MLA + Muon Split: Compressing the Attention Cache and Adapting the Optimizer

    MLA (Multi-head Latent Attention) shares the same roots as mainstream efficient-attention approaches: instead of fully caching K and V for every attention head, it compresses KV into a low-dimensional “latent” vector, so inference only needs to cache this compressed representation. The direct benefit is a significant reduction in KV-cache memory usage during long-text processing, enabling longer contexts and higher concurrency.

    However, the team ran into a practical problem during training: when pairing the Muon optimizer (an optimizer that updates weights via matrix orthogonalization, as opposed to the element-wise Adam) with MLA, convergence actually lagged behind the simpler GQA-8 (Grouped Query Attention) approach.

    Their solution was Muon Split: the original approach orthogonalized the entire projection matrix as a single block, while Muon Split orthogonalizes each attention head separately. This seemingly small change aligned the optimizer’s update granularity with MLA’s per-head structure and restored training performance.

    1. # Pseudocode: whole-matrix orthogonalization vs. per-head orthogonalization
    2. # Original approach: orthogonalize the full projection matrix at once
    3. W_proj = orthogonalize(W_proj_full) # shape: [d_model, d_model]
    4. # Muon Split: orthogonalize each attention head separately
    5. for head in range(num_heads):
    6. W_head = orthogonalize(W_proj[:, head_slice]) # each head processed independently

    Key point: Adapting the optimizer to the attention structure is often more important than simply tuning hyperparameters. When a custom attention structure causes unstable training, first check whether the optimizer’s update granularity is aligned with the structure.

    3.2 Multi-Token Prediction: Denser Training Signals

    A traditional autoregressive model predicts only the “next token” at each step. Multi-token prediction (MTP) adds several auxiliary prediction heads beyond the main one, letting the model predict the 2nd, 3rd, and further future tokens during training.

    Its benefits span two levels:

    1. Training: Each position provides a denser supervision signal, forcing the model to learn longer-range dependencies and improving representation quality;
    2. Inference: The trained multi-prediction heads can serve as a “draft model,” working with speculative decoding to verify multiple tokens at once and accelerate generation while keeping the output distribution unchanged.

    Key point: Multi-token prediction isn’t simply “predicting a few more tokens”—it improves representations through denser supervision and naturally leaves an interface for inference acceleration.

    3.3 DSA Sparse Attention: Breaking the Quadratic Complexity

    Full attention’s compute complexity grows quadratically with sequence length, making it the main bottleneck for long context. DSA (Dynamic Sparse Attention) works by letting each query token attend only to a selected subset of key tokens rather than all of them.

    Sparse attention typically uses a learnable scoring/selection mechanism to dynamically pick the few key-value pairs most important to the current token, significantly reducing complexity from O(n²). It is complementary to MLA: MLA compresses the “volume of the cache,” while DSA reduces the “range that needs to be computed”; stacking the two lets long-context memory and compute benefit simultaneously.

    4. Takeaways for Engineering Deployment

    Overall, the transferable lessons from this architecture include:

    • Decoupling capacity from cost: Use MoE to separate parameter capacity from per-request compute, and evaluate inference cost by activated parameters;
    • Build the attention structure as a system: MLA compresses the cache, sparse attention cuts computation, and multi-token prediction densifies supervision—only together do they support long context;
    • Adapt the optimizer to the structure: When a custom structure causes unstable training, first check whether the optimizer’s update granularity (e.g., per-head orthogonalization) is aligned;
    • Treat leaderboards as reference only: Composite intelligence indices reflect trends; production selection should combine your own evaluations, latency, and cost.

    If you’re doing secondary development or model selection based on open-source LLMs, focus on verifying: whether activated parameters fit your compute budget, whether KV cache is controllable under long context, and whether inference acceleration such as speculative decoding is supported. These dimensions often determine real-world deployment outcomes more than a single leaderboard score.