Modern LLM Architecture Explained: MoE and Sparse Attention in Practice

    A breakdown of the engineering decisions behind the latest generation of open-source large language models: how Mixture-of-Experts (MoE) decouples parameter scale from compute, how latent and sparse attention reduce long-context costs, how optimizer redesign and multi-token prediction improve training efficiency, and how fully asynchronous RL frameworks maximize post-training throughput. Rather than recapping one model's benchmark scores, this article abstracts these choices into a transferable methodology, explaining why each design exists, what it costs, and how to weigh the trade-offs.

    1. Introduction: From “Bigger Is Better” to Refined Design

    Over the past two years, the headline of the large-model race was often “the more parameters, the better.” But as the field entered a new round of iteration, attention has clearly shifted toward a different question: how to keep raising a model’s effective intelligence without letting inference cost balloon without limit. The technical reports of recent open-source models get read page by page by the developer community precisely because they explain the key engineering decisions along this path—using a Mixture-of-Experts architecture to decouple parameter scale from compute, using latent and sparse attention to compress the overhead of long sequences, using optimizer redesign and multi-token prediction to improve training efficiency, and using a fully asynchronous reinforcement learning framework to maximize throughput in the post-training stage.

    This article does not rehash the results of any specific product. Instead, it abstracts these practices into transferable technical methods, explaining block by block “why it is designed this way, what the cost is, and how to make the trade-off.”

    2. The Base Model: Parameters and Activation Under MoE

    2.1 Decoupling Total Parameters from Activated Parameters

    Today’s mainstream large models generally adopt a MoE (Mixture of Experts) architecture. Its core idea is that the model contains a large number of “expert” sub-networks, but for each input token, a router selects only a few of them to participate in the computation.

    Take one representative configuration as an example: about 744 billion total parameters, with only about 40 billion activated per forward pass, 256 experts, and 80 layers. Compared with the previous generation (about 355 billion total, about 32 billion activated), this generation nearly doubles the “knowledge capacity” (total parameters) while only modestly increasing the “compute per pass” (activated parameters).

    The value of this design lies in decoupling cost from capability:

    • Total parameters determine how much the model can “remember”—the ceiling of its capability;
    • Activated parameters determine the actual compute cost of each inference—the floor of its cost.

    The illustrative code below shows the core routing logic of an MoE layer:

    1. # Illustrative code: routing and forward pass of an MoE layer
    2. def moe_forward(x, router, experts, top_k=2):
    3. # router outputs each token's score for every expert
    4. logits = router(x) # [batch, seq, num_experts]
    5. scores = softmax(logits, dim=-1)
    6. # each token selects only the top_k highest-scoring experts
    7. topk_scores, topk_idx = scores.topk(top_k, dim=-1)
    8. # re-normalize the weights of the selected experts
    9. topk_weights = topk_scores / topk_scores.sum(dim=-1, keepdim=True)
    10. out = zeros_like(x)
    11. for k in range(top_k):
    12. for eid in unique(topk_idx[:, :, k]):
    13. mask = (topk_idx[:, :, k] == eid)
    14. out[mask] += topk_weights[mask, k].unsqueeze(-1) * experts[eid](x[mask])
    15. return out

    Note that MoE is no “free lunch”: routing introduces load imbalance (a few experts get overloaded while most sit idle), which requires a load-balancing loss during training; at the same time, expert parallelism places higher demands on distributed communication and memory partitioning.

    2.2 Number of Experts, Layers, and Data Scale

    Beyond the parameter structure, data scale is also a key variable. This generation raises the pre-training corpus from the previous generation’s roughly 23 trillion tokens to about 28.5 trillion tokens (about 27 trillion for pre-training and about 1.5 trillion for mid-training). This reflects two trends:

    1. Data quality matters more than quantity: in the mid-training stage, using a smaller, high-signal corpus for annealed supplementation often improves reasoning and coding ability more effectively than simply piling on more data;
    2. The mixture defines the style: the ratio of general text, code, math, and multilingual data directly determines the model’s relative strengths across different benchmark scenarios.

    2.3 How to Measure a Model’s Real Capability

    You cannot evaluate a model by a single leaderboard alone. The industry typically relies on multi-dimensional cross-validation:

    • composite intelligence index benchmarks (normalizing multiple discipline benchmarks and then weighting them);
    • public arenas (Elo-style rankings based on human blind tests, split into tracks such as text and code);
    • specific capability benchmarks (math, code, long context, instruction following, and so on).

    A mature takeaway is this: the leading open-source models have entered the same tier as top closed-source models in overall capability, with the remaining gap showing up mainly in a few long-tail tasks and ecosystem maturity—not in a “usable or not” dividing line.

    3. Three Core Architectural Changes

    Compared with the previous generation, three changes in this generation’s architecture are worth a closer look.

    3.1 Latent Attention Paired with Optimizer Redesign

    Multi-head Latent Attention (MLA) shares its lineage with another mainstream efficient-attention approach. The core idea is to avoid caching the full Key/Value directly, but instead compress them into a low-rank latent space; during inference, only this low-dimensional latent representation is cached, and it is projected back up when needed. The direct payoff is a large reduction in the memory footprint of the KV cache and a significant improvement in decoding throughput in long-text scenarios.

    1. # Illustrative code: latent attention (low-rank KV compression)
    2. def latent_attention(x, W_dkv, W_uk, W_uv, W_q, W_o):
    3. c_kv = x @ W_dkv # compress into a low-rank latent representation—only c_kv is cached at inference
    4. k = c_kv @ W_uk # project up to recover K when needed
    5. v = c_kv @ W_uv # project up to recover V when needed
    6. q = x @ W_q
    7. attn = softmax(q @ k.transpose(-1, -2) / scale)
    8. return (attn @ v) @ W_o

    In training practice, however, the team ran into a counterintuitive problem: applying the Muon optimizer (an efficient optimizer based on matrix orthogonalization) directly to MLA actually failed to keep up with simpler structures (such as standard grouped-query attention). The root cause is that Muon orthogonalizes an entire projection matrix as a whole, while MLA’s latent-space compression breaks the symmetry among the attention heads, so treating the matrix as a whole destroys that structure.

    The solution is to orthogonalize per attention head: split the full matrix into a sub-matrix for each head, run Newton–Schulz orthogonalization iterations on each one, and then stitch them back together.

    1. # Illustrative code: orthogonalize per attention head instead of treating the whole matrix
    2. def muon_update_per_head(grad, num_heads):
    3. head_grads = grad.chunk(num_heads, dim=0)
    4. new_grads = [newton_schulz_orthogonalize(g) for g in head_grads]
    5. return cat(new_grads, dim=0)

    The lesson here is that the optimizer and the architecture must be co-designed. Copying someone else’s configuration tends to backfire; you need to understand the implicit assumptions the optimizer makes about matrix structure.

    3.2 Multi-Token Prediction: A Denser Training Signal

    Standard autoregressive training predicts only the next token at each position, so the training signal is relatively sparse. Multi-Token Prediction has the model, on the same backbone, additionally predict the 2nd, 3rd, and further future tokens, with multiple prediction heads sharing the representation.

    1. # Illustrative code: multi-token prediction heads
    2. def multi_token_predict(hidden, embeddings, num_heads):
    3. losses, h = [], hidden
    4. for i in range(num_heads):
    5. h = projector[i](h) # each prediction head has its own projection
    6. logits = h @ embeddings.T # inner product against the vocabulary
    7. losses.append(cross_entropy(logits, future_labels[i]))
    8. return sum(losses) / num_heads

    The benefits are twofold: during training it provides a denser supervision signal and improves sample efficiency; during inference these prediction heads can also be reused as a draft model for speculative decoding, accelerating generation. The cost is higher training memory and scheduling complexity.

    3.3 Sparse Attention: Reallocating Compute for Long Context

    The compute and memory cost of traditional full attention grows quadratically with sequence length, which becomes unsustainable at long context. Sparse attention (roughly, dynamic sparse / block-selective attention) pushes the cost back to near-linear by having each token attend only to a carefully chosen subset.

    The key difficulty is not “sparsity” but how to select: the mechanism must cover genuinely relevant long-range dependencies, keep the cost of selection itself under control, and ensure consistency between training and inference. Well-designed sparse attention can approach the quality of full attention with a significantly smaller training-token budget—this is the technical origin of community discussions about conclusions like “using about 20 billion tokens to match the effect of nearly a trillion tokens of training.” Such conclusions should be treated with caution: they are more of a controlled comparison on a specific benchmark than a universal law, but they are valuable for understanding the trend that “compute can be spent more intelligently.”

    4. Training Framework: From Synchronous RL to Asynchronous RL

    Post-training (especially reinforcement learning aimed at agents) is a focus of this generation. Traditional RL training mostly follows a synchronous paradigm: generation (rollout) and updates strictly alternate, so GPUs idle while waiting during the generation phase, leading to low utilization.

    A fully asynchronous agent RL framework decouples generation from training—the inference cluster continuously produces samples while the training cluster continuously consumes and updates, with the two aligned through parameter versions and an experience buffer. The payoff is a large improvement in hardware utilization and shorter wall-clock training time; the challenge is the stability problems brought by stale samples (off-policy), which require a whole engineering toolkit such as importance-sampling correction and parameter-synchronization strategies.

    Supporting all of this is in-house RL infrastructure: it must simultaneously manage large-scale inference engines, training engines, data pipelines, reward models, and scheduling. This kind of infrastructure typically relies on general cloud capabilities to carry it—for example, using a container platform for elastic scheduling, object storage for persisting samples and checkpoints, a message queue to decouple generation from training, and monitoring, alerting, and logging services for the observability of long-running training. These are general engineering patterns, independent of any specific vendor.

    5. Practical Implications for Developers and Enterprises

    1. Choose based on “activation cost,” not “total parameters”: MoE makes “big capability, small bill” possible, so when evaluating deployment cost you should look at activated parameters and the KV cache strategy, not just the total parameter count.
    2. Validate KV optimizations for long context first: latent attention and sparse attention have a major impact on the memory and throughput of long-context scenarios, so be sure to stress-test them on your own workloads.
    3. Tune architecture and optimizer jointly: directly copying someone else’s training recipe tends to backfire at the optimizer level, especially with custom structures.
    4. Bet on asynchronous RL for post-training: improvements in agent capability increasingly depend on high-quality RL, and an asynchronous framework plus robust infrastructure is what guarantees throughput.
    5. Cross-validate capability evaluation: a single leaderboard is easily misleading, so base decisions on a combination of composite indices, arena rankings, and your own business benchmarks.

    6. Conclusion

    At its core, the technical report of this generation of open-source large models is a manual on “how to spend compute on the blade’s edge with smarter architecture”: MoE decouples scale from cost, latent and sparse attention compress long-sequence overhead, the co-design of optimizer and structure improves training efficiency, and the asynchronous RL framework maximizes post-training throughput. For developers and enterprises, the real takeaway is not a particular benchmark number but this reusable methodology—understanding the payoff and cost of each design is what lets you find your own optimal balance between budget and capability.