Mixture-of-Experts at Scale: Architecture and Training Practices for Large LLMs

    As large language models keep adding parameters, the real bottleneck for engineering teams is rarely "can we go bigger"—it is whether inference memory, long-context latency, and post-training stability can all be preserved at once. This article walks through a practical technical roadmap for Mixture-of-Experts (MoE) models: how to size the base model, how to compress attention KV caches with MLA, why optimizer choices can conflict with attention design, how to keep sparse attention from losing capability, and why agentic reinforcement learning demands an asynchronous system. A final checklist turns these decisions into a review-ready guide.

    When large language models keep stacking parameters, what actually stalls engineering teams is usually not “can we make it bigger,” but whether inference memory footprint, long-context latency, and post-training stability can be held simultaneously. Mixture of Experts (MoE) separates total parameters from per-token activated parameters: model capacity can grow large, but each token is routed to only a few experts. This design suits scenarios with broad knowledge coverage and strict per-inference cost control. Below, the discussion follows an actionable technical主线—choosing the base model, saving attention costs, understanding why optimizers conflict with attention, making sparse attention genuinely efficient, and understanding why agentic RL must be asynchronous.

    1. Start with the Base Model: Total Parameters, Active Parameters, and Data Mix

    MoE fundamentals should be evaluated with three numbers together, not just total parameter count.

    Total parameters determine the model’s capacity to memorize and combine patterns; active parameters determine per-forward compute and memory pressure; expert count and layer depth determine routing granularity and depth. Empirically, doubling total parameters does not automatically yield proportional gains. Slightly increasing active parameters while adding experts and keeping layers sufficiently deep often fits inference budgets better than uniformly scaling up a dense model. When comparing against previous-generation models, report all of: total parameters, active parameters, expert count, layer count, pretraining token volume, and mid-training token volume. Comparing only total parameters conflates “more expensive dense compute” with “larger sparse capacity.”

    Data should also be staged. Main pretraining covers language, code, and general knowledge; mid-training uses higher-quality, longer-context, or tool-use-adjacent distributions for curriculum learning. Its proportion is usually far smaller than main pretraining, but distribution shift here heavily influences later alignment and agentic behavior. Increasing token volume without matching data deduplication, domain mixing, and evaluation slices only creates the illusion of “training more.”

    Before launch, fix a base model card: per-GPU/multi-GPU memory for the active parameters, KV occupancy at target context lengths, the expert-parallel and tensor-parallel sharding scheme, and whether routing produces hot experts. Without this card, subsequent attention and RL optimizations cannot be judged as architectural gains versus evaluation noise.

    2. Attention: Compress KV, Don’t Blindly Extend Context

    For long sequences, the KV cache of standard multi-head attention grows linearly with sequence length and head count, often becoming the bottleneck before compute does. Multi-latent Attention (MLA)-style methods compress keys and values into a lower-dimensional latent space, then reconstruct on demand during decoding or compute directly in the compressed space. Its goal matches Grouped Query Attention (GQA)—store less KV while retaining expressiveness—but the compression path differs: GQA has multiple query heads share one set of KV heads, while MLA applies low-rank/latent compression before caching. MLA’s memory advantage is more pronounced at long sequences, but its implementation depends more on numerical stability and kernel fusion.

    Don’t rely on peak throughput numbers from papers alone. Measure at your target context (e.g., 32K, 128K): time-to-first-token, per-token decode latency, KV occupancy, and quality on long-document QA, codebase retrieval, and multi-turn tool-calling tasks. When compression is too aggressive, short-context scores may barely move, but long-range dependencies and exact copying (function names, config keys, table cells) degrade first.

    In practice, treat MLA as a systems problem: initialization of compression matrices, whether cache layout supports contiguous reads/writes, whether prefill and decoding share a kernel path, and whether expert-parallel communication competes with attention kernels for bandwidth. Swapping the attention formula without changing cache layout often yields gains that get eaten by communication and memory fragmentation.

    3. Optimizer–Attention Coupling: Per-Head Orthogonalization

    One easily overlooked training phenomenon: the optimizer is strongly coupled with attention parameterization. Orthogonalization-based optimizers like Muon impose near-orthogonal constraints on weight updates, making update directions more uniform and sometimes improving large-scale training stability. But applying a single orthogonalization to an entire projection matrix ties the scales and subspaces of different attention heads together. In that case, MLA’s low-dimensional latents are more prone to being “averaged out,” and training curves can fall behind the structurally simpler GQA.

    A workable fix is to move orthogonalization from the full matrix to per-attention-head (or per-head-group) execution—often called split orthogonalization. Each head keeps its own subspace, and the orthogonal constraint operates only within a head, preventing a single global orthogonalization from wiping out inter-head scale differences. This is not a universal switch but a diagnostic rule: when MLA plus an orthogonalizing optimizer underperforms GQA, check the orthogonalization granularity first before blaming attention itself.

    In practice, run three controlled comparisons changing only this one variable: GQA with a standard optimizer, MLA with full-matrix orthogonalization, and MLA with per-head orthogonalization. Track loss, gradient norms, inter-head norm variance, and long-context slice scores. If per-head orthogonalization restores head norm variance and long-context slices, the problem is mainly the optimizer’s scope—not data or routing.

    4. Multi-Token Prediction: Trading Auxiliary Objectives for Sample Efficiency

    Multi-Token Prediction (MTP) trains the model to predict several future tokens beyond the main loss. It doesn’t change the one-token-per-step deployment interface, but provides more supervision per position during training, helping the model learn local syntax, code token conventions, and short-range planning. Implementation typically uses lightweight extra prediction heads or multi-step heads on a shared backbone; these heads can be dropped at inference, keeping the training trick from becoming a permanent decoding cost.

    MTP’s benefit depends on prediction depth and task. Too many steps: distant-token noise rises and the auxiliary loss interferes with the main loss. Too few: it nearly reduces to standard next-token prediction. For code and structured text, 2–4 steps is a common starting search range; final decisions should rest on task scores beyond validation perplexity—patch generation, unit-test pass rates, and long-document citation accuracy. Track the weight between main and auxiliary losses during training, ensuring the auxiliary term doesn’t dominate gradients late in training.

    5. Dynamic Sparse Attention: Saving on Irrelevant Interactions, Not Context Itself

    Dense attention computes all pairwise interactions, growing quadratically with length. Dynamic Sparse Attention (DSA-style methods) selects a subset of key positions per layer or per head, bringing the effective complexity of long contexts closer to linear or piecewise linear. This is not the same as truncating context: truncation discards evidence directly, whereas sparsification—if the selector is reliable—retains highly relevant positions.

    Sparse mechanisms require three co-designed components. First, the selector: local windows, block-level summaries, importance scores, or learnable routing decide which blocks to keep. Second, train–inference consistency: soft masks or differentiable top-k in training versus hard sparsity at inference—a large distribution gap causes “works in training, fails in production.” Third, a fallback path: for copying, counting, and exact-match tasks, local windows must be preserved, not replaced by global sparse sampling.

    A common conclusion in public discussions: when the sparsity pattern matches the data distribution, sparse attention can approach dense long-context quality with significantly less effective compute. Don’t treat any single token-count comparison as a reproducible guarantee. The right acceptance test fixes model and data budgets, then compares dense attention, static local windows, and dynamic sparsity on task scores at equal memory, and on token throughput at equal task scores. If the selector’s own compute exceeds the attention compute it saves, sparsification has failed.

    Block size must align with hardware. Too small: indexing overhead dominates; too large: sparsity becomes distorted. Kernels should load by block, avoiding per-position indirect addressing that saturates bandwidth.

    6. Agentic Reinforcement Learning: Asynchrony Is a System Requirement, Not an Implementation Detail

    When the model works on agentic tasks—calling tools, iteratively editing code, waiting on environment responses—post-training diverges from ordinary preference optimization. Individual trajectories vary widely in duration, tool latency has high variance, and failures are frequent. If you keep synchronous “wait for the slowest trajectory” updates, most GPU time is spent waiting on the environment rather than computing gradients.

    A fully asynchronous agentic RL framework typically splits into three pipelines: environment and tool execution, trajectory caching and filtering, and learner updates. Executors continuously produce trajectories; the learner consumes batches of completed, quality-gated trajectories; weights sync back to executors with some lag. Lag cannot grow unbounded—otherwise the policy has shifted and old trajectories’ advantage estimates become biased. Version numbers, maximum lag steps, and importance sampling or clipping are needed to keep off-policy data from destabilizing updates.

    On infrastructure, a self-built RL stack should at minimum provide: trajectory-level checkpointing, idempotency and timeouts for tool calls, reward decomposition (format, test pass, process constraints logged separately), and replay of failed trajectories. Collapsing reward into a single final score makes it impossible to tell whether degradation stems from routing collapse, tool protocol errors, or reward hacking. Asynchrony also amplifies repeated-sampling hotspots: expert loads and data sources both need balancing, or a few environments or experts dominate gradients.

    Evaluation should use process metrics, not just final outcomes: average tool calls per trajectory, invalid-call rate, timeout rate, trajectory length distribution, and correlation between weight lag and returns. If final scores rise while invalid calls also rise, the reward definition is likely too narrow.

    7. A Practical Review Checklist

    The choices above can be consolidated into a checklist for a single review meeting.

    Architecture: Do total parameters, active parameters, expert count, layer count, routing top-k, and load-balancing loss all meet targets? Attention: Is KV occupancy acceptable at long contexts; if using latent compression, has it been compared head-to-head with GQA at equal budgets? Optimization: Do orthogonalizing optimizers act per-head; do head norms converge abnormally? Training objectives: Have MTP depth and loss weights been searched against validation tasks? Sparsity: Are selector, train–inference consistency, and local fallback all in place; is block size hardware-aligned? Post-training: Are trajectory production and learning decoupled, is weight lag capped, is reward decomposable? Observability: Use the same evaluation slices across pretraining, mid-training, alignment, and agentic evaluation—avoid reporting a single aggregate score.

    When organizing this kind of training in the cloud, separate compute, data, and observability. Put large-scale training jobs on elastic GPU containers or cluster schedulers; store corpora, trajectories, and checkpoints in object storage with clear versioned directories separating pretraining from mid-training data; route logs and metrics into unified logging with monitoring and alerting—at minimum covering loss, routing entropy, KV occupancy, trajectory lag, and tool timeouts. General-purpose cloud infrastructure such as Baidu AI Cloud can cover cluster scheduling, object storage, and logging/monitoring as foundational capabilities, but architectural gains come from the modeling and training design described above, not from any specific product name. For concrete quotas, parallelism strategies, and billing, rely on actually provisioned resources and your own load tests—never substitute external leaderboard scores for your own acceptance criteria.

    8. Closing Thoughts

    Once MoE separates capacity from activation cost, the real differentiators are a set of interlocking decisions: whether latent attention saves KV at long contexts, whether optimizer orthogonalization granularity preserves multi-head structure, whether multi-token prediction becomes a controllable auxiliary signal, whether dynamic sparsity preserves the locally precise information that must be exact, and whether agentic RL removes environment waiting from the GPU critical path. Each looks like an isolated “trick”; together, they form a scalable training system. In reviews, insist on equal-budget comparisons and decomposable metrics—this gets you closer to a deployable model than chasing a single aggregate score.