Mixture-of-Experts and Sparse Attention: A Practical Training Framework
This article distills the key technical decisions for training large language models with mixture-of-experts (MoE) architectures and sparse attention. It covers separating total parameters from activated parameters, designing attention to reduce cache pressure, using multi-token prediction as an auxiliary objective, making sparse attention selectors learnable and revertible, and building reinforcement learning infrastructure for agentic tasks under asynchronous sampling. The goal is to provide an actionable framework for balancing architecture, optimization, long-context handling, and training stability.
Design Total Parameters and Activated Parameters Separately
The core of Mixture-of-Experts is not simply duplicating a dense network multiple times. Instead, the feed-forward computation is split into several experts, and a routing network selects a small number of experts for each token. Total parameters determine the model’s expressive capacity, while activated parameters determine the compute, memory bandwidth, and latency for a single forward pass. Common practice in the industry is to scale total parameters to hundreds of billions while activating only tens of billions per step, with hundreds of experts and tens to hundreds of layers. The benefit is that pretraining can continue consuming larger corpora, but inference cost does not grow linearly with total parameters.
Compared to previous dense models or those with fewer experts and higher activation ratios, scaling should prioritize increasing the expert pool and total parameters, while being cautious about widening activation. Every increase in activated parameters directly raises per-step matrix multiplications, communication, and computation beyond key-value caching. A safer path is typically: roughly double the total parameters, make only small increases to activated parameters, extend the pretraining corpus from around 20 trillion tokens toward 30 trillion, and insert a mid-training phase after main pretraining using higher-quality, longer-context, or more tool-call samples for distribution correction. Mid-training should not be too long; otherwise, it can erode general capabilities. It is better suited for capability shaping rather than a second round of pretraining from scratch.
Routing must monitor three things simultaneously: load balancing, whether expert specialization collapses, and whether cross-device communication becomes a bottleneck. If a few experts consistently consume most tokens, the “idle experts” in total parameters are just memory decoration. Common techniques include auxiliary load-balancing losses, expert capacity limits, and moderate noise, but too much loss weight forces routing to give up semantic specialization. In practice, use offline statistics and online monitoring to observe the variance of expert utilization, not just average load.
Attention Should Save Cache for Long Contexts, Not Just Chase Theoretical Speed
Traditional multi-head attention stores the key and value of every head completely. When sequences get long, the cache often fills the accelerator memory before parameters do. Grouped-query attention reduces cache by making multiple query heads share a set of keys and values; it is simple to implement and stable to train. Multi-latent attention compresses keys and values into a lower-dimensional latent space and reconstructs them when needed, which offers clear advantages in memory and bandwidth for very long texts, especially beyond 100k tokens.
The real challenge is not the formula but whether the optimizer and parameterization match. If you use an optimizer that emphasizes matrix orthogonalization and more uniform weight updates, directly orthogonalizing the entire projection matrix can sometimes be worse than a simpler grouped-query approach. The reason is usually that different attention heads have different statistical scales; whole-matrix orthogonalization forces them into the same geometric constraint, making compressed latent variables more sensitive and amplifying gradient noise.
A feasible fix is to split orthogonalization by head: each attention head (or each shared key-value group) is orthogonalized and scaled separately, and the projection matrix is no longer treated as one indivisible block. This preserves the benefit of more uniform update directions and avoids dominance by a few singular directions, while allowing each head to learn according to its own norm. When implementing, note three points. First, the split communication and kernels should be fused whenever possible; otherwise per-head loops will cancel out the time saved by attention. Second, orthogonalization is numerically sensitive; in mixed precision, perform the decomposition or approximate orthogonalization at higher precision before writing updates back to low-precision weights. Third, after changing the optimizer, you must redo the joint search for learning rate, weight decay, and gradient clipping; do not reuse hyperparameters from a dense model.
Multi-Token Prediction Is an Auxiliary Objective, Not a Replacement for the Main Loss
Predicting only the next token gives sparse supervision, and long-range dependencies must propagate over many steps. Multi-token prediction lets the same hidden state predict several future positions at once, adding short-term self-supervised density to the main trunk. It often improves code, mathematics, and tasks requiring local planning, and it naturally pairs with speculative decoding at inference.
It should not overshadow the main objective. If the prediction span is too large, label noise rises and the primary token perplexity may worsen; if too small, it is just repeated computation with the same target. In practice, make the multi-token head a lightweight side branch that shares the trunk with its own small output layer, and decay the loss weight over training or enable it only in the middle and later phases. Evaluate the main token loss and the auxiliary loss separately, so you are not misled by auxiliary metrics improving on their own.
Sparse Attention Should Trade Computation for Data Efficiency, but the Selector Must Be Learnable and Revertible
Dense attention costs grow quadratically with sequence length. Sparse or dynamic sparse attention first selects “likely relevant” blocks at the block level, then performs fine-grained attention only on those blocks. Its value is marginal on short texts but significant on long documents, multi-turn tool traces, and repository-level code: under the same quality target, effective compute and key-value reads/writes can drop by an order of magnitude, allowing medium-scale long-context continued training to approach results that previously required extremely long dense training.
Design and validation should separate “selection” from “computation.” The selector can use block-level summaries, a local sliding window plus a small number of global blocks, or query-based learnable scoring. Regardless of the method, keep a sliding window and a few sentinel blocks as a lower bound so that key evidence is not lost when the selector misses something. Early in training, the selector is unstable; start with a higher retention rate and gradually sparsify. On the data side, instead of stacking unlimited low-quality long texts, use a corpus one or two orders of magnitude smaller but with length distribution and task structure closer to the target for targeted alignment. Do not interpret “fewer tokens” as permission to skip data cleaning. Sparse models are more sensitive to noisy positions because a missed block may never be seen again.
The memory budget also needs to be recalculated. Sparsity does not mean the cache can be discarded carelessly: the selector itself, block indices, and irregular cross-device access all add overhead. If the kernel falls back to many small matrix multiplications, theoretical FLOPs drop but wall-clock time may rise. Before deployment, profile with fixed batch sizes and fixed length buckets to confirm that the gain comes from bandwidth, not benchmark noise.
Agentic Reinforcement Learning Must Accept Asynchrony but Cannot Accept Unconstrained Off-Policy Behavior
When a model uses tools and executes multi-step tasks, the environment feedback is slow and trajectory lengths vary widely. Synchronously waiting for all samples before updating will leave accelerators idle on I/O. Fully asynchronous approaches decouple environment sampling, reward evaluation, experience replay, and policy updates: samplers continuously write to a buffer, and learners consume at their own pace.
Asynchrony introduces off-policy bias. Trajectories may be generated by a policy several versions old; if you update with the assumptions of an online algorithm, the policy can oscillate or collapse. The infrastructure should at least include these capabilities: each trajectory carries a policy version and sampling timestamp; replay decays by freshness or sets a maximum lag step; advantage estimation uses the old log-probabilities consistent with the behavior policy, not newly recomputed log-probabilities pretending they are still on-policy; and reward models or rule-based rewards must be reproducible, avoiding environment-side random score rewrites. Clipping, importance sampling ratio caps, and reference-model constraints are still necessary, but the hyperparameters should be more conservative than for short-response alignment.
Rewards should be split into separate components. Whether a task completes, whether tool calls are legal, whether there is leakage of intermediate keys or useless loops, and whether the answer is overly verbose should all be observable sub-scores that are then combined into a total reward. A single vague “good/bad” score only amplifies noise in an asynchronous system. The evaluation set must include failure trajectories and partially successful trajectories; otherwise, the policy will learn to exploit the reward rather than complete the task.
A Joint Checklist for Training and Inference
Parameters and parallelism: Expert parallelism, tensor parallelism, and sequence parallelism should be partitioned based on activated parameters, not total parameters. Cold experts can be swapped to cheaper storage when possible, but avoid frequent swapping during routing jitter. Checkpoints should also save routing statistics, so you can roll back from expert collapse.
Numerical and optimization: Attention projections, sparse selectors, and expert routers are the three parameter types most prone to numerical instability; group them by learning rate. When mixing orthogonalization-style optimizers with adaptive optimizers, document which tensors follow which update path to avoid double-scaling the same weight with two state definitions.
Long context: Verify positional encoding behavior at extrapolated lengths before turning on sparsity. Do not change cache compression and sparse selection in the same experiment. Use a curriculum from short to long lengths and watch for sudden perplexity jumps at length switches.
Observability: Log expert load, attention block hit rate, cache hit rate, sampling-to-learning version gap, reward sub-scores, and rejection rates. Logs, metrics, and object storage should use generic interfaces; no need to bind to a proprietary system. Set alerts on leading indicators such as sustained increase in load variance or version lag exceeding thresholds.
Evaluation: General capability, code, long-context retrieval, and agentic tasks should be reported in separate leaderboards; a single composite score can hide regressions. Compare using task definitions and evaluation protocols, not marketing language. Open-source weights and closed-source APIs can be evaluated under the same protocol for trend comparison, but report temperature, context length, tool permissions, and whether multiple attempts are allowed.
How to Make Trade-Offs
If the business is mainly short conversations, grouped-query attention with a moderate MoE is usually simpler. If documents, codebases, and multi-turn tool traces are the primary scenario, treat cache compression and block-level sparsity as first-class citizens, and use per-head orthogonalization to fix optimizer mismatches. If the goal is an agent that calls tools, asynchronous sampling is a prerequisite for throughput, and versioned replay plus component-wise rewards are prerequisites for stable training. Multi-token prediction is useful as auxiliary density and serves speculative decoding later, but do not expect it to lift all benchmarks by itself.
There is no single parameter recipe. Total parameters, activated parameters, sparsity ratio, and asynchronous lag are coupled: wider activation leads to heavier expert communication; more aggressive sparsity demands a more stable selector; more asynchronous sampling demands stronger off-policy constraints. Ablate these four factors separately and then combine them into one training pipeline. That approach is closer to reproducible engineering than chasing any single leaderboard score.
