How MoE and Sparse Attention Are Engineered for Large Models

    An open large-model report tries to grow capacity without letting inference latency and training cost run away. This article walks through its MoE, MLA, Muon Split, sparse attention, multi-token prediction, and asynchronous RL choices, then turns the disclosed numbers into a deployment checklist. All figures come from the report and have not been independently reproduced.

    Large-model parameter counts keep growing, but inference latency and training cost cannot grow with them. The same model has to hold a large body of knowledge and still run in production. This article breaks down several design choices in an open large-model technical report and what each one means in engineering practice. Figures below are taken from the report and have not been independently reproduced; cite the original report when quoting them.

    MoE: control inference cost with activated parameters

    A mixture-of-experts (MoE) model uses a gating network so that each token is computed by only a small number of expert networks. The report discloses 744B total parameters, 40B activated per inference step, 80 layers, and 256 experts. Compared with the previous generation (355B total, 32B activated), total parameters roughly double while activated parameters rise by about 25%. The design emphasis is the balance between capacity and inference cost.

    Pretraining used 28.5T tokens: 27T in regular pretraining and 1.5T in mid-training. The summary does not describe data mix, deduplication, or quality filtering, so data quality cannot be judged from it.

    The report says the model scored 50 on a composite benchmark and ranked first among the open models it compared. Benchmark version, test date, and the comparison set should be checked in the original report.

    MLA and Muon Split: compress the KV cache without losing optimizer stability

    Multi-head latent attention (MLA) applies low-rank compression to the key-value (KV) cache to cut memory use during long-context inference. A smaller cache lets the same memory support a longer context or a larger batch.

    The report says that training MLA with the Muon optimizer converged worse than a simpler grouped-query attention setup, GQA-8. Muon orthogonalizes weight matrices to keep updates stable. In MLA, orthogonalizing an entire projection matrix as one block can weaken independence across attention heads.

    Muon Split cuts the projection matrix by attention head and orthogonalizes each head separately. That keeps Muon’s convergence behavior while letting each head keep its own features. The report does not include ablations across scales, so the method still needs separate validation on other architectures.

    DSA: data-efficiency numbers need the training setup

    Full attention scales quadratically with sequence length, so long context is a major cost. Dynamic sparse attention (DSA) computes attention only between each query and a dynamically selected subset of key tokens, cutting redundant work.

    The report says a DSA model trained on 20B tokens matched a dense-attention model trained on 440B tokens. Confirming that claim requires at least three checks: whether parameter scale and training configuration match, which tasks and metrics were compared, and whether data mix and training steps were the same. Until those are confirmed, treat it as a single result reported in the paper.

    Multi-token prediction: predict several future tokens in one forward pass

    A conventional model predicts only the next token at each step. Multi-token prediction (MTP) predicts several later positions in one forward pass. The report argues this can speed generation and, during training, supply extra planning signal that helps coherence in code generation and logical reasoning. Actual speedup depends on whether the inference stack can use those extra predictions, and the gain has to be measured.

    Asynchronous RL: do not make generation, collection, and updates wait on each other

    In synchronous reinforcement-learning training, the generation side waits for evaluation or environment feedback, so GPUs often sit idle. The asynchronous framework in the report decouples policy generation, experience collection, and model updates: the generator can keep exploring while the updater computes gradients from experience already collected.

    Three issues matter in practice. First, queue length and timeout policy decide how fresh the training data is. Second, policy staleness needs a cap—for example, drop samples older than a set number of versions, or apply an importance correction before using them. Third, failed jobs must be able to roll back to the latest checkpoint so one failure does not abort the whole run.

    Deployment checklist

    Before deploying this class of MoE model, check the following.

    1. Parallelism. Expert parallelism (EP) spreads 256 experts across devices; measure per-expert load so a few devices do not become hotspots. Tensor parallelism (TP) splits large matrices inside a layer. Pipeline parallelism (PP) splits layers across nodes. The mix depends on per-node memory and inter-node bandwidth.

    2. Memory estimate. Weight memory is roughly parameter count times bytes per parameter. At 744B parameters that is about 1.49TB in BF16, about 744GB in FP8, and about 372GB in INT4. Those figures cover weights only, not activations or communication buffers. KV-cache memory is roughly batch size times sequence length times number of layers times cache bytes per token per layer. The longer the context, the larger this share becomes.

    3. Metrics. At minimum, track time to first token (TTFT), time per output token (TPOT), token share per expert, GPU utilization, KV-cache occupancy, and all-to-all time in expert parallelism.

    4. Load testing. Bucket by input length (for example 1K, 8K, and 32K), raise concurrency step by step, and record P50 and P99 latency plus throughput. Sparse attention usually pays off more on long sequences, so long-context traffic needs its own test.

    Takeaway

    The report’s value is that several designs serve one goal: MoE limits activated cost, MLA compresses the cache, DSA cuts long-context compute, and asynchronous RL reduces idle GPUs. Treat each design as a hypothesis you can check on its own, against the report’s numbers and your own hardware, before deciding whether it fits your setup.