Ultra-Large MoE Model Training and Inference: Sparse Activation, Long Context, and Agentic RL

    This article examines how ultra-large mixture-of-experts (MoE) models are trained and deployed for agentic use cases. Using the public technical characteristics of a recently released general-purpose model as a reference, it explains how 744 billion total parameters can be served with roughly 40 billion active parameters, how 28.5T pre-training tokens are organized, how asynchronous agentic reinforcement learning decouples sampling from training, and why 128K output and 200K context windows matter. It also provides practical guidance on cost estimation, retrieval use, evaluation, and cloud-based serving.

    1. Background: Why the Agent Era Needs a New Model Shape

    As large models move from conversational question answering toward autonomous task execution, the capability profile required from them is changing significantly. Traditional conversational models emphasize single-turn understanding and generation quality. Agentic scenarios, however, require long-horizon planning, tool use, multi-step reasoning, and result reflection.

    The key characteristic of these tasks is that a single task may span dozens of interaction turns. Each turn may include long tool-returned text, placing heavy demands on both context length and output length.

    A recently released general-purpose model is designed around this trend. Its maximum output is 128K tokens, and its context window reaches 200K tokens, making it a typical “long input, long output” configuration.

    This article does not focus on one specific product. Instead, it uses the publicly described technical characteristics of this type of model as material to analyze the general technical logic of ultra-large MoE models and agentic reinforcement learning training.

    2. MoE Architecture: Why 744B Parameters Only Activate About 40B

    2.1 The Basic Principle of Sparse Activation

    The model has 744 billion total parameters, but each forward pass activates only about 40B to 44B parameters. This is a typical characteristic of the Mixture-of-Experts architecture.

    The core idea is to replace the feed-forward network (FFN) layers in a Transformer with multiple “expert” sub-networks. A router then selects a small subset of experts to participate in computation based on the semantic features of each input token.

    The model uses an architecture with 80 layers and 256 experts. Its capacity design can be roughly understood as follows:

    1. # Example: MoE routing process (pseudocode)
    2. def moe_forward(x, experts, router, top_k=2):
    3. logits = router(x) # [batch, seq, num_experts]
    4. weights, indices = topk(logits, k=top_k)
    5. output = sum(
    6. weights[..., i] * experts[indices[..., i]](x)
    7. for i in range(top_k)
    8. )
    9. return output

    This design brings three main benefits:

    1. Decoupling capability ceiling from compute cost: Total parameters determine the knowledge capacity of the model, while active parameters determine the compute cost of a single inference. With 744 billion parameters of knowledge storage and about 40B active parameters, inference cost can approach that of a medium-sized dense model.

    2. Specialization through expert division of labor: Different experts can naturally specialize during training, becoming better at patterns such as code, mathematics, multilingual processing, or tool calling.

    3. Load balancing becomes a critical engineering issue: If routing over-concentrates on a small number of experts, training may become unstable. In practice, load-balancing auxiliary losses or capacity factor limits are usually introduced to ensure that all 256 experts are fully and evenly utilized.

    2.2 Tradeoffs of an 80-Layer Depth

    Layer depth determines how many times the model can perform serial representation refinement. An 80-layer configuration is relatively deep. Deep networks are helpful for constructing complex reasoning chains, but they also introduce training stability challenges. These usually require adjustments such as normalization strategy tuning, residual scaling, and careful learning-rate warm-up.

    3. Pre-training: Data Engineering with 28.5T Tokens

    The model is pre-trained on 28.5T tokens. At this scale, it represents a complete data engineering system:

    • Multi-source data mixture: The data covers web text, code, mathematics, books, and multilingual corpora. For agentic models, the proportion of code and synthetic reasoning data often increases significantly, because tool calling is essentially structured procedural knowledge.

    • Multi-stage curriculum: A common approach is to first use general corpora to build foundational capabilities, then increase the weight of high-quality reasoning data in the middle and later stages, and finally perform extension training with long-context corpora. This helps make the 200K context window truly usable rather than merely supported.

    • Quality filtering and deduplication: At the scale of tens of trillions of tokens, heuristic filtering, model-based scoring, and MinHash-like deduplication are standard processes. Data quality often contributes more to final capability than simply increasing data volume.

    4. Training Framework and Asynchronous Agentic Reinforcement Learning

    4.1 Why Agentic Training Needs Asynchrony

    The traditional RLHF process is a synchronous loop of “generation—annotation—update”. GPUs alternate between idle and active states during sampling and training phases, leading to low utilization.

    The problem is even more severe in agentic scenarios. One task trajectory may contain dozens of tool-calling rounds, and environment response times are unpredictable. A synchronous pipeline can therefore be seriously slowed down.

    The idea of asynchronous agentic reinforcement learning is:

    1. Decouple sampling from training: Many agents interact with the environment in parallel and continuously produce trajectories.

    2. Pipeline trajectories: Completed trajectories enter an experience buffer, and the training process consumes them in batches.

    3. Parallelize environments: Environment components such as tool execution and code execution sandboxes can be scaled independently to avoid becoming bottlenecks.

    The Slime-like training framework mentioned in the article is an engineering embodiment of this approach. It decouples the inference engine, training engine, and environment scheduler, allowing each to scale independently and supporting high-throughput sampling of ultra-long trajectories.

    4.2 Reward Design: From Outcome Rewards to Process Signals

    Rewards in agentic tasks usually come from whether a task is completed, such as whether unit tests pass or whether information retrieval is correct. These are sparse rewards.

    Common ways to improve training efficiency include:

    • Process-level rewards: Scoring intermediate reasoning steps.
    • Format rewards: Constraining the correctness of tool-calling syntax.
    • Partial matching against reference answers: Providing scores based on reference solutions.

    Reward function design also needs to prevent “reward hacking”. For example, the model may learn to repeatedly call tools without making actual progress on the task.

    5. The Practical Value of 128K Output and 200K Context

    Long output capability is often underestimated. When agents complete tasks, they may need to generate a complete multi-file code project, a long report, or structured data in a single pass.

    A 128K output limit means:

    • A single generation can cover the core files of a medium-sized code repository, reducing consistency breaks caused by multi-turn stitching.
    • Complex reasoning chains can be fully expanded without losing conclusions due to truncation.
    • Combined with a 200K context window, the model can read large codebases or document sets in one session and complete a full “understand—modify—output” loop.

    From an engineering perspective, long-context support depends on several general techniques:

    • RoPE-based extrapolation extensions.
    • Efficient KV cache management, such as PagedAttention-like paging mechanisms.
    • Sparse approximation in attention computation.

    6. Practical Recommendations

    For teams planning to integrate this type of model into their business, the following points deserve attention:

    1. Estimate cost using active parameters: The inference cost of MoE models aligns with active parameters rather than total parameters. However, in batch scenarios, memory usage caused by expert routing also needs attention.

    2. Long context still requires retrieval: A long context window does not mean RAG is unnecessary. The “lost in the middle” problem still exists. Key evidence should be placed earlier or later in the context.

    3. Build evaluation sets for async RL-friendly models: Agentic capability should be evaluated using end-to-end task success rates rather than single-turn question-answering accuracy.

    4. Leverage cloud managed capabilities: On the inference side, cloud platforms can provide elastic compute, model services, monitoring alerts, and logging services to build an observable production pipeline and reduce operational burden.

    7. Conclusion

    From the combination of 744B total parameters and about 40B active parameters, to the specification of 200K context and 128K output, and further to the asynchronous reinforcement learning training paradigm for agentic tasks, a clear mainline of large model technology evolution emerges:

    Sparsity controls cost, long sequences support complex tasks, and environment-driven reinforcement learning aligns models with real application needs.

    Understanding these underlying logic is more valuable in the long term than memorizing the parameter list of a specific model. They constitute the general design language of next-generation agent infrastructure.