Ultra-Large MoE Models: Capability Boundaries and Key Deployment Considerations

    This article explains why Mixture of Experts (MoE) architecture has become a common choice for large language models, along with its memory and network bandwidth costs. It also covers the KV Cache cost of million-token context windows, common mitigation methods, how to test long-context retrieval, and the post-training abilities that long-horizon AI agents need, including task decomposition, error recovery, and context compression.

    1. Why MoE Has Become the Mainstream Choice for Large Models

    Dense models use all parameters for every generated token, so the more parameters a model has, the higher the compute and latency cost of each inference. The Mixture of Experts (MoE) architecture splits the feed-forward network into multiple groups of experts. A router selects a small number of experts for each token. As a result, a model’s total parameter count and the number of parameters actually used per inference are two different figures.

    Consider a hypothetical model with 1 trillion total parameters and about 30 billion active parameters per inference, an activation ratio of about 3%. This design can store more knowledge in its parameters while keeping per-token compute close to that of a mid-sized dense model.

    The architecture also brings new costs. First, all total parameters must still fit in GPU memory or be spread across multiple cards, so the memory requirement does not drop just because fewer parameters are active. Second, the router may concentrate many tokens on a few experts, so load-balancing losses or other auxiliary strategies are needed to keep some experts from staying idle for long periods. Third, when experts are distributed across devices, tokens must be communicated between cards, and network bandwidth often becomes the bottleneck.

    2. The Practical Cost of a One-Million-Token Context

    Extending the context window to 1 million tokens means the model can read hundreds of thousands of words at once, such as a long research report, a large code repository, or a lengthy conversation log. The main cost of this capability falls on the KV Cache during inference.

    The KV Cache size can be estimated roughly. For each token, the model must store the Key and Value for every layer. Take a model with 80 layers, 8 KV heads, a head dimension of 128, and FP16 storage. The cache for a single token is about 2 × 80 × 8 × 128 × 2 bytes, or roughly 320 KB. At a sequence length of 1 million tokens, the KV Cache for one request reaches about 320 GB, not counting model weights and activations.

    Common mitigation methods include:

    • Grouped-Query Attention (GQA), which reduces the number of KV heads.
    • KV Cache quantization, such as reducing from FP16 to FP8 or INT8, which halves memory usage.
    • Context truncation or summarization, which compresses historical content that is no longer needed and keeps the rest.

    At the algorithm level, long context relies on extending position encodings (for example, scaling RoPE frequencies), on efficient attention kernels such as FlashAttention, and on methods like Ring Attention that split sequences across devices. After position encoding is extended, the model’s retrieval ability may not be consistent across different positions. Before deployment, test the model on your own long documents: bury key information in a long text and check whether the model can find it, and also measure accuracy on reasoning that spans multiple passages.

    3. Post-Training Capabilities Needed for Long-Horizon Agents

    Long-horizon agents must stay coherent across multi-step reasoning, tool calls, and state management. Their failures are usually not single-step errors. Instead, they drift from the goal after a dozen or more steps, or they repeatedly run the same failing command after an error. Post-training should therefore focus on three capabilities:

    • Task decomposition: break a goal into verifiable subtasks, each with clear inputs and completion criteria.
    • Error recovery: read the error messages returned by tools, identify the cause, and adjust parameters or re-plan, instead of retrying blindly.
    • Context compression: distill key conclusions over a long execution chain, keep file paths and unfinished items, and discard verbose intermediate logs.

    During the reinforcement learning stage, the training environment is typically an isolated sandbox