Trillion-Parameter MoE Models: Sparse Architecture and the Agentic Paradigm

In recent years, the parameter scale of large language models has continued to climb, and teams across the industry often name new models after peaks, stars, and similar imagery—symbolizing repeated attempts to push the boundaries of capability. Among the many technical routes, Mixture-of-Experts (MoE) architectures have become a mainstream choice for trillion-parameter models thanks to their combination of “massive capacity, controllable cost.” Using a typical trillion-parameter MoE design from publicly available technical materials as a reference, this article systematically breaks down its architectural details and extends the discussion to the paradigm shift toward agentic intelligence, offering developers a reference that combines depth of principle with a practical perspective.

1. Background: The Dilemma Between Parameter Scale and Compute Cost

The capability ceiling of a large model depends heavily on parameter scale, but dense architectures face an inherent contradiction: every forward pass activates all parameters, training and inference costs grow approximately linearly with parameter count, and memory footprint quickly hits the ceiling. As parameter counts move toward the hundreds of billions and trillions, both compute budgets and serving latency run into walls that are hard to overcome.

The core idea of MoE (Mixture of Experts) is precisely to decouple “total model capacity” from “per-inference compute”: the model contains a large number of parameterized “expert” subnetworks, and a router dynamically selects a small number of experts for each token to participate in computation. In this way, the model can hold trillions of parameters’ worth of knowledge while the inference cost of a single token is comparable to a medium-sized dense model.

2. Breaking Down the Core Design of MoE Architecture

Using publicly disclosed trillion-parameter MoE designs as a reference, the key configurations can be summarized as follows.

2.1 Decoupling Total Parameters from Activated Parameters

A typical trillion-scale MoE model has a total parameter count of about 1.04 trillion, but only about 32 billion parameters are activated per token during inference—roughly 3%. The value of this design lies in:

• Capacity: 1T parameters provide enormous knowledge storage, allowing the model to cover a broader range of linguistic phenomena, code patterns, and domain knowledge;
• Cost: an activation level of around 32B makes per-token compute cost comparable to a medium-sized dense model, with controllable inference latency and memory bandwidth pressure;
• Scalability: continuing to add experts expands total capacity while activation cost stays essentially unchanged, leaving room for future scaling.

2.2 Expert Granularity: More, Smaller Experts

The reference design uses 384 experts, reflecting the philosophy of “more, smaller experts.” Compared with configuring a small number of large experts, fine-grained experts offer three advantages:

• Greater combinatorial flexibility: the combination space of 8 small experts is far larger than that of a few large experts, allowing the model to express finer-grained knowledge partitioning;
• More precise routing: the more singular an expert’s responsibility, the clearer the router’s learning target, and the easier it is to form stable division of labor;
• More balanced load: fine-grained experts, combined with auxiliary balance losses, make it easier to avoid the collapse problem where “popular experts are overloaded while unpopular experts sit idle.”

2.3 Routing Mechanism and Shared Experts

After router computation, each token is sent to 8 selected experts while also always passing through 1 shared expert. This combination of “Top-K routing + shared expert” is an important lesson from current MoE design:

• Routed experts: handle long-tail, specialized knowledge, activated on demand to achieve sparse division of labor;
• Shared expert: all tokens pass through it, carrying common knowledge such as syntactic structure and high-frequency reasoning patterns. Separating common computation from routed experts reduces redundant learning across experts and improves parameter utilization efficiency.

An illustrative routing logic is shown below (simplified pseudocode to explain the principle):

import torch
import torch.nn as nn
import torch.nn.functional as F
class MoELayer(nn.Module):
    def __init__(self, d_model, d_hidden, num_experts=384, top_k=8, num_shared=1):
        super().__init__()
        self.top_k = top_k
        self.router = nn.Linear(d_model, num_experts, bias=False)
        self.experts = nn.ModuleList([
            nn.Sequential(nn.Linear(d_model, d_hidden), nn.SiLU(),
                          nn.Linear(d_hidden, d_model))
            for _ in range(num_experts)])
        self.shared = nn.Sequential(nn.Linear(d_model, d_hidden), nn.SiLU(),
                                    nn.Linear(d_hidden, d_model))
    def forward(self, x):
        probs = F.softmax(self.router(x), dim=-1)
        topk_p, topk_i = torch.topk(probs, self.top_k, dim=-1)
        topk_p = topk_p / topk_p.sum(dim=-1, keepdim=True)
        out = torch.zeros_like(x)
        for e, expert in enumerate(self.experts):
            hit = (topk_i == e).any(dim=-1)
            if hit.any():
                w = (topk_p * (topk_i == e)).sum(dim=-1)[hit].unsqueeze(-1)
                out[hit] += w * expert(x[hit])
        return out + self.shared(x)  # the shared expert applies to all tokens

2.4 Where to Introduce MoE: Keeping a Dense FFN in the First Layer

An easily overlooked detail is that the reference design keeps a standard FFN only in the first transformer block and introduces MoE starting from the second block. The reasons are:

• The first layer is closer to low-level generic feature transformations (lexical, local patterns), where expertification yields limited benefit;
• Routing decisions in the first layer lack sufficiently abstract semantic signals, and forcing sparsity too early can introduce routing noise and hurt training stability;
• From the second layer onward, features are already separable, and sparse activation can save compute earlier, yielding the best overall benefit.

2.5 Vocabulary and Long-Context Design

The vocabulary size reaches 160k, significantly larger than the commonly seen tens-of-thousands level. A larger vocabulary brings two benefits:

• Encoding efficiency: multilingual text, specialized terminology, and code symbols can be segmented more completely into semantic units, reducing fragmentation;
• Cost compression: the same long text requires fewer tokens, which directly reduces cost and improves throughput in token-metered inference pipelines.

Combined with a context window of up to 256k, the model can process hundreds of pages of documents or a large codebase in one pass, providing a foundation for long-range tasks such as deep reading comprehension and cross-file code refactoring. Of course, long context also imposes higher requirements on positional encoding extrapolation and attention memory optimization; in practice, this usually requires techniques such as parallel attention and KV cache compression.

3. Paradigm Shift: From Static Inference to Agentic Intelligence

Architecture is only the “body”; the usage paradigm is the “soul.” Large model applications are currently undergoing an important paradigm migration.

3.1 Limitations of the Traditional Static Mode

Traditional LLM usage is mainly “one input, one inference, one output.” Even chain-of-thought (CoT) is merely an internal logical unfolding within the model; the model does not interact with the outside world. This mode has clear bottlenecks: knowledge is limited by the training data cutoff and cannot access real-time information; the model cannot verify its own computation results, and long reasoning chains tend to accumulate errors.

3.2 The Think–Act–Think Again Loop

The design goal of the new generation of models is to make them agents: autonomously perceiving, planning, reasoning, and acting in complex dynamic environments. Their working loop can be summarized as:

Think (decompose the task, make a plan) → Act (call tools such as search, code execution, external interfaces) → Observe results → Think again (revise assumptions, adjust the path) → loop until completion.

Through this “think while using tools” approach, the model can break through the limits of static data and use the determinism of external tools to compensate for its own uncertainty—for example, using a code interpreter to verify mathematical derivations or retrieval to obtain the latest facts.

3.3 The Accompanying Evolution of Training Paths

To give a model agentic capabilities, pretraining alone is not enough; two accompanying paths are also needed: first, post-training on large-scale agent trajectory data, so the model has seen many complete sequences of “task—tool call—feedback”; second, reinforcement learning, using whether the task is ultimately completed as the reward signal, so the model learns when to think, when to act, and how to adjust strategy based on observations.

4. Engineering Deployment Considerations

When actually deploying trillion-parameter MoE models, the following points deserve close attention:

• Parallelism strategy: expert parameters are enormous but activations are sparse, so expert parallelism (EP) is usually combined with tensor parallelism and pipeline parallelism; All-to-All communication between experts is a key focus of performance tuning;
• Load balancing: during training, auxiliary losses or routing jitter are relied upon to prevent expert collapse; during inference, replicas and warm-up optimization are needed for popular experts;
• Memory management: keeping expert weights resident in memory is the main cost and can be compressed through quantization, CPU offload, and scheduling strategies;
• Cloud deployment: general-purpose capabilities such as GPU compute, container services, logging services, and monitoring/alerting from cloud platforms can be used to build inference clusters. For example, Baidu AI Cloud provides elastic compute, containerized deployment, and monitoring/alerting capabilities that can support the construction and operation of such services, letting developers focus on tuning routing and parallelism.

5. Conclusion

The essence of trillion-parameter MoE architecture is to strike a balance between enormous model capacity and controllable inference cost through the combination of “fine-grained experts + sparse routing + shared experts.” The agentic paradigm, in turn, evolves the model from a passive text generator into an actor that can perceive the environment, use tools, and self-correct. Together, these two form the main line of current ultra-large-scale model evolution. Understanding the trade-off logic behind these designs is more valuable than memorizing specific parameters—they also apply to developers evaluating and building their own large model application systems.