Inside Trillion-Parameter MoE Models: Key Architectural Innovations and Design Tradeoffs

1. Background: Reconciling “Larger” and “Affordable” Under Scaling Laws

Progress in large model capability has long been driven by scaling laws: more parameters, more data, and more compute yield stronger capabilities. But dense architectures have an unavoidable constraint—every token must flow through all parameters, so training and inference costs grow almost linearly with parameter count. As models approach the trillion-parameter scale, sticking with a dense structure would make both training budgets and deployment costs unmanageable.

Mixture of Experts (MoE) is the answer to this tension. It fully decouples two quantities that were previously bound together:

• Knowledge capacity: Determined by total parameter count, which can reach the trillions and is used to carry massive world knowledge and specialized skills.
• Per-token compute: Determined by the number of activated parameters, which can be compressed to a very small fraction.

Take a representative publicly disclosed configuration as an example: roughly 1.04 trillion total parameters, with only about 32 billion activated per token during inference—an activation ratio of about 3%. Intuitively, such a model delivers “trillion-scale knowledge capacity” at “the inference cost of a 32B dense model.” This is the fundamental reason it sparked broad discussion in the international community—not as a parameter-count gimmick, but because the industry proposition of “capability keeps growing while cost stays controllable” has been genuinely engineered.

2. Core Architecture: Key Design Choices of Fine-Grained MoE

2.1 Many Small Experts: Fine-Grained Specialization and Combinatorial Explosion

The representative configuration does not adopt a coarse-grained approach with “a few large experts.” Instead, it deploys 384 experts, with each token routed to only 8 of them. This “many and small” design offers two benefits:

1. Finer knowledge specialization: Smaller experts are more likely to converge to specific domains during training—mathematical derivation, code structure, and multilingual expression each land on different experts, avoiding the situation where “one expert knows a little about everything and excels at nothing.”
2. Exponentially larger combinatorial space: Choosing 8 out of 384 experts yields a theoretical combination count on the order of 10^16. Different inputs can receive vastly different “expert recipes,” equivalent to the model possessing representational diversity far beyond the upper bound of the expert count itself.

2.2 Routing Mechanism: Top-K Complemented by a Shared Expert

The router scores all experts for each token, selects the Top-8 experts, and fuses their outputs by weight. On top of this, one shared expert is added: it is unconditionally open to all tokens and carries high-frequency general knowledge, while routed experts focus on differentiated long-tail knowledge. The two complement each other and significantly reduce knowledge redundancy across experts. The process can be described with the following language-agnostic pseudocode:

function moe_ffn(x):
    # Shared expert: always active, carries general knowledge
    out = shared_expert(x)
    # Router scores all experts for each token
    scores = router(x)              # [num_tokens, 384]
    top_idx, top_w = topk(scores, k=8)
    # Dispatch only the hit experts and accumulate with weights
    for (i, w) in zip(top_idx, top_w):
        out = out + w * expert[i](x)
    return out

2.3 Training Stability: MoE Is Not Introduced in the First Layer

One easily overlooked detail: in the representative implementation, the first Transformer block retains a standard FFN, and the MoE structure is introduced only from the second block onward. The reason is that shallow layers mainly learn general low-level representations, and routing signals are noisy at that stage—premature sparsification tends to cause optimization instability. Once the low-level representations take shape, introducing sparse activation allows the model to enjoy compute benefits early without sacrificing training quality. In addition, MoE training typically requires an auxiliary load-balancing loss to constrain the expert hit distribution to be as uniform as possible, preventing routing from “clustering” and causing some experts to be overloaded while others sit idle, which would shrink the effective capacity.

3. Vocabulary and Context: Two Underestimated Engineering Levers

3.1 A 160k Vocabulary: Saving Tokens Saves Money

The representative configuration expands the vocabulary to 160k, with benefits in three areas: multilingual text and specialized terminology (code keywords, academic terms) can be encoded as whole words, so the same passage consumes noticeably fewer tokens, directly reducing inference and fine-tuning costs in long-text scenarios; with “longer” tokens, the same context window can hold more effective information; and fewer segmentation fragments reduce noise when the model parses long, complex sentences. The cost is increased Embedding parameters and output-layer softmax overhead—a typical case of “one-time parameter overhead in exchange for ongoing inference gains.”

3.2 A 256k Context Window: From “Q&A” to “Assignments”

256k tokens correspond roughly to hundreds of pages of documents or a medium-sized codebase—precisely the capability boundary required for deep research and long-form code engineering. But two caveats apply: first, a nominal window length is not the same as effective utilization length; information in the middle of long contexts often degrades in utilization and must be verified with needle-in-a-haystack style retrieval evaluations. Second, long context depends on a whole set of supporting techniques—rotary position encoding and its various scaling and extrapolation strategies, KV Cache management, and attention optimization—not a single metric.

4. Paradigm Shift: From Chain-of-Thought to “Reason–Act–Reason”

If MoE solves “affording the compute,” the agentic paradigm solves “using it flexibly.”

Traditional large-model reasoning is static: the chain of thought unfolds inside the model in one pass, with no environment participation and no opportunity to intervene in results. The agentic paradigm explicitly externalizes the interaction loop: Reasoning → Action (calling search, executing code, requesting external interfaces) → Observation (reading feedback) → Reasoning again.

On the training path, the industry-standard approach is a two-stage process of “large-scale agent data + reinforcement learning”: first construct tool-use trajectories covering real environments such as search, code execution, and API calls, so the model learns “what a trajectory looks like”; then use final task success or failure as the reward signal for reinforcement learning, so the model autonomously learns across multi-turn interactions “when to call tools and how to revise strategy based on feedback,” rather than frame-by-frame imitation.

The key change lies in how capabilities are acquired: the model is no longer capped by static corpora and can acquire new capabilities that do not exist in the corpus through exploration and feedback in dynamic environments—evolving from a “knowledge container” into a “task executor.”

5. Engineering Deployment: Three Practical Ledgers

• Memory ledger: Sparse activation saves compute, not memory. Routing is dynamic, and any expert may be hit, so all trillion parameters must reside in memory at all times. As a rough estimate, storing about 1 trillion parameters in INT8 alone requires roughly 1TB of memory; adding KV Cache and activation overhead means multi-machine, multi-GPU deployment with expert parallelism (EP) alongside tensor/pipeline parallelism (TP/PP) is mandatory.
• Throughput ledger: MoE’s throughput advantage is more pronounced at large batch sizes—different tokens hit different experts, so the batch as a whole covers more experts and hardware utilization is higher. In production, expert hit distribution should be continuously monitored to prevent traffic skew.
• Infrastructure ledger: There is no need to build everything from scratch. Leveraging general-purpose services from cloud platforms such as Baidu AI Cloud—GPU compute and container platforms for multi-machine orchestration and elastic scaling, object storage for model weights and datasets, and logging and monitoring to observe inference latency, memory levels, and expert load—can help teams stand up a sustainably operable inference platform faster and leave their energy for the model and the business itself.

6. Model Selection and Evaluation Recommendations

1. Scenario fit: Long-document research, complex code engineering, and multi-step automation are the home turf of “large-context + agentic” models; simple short Q&A may not be worth paying inference costs for.
2. Distinguish Base from agentic versions: Base models are suited as a foundation for secondary development and domain fine-tuning; agentic versions target direct deployment, and evaluation criteria should match actual usage.
3. Suggested evaluation dimensions: general and mathematical/code reasoning, tool-call success rate and end-to-end task completion rate, long-context retrieval accuracy, and token encoding efficiency for Chinese and code.

7. Conclusion: The “Deeper Meaning” Written into the Architecture

The attention that trillion-parameter MoE models have received from the international community is not a victory of number-driven marketing—it simultaneously answers three ultimate questions of large-model industrialization: how capability can grow larger—trillion-level knowledge capacity; how cost can shrink—roughly 3% sparse activation; and how capability can become more flexible—the “reason–act–reason” agentic loop. Many teams in the industry like to name their flagship models after climbing peaks or expeditions to the stars, metaphorically expressing this engineering devotion to “deliberately taking the hardest route”—but the name is only the prologue; the real deeper meaning is always written into the architecture, the data, and the training methods.