1. Introduction: Two Main Threads in LLM Evolution

Over the past few years, the evolution of large language models has followed two clear threads:

• Vertical: continuously scaling parameter count and training data to improve base capabilities—what the industry calls the scaling law;
• Horizontal: architectural innovation and paradigm shifts that move models from “passive response” toward “active action.”

MoE (Mixture of Experts) is a key breakthrough on the first thread, allowing “trillion-level total parameters” to coexist with “controllable inference cost.” The agentic paradigm represents the second thread—models no longer output only a static block of text, but perceive, plan, call tools, and reflect continuously in dynamic environments. This article examines both threads, breaking down the key design decisions of trillion-parameter MoE models and discussing their engineering deployment paths.

2. MoE Architecture: How Sparse Activation Balances Scale and Cost

2.1 The Basic Principle of Sparse Activation

In traditional dense architectures, every token passes through all feed-forward network (FFN) parameters, so parameter count scales proportionally with per-inference compute—larger models mean linearly rising costs. MoE splits the large FFN into multiple parallel “expert” networks and introduces a lightweight router (gate) that dynamically selects a small subset of experts for each token.

This creates a mismatched structure: “large total parameters, small activated parameters.” In representative publicly disclosed designs, total parameters can reach the trillion level, while the parameters actually activated per token typically account for only 3%–5% of the total (tens of billions). This means:

• The model’s “knowledge capacity” grows with total parameters;
• Per-token inference compute depends approximately only on activated parameters;
• Under a similar compute budget, training can accommodate more knowledge and data.

2.2 Expert Design Philosophy: More, Smaller Experts

On expert count and granularity, the industry trend is “more and smaller”:

• Expert counts have grown from a few or a dozen early on to dozens or even hundreds;
• Individual expert parameter size shrinks accordingly, and routing can select finer-grained knowledge combinations;
• Fine-grained experts increase combinatorial flexibility—under the same activation budget, the number of selectable expert combinations grows exponentially, letting the model tailor computation paths for different tokens.

The cost is a larger routing decision space and harder training stability, which requires techniques such as load balancing (see Section 5).

2.3 Routing Mechanism and Shared Experts

A typical routing flow: apply a linear mapping to the token hidden vector to obtain scores for each expert, select the top-scoring “routed experts,” then compute a weighted sum of expert outputs using normalized weights. In addition, most modern implementations include one or a few “shared experts”: regardless of routing results, shared experts always participate, dedicated to general knowledge such as syntax and high-frequency common sense, so routed experts can focus on long-tail and specialized divisions of labor, reducing functional redundancy.

Illustrative code (simplified router, mechanism only):

import torch
import torch.nn.functional as F
class MoERouter(torch.nn.Module):
    def __init__(self, hidden_dim, num_experts, top_k):
        super().__init__()
        self.gate = torch.nn.Linear(hidden_dim, num_experts, bias=False)
        self.top_k = top_k
    def forward(self, x):
        # x: [batch, seq_len, hidden_dim]
        logits = self.gate(x)                        # scores for each expert
        probs = F.softmax(logits, dim=-1)
        topk_probs, topk_idx = probs.topk(self.top_k, dim=-1)
        topk_probs = topk_probs / topk_probs.sum(-1, keepdim=True)  # renormalize
        return topk_probs, topk_idx

2.4 Where to Introduce MoE: No Need to “Start from the First Layer”

An easily overlooked detail is where MoE is introduced. Many implementations keep standard FFN in the first one or two Transformer blocks and replace with MoE layers only from later blocks. The reasons: shallow layers handle more general feature transformation and syntax-level processing, where tasks are relatively homogeneous and sparsification yields limited benefit; deeper layers show clearer knowledge specialization, where sparse activation pays off more; and the earlier MoE is introduced, the greater the impact of routing training instability on the whole model. “FFN in the first block, MoE thereafter” has thus become a common compromise between efficiency and stability.

3. Vocabulary and Long Context: Two Underappreciated Design Points

3.1 Larger Vocabularies

Mainstream model vocabularies have grown from tens of thousands of tokens early on to over a hundred thousand or even more than two hundred thousand, with benefits in three areas:

• More efficient encoding of multilingual and technical terms: Chinese, code, and math symbols can be split into more complete “whole words/symbols,” reducing fragmentation;
• Fewer tokens for long text: The same document consumes fewer tokens, directly lowering inference cost and effectively expanding usable context;
• More complete semantic units: Less semantic noise from over-segmentation.

The trade-off: embedding matrix and output layer parameter counts and softmax overhead grow linearly with vocabulary size, and training sufficiency for low-frequency tokens requires more corpus, so vocabulary size should match training corpus scale.

3.2 Long Context Windows

Context windows in the hundreds of thousands of tokens (e.g., 256k) are becoming a common configuration for high-end models. Their value:

• Long document understanding: Hundreds of pages of research reports, contracts, or papers can be placed entirely in context for cross-chapter reasoning;
• Large codebases: Cross-file dependency analysis and repository-level refactoring become possible;
• Agent tasks: The large volume of intermediate observations from multi-turn tool calls requires sufficiently long context.

In engineering, long context challenges KV cache memory and attention compute complexity, typically requiring general optimizations such as chunked fused attention, sparse or linear attention, plus KV cache quantization, paged management, and rolling summarization to control costs.

4. Paradigm Shift: From Static Response to Agentic Intelligence

4.1 Limitations of Traditional LLMs

The traditional LLM working mode can be summarized as “static learning + one-shot response”: techniques such as chain-of-thought (CoT) let the model unfold multi-step reasoning internally, but the process is closed within the model and cannot interact with the external world; the model’s capability boundary is fixed at training completion, leaving it helpless with new tools or environments; complex tasks must be manually decomposed into multiple Q&A rounds, and the model itself lacks “continuous operation” capability.

4.2 The “Think—Act—Think Again” Agentic Paradigm

A new generation of models shifts the paradigm to a “think → act → observe → think again” loop:

• Think: Decompose goals, plan the next step, decide whether external information is needed;
• Act: Call external tools such as retrieval, code execution, and database queries;
• Observe: Obtain tool return results;
• Think again: Revise the plan based on observations and enter the next loop until the task is complete.

The model can thus autonomously perceive, plan, reason, and act in complex dynamic environments, acquiring capabilities beyond training data through continuous tool use and breaking free from static knowledge limits.

def agent_loop(task, llm, tools, max_steps=10):
    context = build_initial_context(task)
    for _ in range(max_steps):
        thought = llm.generate(context, mode="think")  # think: plan the next step
        action = parse_action(thought)
        if action.type == "FINISH":                    # check whether the task is complete
            return action.result
        observation = tools.execute(action)            # act: call external tools
        context = update(context, thought, action, observation)  # think again
    return fallback(context)

4.3 Training Methods That Support Agentic Capabilities

Agentic capabilities cannot be obtained from prompt engineering alone; they usually depend on systematic training:

• Large-scale agentic data construction: Build high-quality interaction trajectories around tool calls and multi-turn task solving, first teaching the model the “syntax of actions” through supervised fine-tuning;
• Reinforcement learning from execution feedback: Use whether the task is actually completed as the reward signal, letting the model trial and error repeatedly in real or simulated environments to optimize long-horizon decisions;
• Coordinating reasoning and agentic capabilities: Avoid sacrificing base reasoning for tool use; the two must be jointly weighed in training objectives.

5. Engineering Deployment: Key Considerations from Training to Serving

5.1 Training Side

• Load balancing: Router collapse (tokens concentrating on a few experts) is the primary risk, commonly mitigated with auxiliary balance losses, expert capacity factors, and token dropping;
• Training stability: Router and expert parameters update in a coupled way, requiring careful learning rate strategies and numerical stability handling;
• Parallelism strategy: The combined scheduling of expert parallelism, data parallelism, and pipeline parallelism plus all-to-all communication optimization directly determines training throughput for trillion-parameter MoE.

5.2 Inference Side

• Memory vs. activation trade-off: Although total parameters are large, resident memory can be reduced through strategies such as on-demand expert loading; small activated parameters keep per-token compute manageable;
• Batching and concurrency: Different tokens hit different experts, and uneven expert load during batching affects throughput, requiring dynamic scheduling;
• Long-context cost: KV cache grows linearly with context length and is the main cost item in long-text scenarios.

5.3 A General Path for Cloud Deployment

For most teams, building trillion-scale training/inference clusters in-house is unrealistic. A common path:

1. Launch inference services on demand based on cloud GPU compute and container platforms;
2. Use model hosting services to directly call prebuilt LLM capabilities, integrating through general APIs;
3. Manage corpora and artifacts with object storage, and ensure observability with logging services and monitoring alerts;
4. Complete domain adaptation through general means such as prompt engineering, retrieval-augmented generation (RAG), and fine-tuning.

Platforms such as Baidu AI Cloud provide products in the above general categories, and enterprises can select based on data compliance requirements and cost budgets without rebuilding底层 compute infrastructure.

6. Conclusion

The technical map of trillion-parameter MoE models can be summed up in four points: architecturally, “sparse activation + fine-grained experts + shared experts” resolves the tension between scale and cost; on the input side, large vocabularies and ultra-long context improve encoding efficiency and task coverage; in paradigm, the shift is from “static response” to the “think—act—think again” agentic loop; in engineering, load balancing, parallel scheduling, and cloud hosting capabilities support deployment.

For developers, the point of understanding this design logic is not to replicate a specific model, but to grasp the general methodology of “how architecture should evolve as model scale, cost, and task forms change.” Whichever direction future models take, sparsification, long context, and agentic capability will remain technical threads worth sustained investment.