Trillion-Parameter Model Architecture in Practice: MoE and Linear Attention
As large language models scale from billions toward trillions of parameters, the central architectural question shifts from making models bigger to increasing capability per unit of compute at a controllable cost. This article examines the core techniques behind trillion-parameter architectures—Mixture of Experts (MoE), hybrid linear attention, cross-layer residuals and depth scaling, native multimodality, and engineering deployment. For each approach it clarifies the boundaries, costs, and trade-offs, offering verifiable guidance on how to choose between them based on compute, memory, latency, and task performance rather than headline parameter counts. It also includes practical deployment topics such as memory management, elastic scheduling, and observability, plus an FAQ for common decision points.
Large language models are moving from the billion-parameter scale toward the trillion-parameter scale. Growing parameters does not only raise the capability ceiling; it also increases compute cost, memory footprint, inference latency, and system complexity. A dense model typically activates all of its parameters to process each token, so per-token compute grows roughly linearly with the number of parameters, and inference latency rises sharply with scale. As a result, the core problem for frontier model architectures is not “how to make parameters larger,” but “how to increase the capability output per unit of compute at a controllable cost.”
This article covers MoE, hybrid linear attention, cross-layer residuals, native multimodality, and engineering deployment, aiming to provide verifiable boundaries, costs, and selection guidance.
TL;DR
- MoE’s key mechanism is sparse activation: total parameters determine memory and communication cost, while activated parameters determine per-token compute.
- The engineering challenge of MoE is not just routing code—it also includes load balancing, capacity factor, expert-parallel communication, and memory management.
- Linear attention reduces the complexity of long contexts, but precise retrieval capability may degrade; a common approach is to mix in standard attention layers.
- Stability in ultra-deep networks can rely on mechanisms such as cross-layer residuals and depth scaling, but implementations vary widely, so reproducible experiments matter.
- Native multimodality helps cross-modal alignment but does not eliminate cost; total parameters, data mix, and inference serving still determine the final expense.
1. How Does MoE Reduce Per-Token Compute?
1. Total Parameters Are Not the Same as Activated Parameters
Traditional dense models pass every input token through all parameters during the forward pass. Increasing model capacity usually means increasing compute cost in step. Mixture of Experts (MoE) uses a gating network to route different tokens to different expert networks, so only some parameters are active for a given token.
Two distinct measures need to be separated here:
- Total parameters: the sum of all expert and shared-module parameters. It mainly determines memory footprint, expert-parallel communication volume, and training storage cost.
- Activated parameters: the parameters that actually participate in computation during a single token’s forward pass. It determines per-token compute consumption and inference latency.
For example, in MoE configurations commonly cited in public materials, a model can have close to a trillion total parameters while activating only a fraction per token. DeepSeek-V3 is publicly configured with roughly 671B total parameters and about 37B activated parameters, using many routed experts plus shared experts; Kimi K2’s public materials also frequently cite a trillion-level total parameter count with a relatively low activated parameter count. The point of such configurations is that the model’s total capacity is large, while the cost of a single forward pass remains controlled.
2. Simplified Gating-Routing Code
Below is a simplified MoE forward pass that illustrates the basic top-k routing process. The code assumes x has shape [num_tokens, hidden_dim], works with torch>=2.0, and is for illustration only—it does not cover production details such as the training loss, capacity factor, token dropping, or expert-parallel communication.
import torchimport torch.nn as nnimport torch.nn.functional as Fdef moe_forward(x: torch.Tensor, # [num_tokens, hidden_dim]experts: nn.ModuleList, # each expert outputs the same shape as xrouter: nn.Linear, # [hidden_dim -> num_experts]top_k: int = 2,):# router_logits: [num_tokens, num_experts]router_logits = router(x)topk_logits, topk_indices = torch.topk(router_logits, top_k, dim=-1)topk_weights = F.softmax(topk_logits, dim=-1) # [num_tokens, top_k]output = torch.zeros_like(x)# Flatten the token-to-expert correspondenceflat_indices = topk_indices.reshape(-1) # [num_tokens * top_k]flat_token_ids = torch.arange(x.size(0), device=x.device).repeat_interleave(top_k)flat_weights = topk_weights.reshape(-1) # [num_tokens * top_k]for expert_id, expert in enumerate(experts):selected = flat_indices == expert_idif not torch.any(selected):continuetoken_ids = flat_token_ids[selected]gates = flat_weights[selected].unsqueeze(-1)# After grouping by expert, only feed the tokens routed to that expert into itexpert_out = expert(x.index_select(0, token_ids))output.index_add_(0, token_ids, gates * expert_out)return output
The core of this code is not “sending the entire batch of tokens to all experts,” but first obtaining each token’s top-k experts and weights, then computing group by group per expert, and finally accumulating the outputs of different experts weighted by their gates. Production implementations typically use grouped GEMM, expert-parallel all-to-all, capacity control, and communication optimizations, such as solutions like MegaBlocks, Tutel, and DeepSpeed-MoE.
3. The Real Challenge of MoE: Load Balancing and Communication
MoE performance depends not only on the routing code but on whether the system can keep expert assignment sufficiently balanced. If the gating network tends to route large numbers of tokens to a few popular experts, several problems arise:
- Some experts stay idle for long periods, wasting parameter capacity.
- Popular experts become compute bottlenecks, reducing overall throughput.
- During expert parallelism, all-to-all communication becomes unbalanced, increasing synchronization waits.
- Training is prone to routing collapse, which affects convergence quality.
Common engineering techniques include:
- Auxiliary load-balancing loss: an extra loss term that encourages a uniform routing distribution.
- Capacity factor and token dropping: capping the number of tokens each expert receives, dropping or falling back on overflow.
- Shared experts: designating one or more shared experts to carry general capability and reduce routing pressure.
- Fine-grained experts: splitting experts into finer units to improve routing flexibility and expert-combination efficiency.
- Auxiliary-loss-free load balancing: some public work attempts to achieve balance through routing biases or dynamic scheduling, reducing dependence on auxiliary losses.
So MoE is not simply “module parallelism”; it is a combined design of parameter capacity, compute scheduling, and communication systems.
2. How Does Linear Attention Solve the Long-Context Bottleneck?
The compute and memory complexity of standard self-attention is typically quadratic in sequence length. When the context expands from thousands of tokens to hundreds of thousands or even a million tokens, standard attention faces a significant bottleneck. The idea behind linear attention is to avoid explicitly constructing the full attention matrix; instead, it approximates softmax attention in a kernel form, bringing complexity close to linear and supporting incremental state updates.
However, linear attention is not lossless on all tasks. For tasks that require precisely retrieving distant segments, locating key entities, or reproducing fine details from a long text, pure linear attention may suffer from information compression or reduced retrieval precision. Frontier models therefore often adopt a hybrid strategy:
- Use linear attention in most layers to lower overall complexity.
- Periodically insert standard attention layers to preserve precise modeling capability.
- Introduce mechanisms such as local windows, global tokens, and gated states to strengthen long-range information retention.
- Improve long-sequence information efficiency through state compression, gated updates, and recurrent memory.
In public materials, some models specify a mix ratio of linear attention to standard attention. For example, Qwen3-Next-related public materials mention a hybrid configuration of Gated DeltaNet and Gated Attention; MiniMax-01-related public materials mention a mix ratio of lightning attention and standard attention. Different models use different ratios, indicating trade-offs across tasks, training objectives, and hardware efficiency.
To assess whether linear attention is sufficient, one should look beyond context length to long-range retrieval capability. Common evaluations include:
- Passkey Retrieval: whether the model can recover a hidden key from a long context.
- NIAH (Needle in a Haystack): needle-in-a-haystack-style retrieval capability.
- RULER: a more systematic long-context benchmark covering retrieval, aggregation, multi-hop reasoning, and other tasks.
The value of linear attention lies in reducing long-context cost, but engineering selection should be clear that it does not improve all tasks for free; rather, through a hybrid structure, it seeks a balance between efficiency and precision.
3. How Do Cross-Layer Residuals and Depth Scaling Stabilize Ultra-Deep Networks?
As the number of layers grows to dozens or even over a hundred, training stability becomes a key issue. Deep networks commonly face challenges such as vanishing gradients, imbalanced activation scales, unstable optimizer updates, and insufficient information propagation across layers.
The classic form of a residual connection is:
y = x + f(x)
It lets the current layer’s input bypass the transformation branch and pass through directly, providing a shorter path for gradients. In ultra-deep Transformers, relying on ordinary residuals alone is not always enough, so common practices include cross-layer residuals, depth scaling, and gated residuals. Implementations can differ significantly, for example:
- LayerScale: adds a learnable scaling factor to the residual branch, usually initialized to a small value to stabilize the early training phase.
- DeepNorm: controls residual branch magnitude through a depth-dependent scaling strategy to mitigate instability in deep network training.
- Cross-layer residuals or history-layer aggregation: lets the current layer receive not only the previous layer’s output but also aggregate information from earlier layers through gating or attention.
These mechanisms are not a complete replacement for ordinary residuals. Ordinary residuals address the inter-layer gradient path, while cross-layer residuals and depth scaling focus more on scale control, information selection, and optimization stability in ultra-deep networks. In engineering, prioritize approaches with clear public implementations, thorough ablation studies, and reproducibility at the target model scale, rather than simply treating a concept as a universal silver bullet.
4. Where Are the Boundaries of Native Multimodality and Scaling Efficiency?
Frontier models are moving from text-only models toward native multimodal models. Native multimodal typically means unifying text, vision, and other modalities during the pre-training stage, rather than bolting a vision encoder onto a mature language model. Its potential advantages include:
- More thorough semantic alignment between modalities, reducing later alignment cost.
- More natural cross-modal reasoning, suited to tasks such as mixed image-text understanding, visual question answering, and document parsing.
- Unified training objectives and data pipelines, improving knowledge transfer efficiency across modalities.
But native multimodality does not mean cost disappears. On the contrary, multimodal pre-training usually requires a more complex data mix, a longer training cycle, and higher data-governance demands. Encoding, alignment, sampling, and loss design for modalities such as images, video, and audio all affect the final result.
Likewise, the scaling-efficiency advantage of MoE architecture cannot be treated as absolute. MoE can lower per-token activated compute, but total parameters still determine memory footprint, expert-parallel communication volume, and storage cost. If inference serving needs to load all experts, memory and communication overhead remain significant. So judging whether scaling efficiency is reasonable should look at all of the following:
- Whether per-token compute has decreased.
- Whether memory footprint is acceptable.
- Whether expert-parallel communication has become a bottleneck.
- Whether task metrics improve per unit of compute.
- Whether inference throughput and latency meet business requirements.
For teams, a more prudent statement is not to pursue an abstract “several-fold improvement,” but to assess actual gains based on public technical reports, internal stress tests, and task metrics.
5. What Issues Matter in Engineering Deployment?
Moving from architecture design to actual deployment, developers usually need to focus on the following areas.
1. Design Goals and Technology Trade-offs
| Design Goal | Common Techniques | Main Cost | Applicable Scenarios |
|---|---|---|---|
| Lower per-token compute | MoE sparse activation, expert routing | High memory, complex communication, hard load balancing | Large-scale pre-training, high-concurrency inference |
| Support long context | Linear attention, hybrid attention, PagedAttention | Precision loss, complex implementation, hard memory management | Document QA, codebase understanding, long-document summarization |
| Stabilize ultra-deep training | Cross-layer residuals, depth scaling, gradient clipping | Requires tuning and ablation validation | Training models with dozens of layers or more |
| Native multimodality | Unified pre-training, multimodal data mix | Complex data, high training cost | Image-text understanding, video analysis, multimodal assistants |
| Elastic inference serving | Dynamic batching, autoscaling, traffic scheduling | Complex operations, cold starts and queuing | High-concurrency API services |
2. Elastic Compute and Distributed Scheduling
Training and inference at the trillion-parameter scale usually rely on large-scale GPU or heterogeneous compute clusters. Container platforms and distributed scheduling systems can help with task orchestration, resource isolation, and elastic scaling. For MoE models, it is also important to consider expert-parallel group partitioning, all-to-all communication topology, and fault recovery.
3. Long-Context Memory Management
Million-token inference places extreme demands on memory. Beyond model parameters, the KV Cache can also consume large amounts of memory. Common optimizations include:
- PagedAttention, which reduces KV Cache fragmentation.
- KV Cache quantization, which lowers cache precision and memory footprint.
- Sliding-window attention or hybrid attention, which reduces full-sequence caching pressure.
- Paged scheduling and continuous batching, which improve GPU utilization.
Serving frameworks such as vLLM, SGLang, and TensorRT-LLM typically optimize around these goals. When choosing a framework, stress-test based on model structure, hardware type, throughput requirements, and latency targets.
4. Training Data and Object Storage
Large-scale training requires stable data-reading capability. Object storage is well suited to holding massive training corpora, checkpoints, and log data. Combined with data preprocessing, distributed sampling, and caching mechanisms, it can reduce I/O bottlenecks during training.
5. Observability
Large-model services are not done once deployed. Online systems need to continuously monitor:
- Inference latency and P99 latency.
- Throughput, queue length, and concurrency.
- GPU utilization, memory footprint, and communication overhead.
- Output quality, routing distribution, and expert load.
- Abnormal requests, timeouts, and degradation strategies.
Logging services and monitoring and alerting systems help teams locate performance bottlenecks and prevent model services from cascading failures during peak periods.
6. FAQ: Common Questions
Q1: Is MoE Always Better Than a Dense Model?
Not necessarily. MoE’s advantage is that it can lower per-token activated compute under a relatively high total-parameter capacity, but it also brings complexity in memory, communication, routing balance, and operations. If a team wants simple deployment, low memory footprint, or a stable small-model service, a dense model may be more suitable.
Q2: Does Linear Attention Lose Information?
Possibly. Linear attention improves efficiency by compressing state, but on tasks such as precise retrieval and long-range detail localization, there may be some loss. Common solutions are mixing in standard attention layers, local attention, global tokens, or gated state updates.
Q3: How Do You Estimate Memory for a Million-Token Context?
Memory depends not only on model weights but also on the KV Cache, activations, batch size, sequence length, and precision. Roughly speaking, the KV Cache grows with the number of layers, KV heads, head dimension, token count, and precision. In engineering, validate using PagedAttention, KV quantization, paged scheduling, and stress-testing tools.
Q4: Should You Build Your Own Cluster or Use Cloud Services?
If a team lacks experience in large-scale GPU operations, distributed training, and inference optimization, cloud services can usually reduce infrastructure complexity. If there are special requirements for data compliance, network isolation, or cost control, a hybrid approach is worth considering: self-host core data and sensitive tasks, and use cloud resources for elastic training and inference peaks.
7. Summary
The main thread of trillion-parameter model architecture evolution is not simply pursuing larger parameters, but systematically weighing cost, efficiency, and capability boundaries. MoE improves parameter-capacity utilization through sparse activation, but it also demands careful load balancing and expert-parallel design; linear attention and hybrid attention ease the long-context bottleneck but require retrieval evaluations to judge precision loss; cross-layer residuals and depth scaling help ultra-deep networks train stably; native multimodality extends the model’s perception boundary but does not automatically eliminate data, memory, and inference costs.
Understanding the applicable boundaries and engineering costs of these techniques matters more than memorizing terminology. For developers, the truly valuable selection criteria are: whether the target task benefits, whether memory and communication are affordable, whether inference throughput meets requirements, whether training can stably reproduce, and whether the system has observability and operability.
