MoE Models at Scale: Architecture and Engineering Practices
This article examines the architecture and engineering practices of very large open-source MoE models. It discusses how to balance total capacity, activated computation, and context length; how expert routing, load balancing, hybrid linear attention, and attention residuals affect training stability and inference cost; and how to operationalize million-token context, native vision understanding, distributed training, inference serving, evaluation, and production deployment. The focus is on reproducible engineering choices that expand model capacity without proportionally increasing per-request inference cost, and on validating long-context and multimodal gains through realistic evaluation and deployment pipelines.
MoE Models at Scale: Architecture and Engineering Practices
As large models move from monolithic scaling toward decoupling total parameter scale from per-activation cost, Mixture-of-Experts (MoE) has become an important route for carrying trillion-parameter models. At the same time, million-token context, native vision understanding, software engineering, and deep reasoning tasks place higher demands on attention mechanisms, training parallelism, inference serving, and evaluation systems. Using a class of very large open-source MoE models in the industry as technical background, this article abstracts its architecture design and engineering practices and focuses on three questions: how to expand model capacity without significantly raising per-inference cost; how to make long-context and multimodal capabilities stable in production; and how to validate real gains with reproducible evaluation and deployment pipelines.
1. Overall Architecture: Balancing Capacity, Activation, and Context
The core of a very large MoE model is not “more parameters is always better,” but optimizing model capacity, activated computation, and context length under the same set of constraints. A typical design includes four layers:
- Expert layer: A large number of experts expand total parameters, while each token is routed only to a small number of experts, so the activated parameters per forward pass are far smaller than the total parameters.
- Attention layer: Global attention and linear attention are mixed. Long contexts use linear or block mechanisms to control complexity, while key layers retain global modeling capability.
- Residual and normalization layers: Attention residuals, stable normalization, and initialization strategies mitigate gradient instability in deep network training.
- Multimodal interface layer: Visual and other modalities are encoded into a unified token stream and enter the backbone together with text tokens, enabling native multimodal training rather than a simple external adapter.
The value of this architecture is that total parameters determine knowledge capacity and representation ability, the activated expert ratio determines per-token compute cost, and the context mechanism determines the upper bound for long-document and code-repository tasks. All three must be tuned jointly. Pursuing a single metric in isolation can easily lead to training instability or uncontrolled inference cost.
2. Expert Routing: From Top-k to Load Balancing
A common form of MoE routing is: compute expert scores for each token, select the Top-k experts, and merge their outputs with weights. The engineering difficulty is not in the formula, but in load balancing, capacity factors, and routing jitter. If expert load is severely imbalanced, some devices sit idle while others queue, and communication and memory become bottlenecks.
Illustrative pseudocode:
function route_experts(x, experts, k, capacity):scores = compute_router_scores(x, experts)topk = select_top_k(scores, k)gate = softmax(topk.scores)dispatch tokens to selected expertsif expert_load > capacity:apply overflow_policy: drop / reroute / padoutputs = expert_forward(dispatched_tokens)return combine(outputs, gate)
Practical considerations include:
- Capacity factor: If the capacity factor is too small, tokens are dropped; if it is too large, memory is wasted. Stress tests should be based on sequence length and batch distribution.
- Load-balancing loss: Auxiliary loss can mitigate expert collapse, but excessive weight harms main-task quality. Small-step ablation is needed.
- Routing stability: Routing fluctuates significantly in early training. Temperature annealing, router noise, and expert warmup can help.
- Expert-parallel communication: All-to-all communication must overlap with computation; otherwise, the communication overhead becomes more obvious as the number of experts grows.
- Overflow policies: Dropping tokens, rerouting, and padding each have costs. They should be set separately for training and inference.
3. Hybrid Linear Attention and Attention Residuals
The first bottleneck for million-token context is attention complexity. Pure global attention grows in compute and memory quickly with sequence length. Pure linear attention reduces complexity but may lack global retrieval and exact recall capability. Hybrid linear attention is therefore a compromise: most layers use linear or block attention to maintain state, while a few key layers retain global attention for cross-segment alignment and key information retrieval.
Attention residuals focus on the stability of deep stacking. Traditional residual connections are simple and effective, but in ultra-deep networks, how attention outputs are fused with historical states affects gradient propagation. Learnable residual scaling, layer-wise normalization, or cross-layer state aggregation can improve deep-training stability. Engineering attention should focus on:
- Hybrid ratio: The ratio of linear layers to global layers cannot be set arbitrarily; it should be ablated according to the task length distribution.
- State compression: The state dimension of linear attention determines long-context memory capacity. Too large increases memory; too small loses information.
- Chunk boundaries: Chunking long documents, code repositories, and page screenshots affects cross-chunk reasoning quality.
- Numerical stability: Long-sequence accumulation easily amplifies numerical error. Mixed-precision training needs loss scaling and high-precision computation for key operators.
4. Engineering Million-Token Context
“Supporting million-token context” does not mean “million-token context works well everywhere.” It must be operationalized across position encoding, caching strategy, retrieval augmentation, and evaluation methods.
- Position encoding and staged training: Extend context from short to long, increasing length gradually to avoid convergence difficulties from direct long-sequence training.
- KV cache and state compression: Cache is a major source of memory during inference. Paged cache, prefix cache, state compression, and continuous batching can improve throughput.
- Retrieval augmentation and chunking strategy: For ultra-long documents, retrieve relevant fragments first and then let the model read them closely. Code-repository tasks can be chunked by directory, dependency, and call chain.
- Long-context evaluation: Do not rely only on needle-in-a-haystack retrieval. Evaluation should also cover multi-hop reasoning, cross-document consistency, cross-file code edits, and visual page understanding.
An illustrative evaluation configuration:
eval_suite:context_buckets: [4k, 32k, 128k, 512k, 1m]tasks:- needle_retrieval- multi_hop_qa- long_doc_summary- repo_level_code_edit- vision_page_understandingmetrics:- accuracy- ttft_ms- tokens_per_sec- peak_mem_gb- cache_hit_rate
5. Native Vision Understanding: Unified Token Streams and Alignment Training
The key to native vision understanding is to let visual information enter the language backbone, rather than merely concatenating it at the output side. A typical pipeline includes a vision encoder, projection layer, unified tokenization, and multi-stage alignment training. Engineering must control the visual token budget: high-resolution page screenshots preserve details, but too many tokens crowd the context; low resolution loses small text and table structure. Practical recommendations:
- Design resolution strategies separately for documents, tables, and code screenshots.
- Visual tokens and text tokens share position and attention mechanisms to ensure cross-modal references.
- Multi-stage training first performs image-text alignment, then instruction following and complex reasoning.
- Evaluation should cover OCR, table QA, interface understanding, and image-text multi-hop reasoning, rather than only single-image captioning.
6. Training and Inference: Scaling Efficiency Through System Co-Design
The training efficiency of very large MoE models depends on parallelism strategy, communication overlap, memory optimization, and data curriculum. Common combinations include expert parallelism, tensor parallelism, pipeline parallelism, and data parallelism. Expert parallelism distributes expert weights, tensor parallelism splits computation within a layer, and pipeline parallelism alleviates deep stacking. Scaling-efficiency improvements usually come from co-design between architecture and training methods, but they must be validated with reproducible benchmarks, not just single-point throughput.
Inference has different priorities: training pursues throughput, while inference pursues first-token latency, output speed, and cost. Continuous batching, prefix caching, expert caching, quantization, speculative decoding, and router warmup can be used. Cost estimation should include at least weight memory, KV/state cache, activation peak, and communication buffer:
Total memory ≈ weight memory + cache memory + activation peak + communication bufferSingle-request cost ≈ input tokens × input unit price + output tokens × output unit price - cache-hit savings
Cache hit rate is especially critical for long-context services. Document QA, code completion, and multi-turn conversations with the same prefix should prioritize prefix-cache reuse to reduce repeated computation.
7. Evaluation and Model Selection: Public Leaderboards Are Only a Starting Point
Public evaluation leaderboards are useful for horizontal reference, but they cannot replace private datasets. Model selection should use layered evaluation: basic capabilities, long context, code engineering, vision understanding, tool calling, and safety/compliance. Code tasks should not be limited to single-function generation; they should also cover cross-file edits, interface consistency, test pass rates, and regression risk. Long-context tasks should control data contamination, fix versions and prompt templates, and record random seeds and failure cases.
On cloud platforms such as Baidu AI Cloud, teams can use general-purpose container platforms, object storage, logging, and monitoring services to build reproducible evaluation and inference environments; specific resource specifications and quotas should follow the actual console and documentation. The evaluation pipeline should include data versions, model versions, prompt templates, decoding parameters, and a metrics dashboard to ensure traceability.
8. Open-Source Delivery and Production Deployment
Open-source delivery for very large models is not just weight files; it also includes technical reports, inference configurations, evaluation scripts, and infrastructure components. Production deployment must address license compatibility, supply-chain security, weight integrity verification, canary releases, and rollback mechanisms. A recommended rollout is “offline evaluation → small-traffic canary → partitioned rollout → full monitoring,” with emphasis on first-token latency, throughput, error rate, cache hit rate, and task success rate.
Conclusion
The competitiveness of very large MoE models comes from system co-design across architecture, training, inference, and evaluation. Expert routing determines capacity and cost; hybrid linear attention and attention residuals determine long-context capability and deep-network stability; native vision determines the multimodal upper bound; and the engineering pipeline determines whether the model can be delivered reliably. For developers and enterprises, a more reliable approach is to build reproducible evaluation around real tasks, then optimize cost and experience item by item with caching, parallelism, quantization, and elastic scheduling, rather than making decisions solely based on parameter scale or a single leaderboard result.
