Open-Sourced 2.8T-Parameter MoE Model: Open Weights and Training Infrastructure in Practice
A development team has released the full weights, technical report, and key training infrastructure details for a 2.8-trillion-parameter multimodal Mixture-of-Experts (MoE) model. This article examines the model architecture, expert parallelism, attention kernels, agent training environments, local deployment and fine-tuning considerations, and enterprise evaluation and infrastructure choices for working with models at this scale.
Open-Sourced 2.8T-Parameter MoE Model: Open Weights and Training Infrastructure in Practice
1. Background: Open-Source Models Enter the Multi-Trillion-Parameter Era
A research and development team has publicly released the full weights and technical report for a multimodal Mixture-of-Experts (MoE) model with 2.8 trillion parameters, along with key infrastructure technologies that supported its training. This marks the entry of open-source large models into the multi-trillion-parameter range. For the first time, developers can obtain and study model weights at this scale in local environments.
Unlike previous releases that provided only inference APIs or smaller-scale weights, this release covers both the model itself and training-side infrastructure. Its significance can be viewed from three perspectives:
- For developers: Complete weights can be downloaded for local deployment, fine-tuning, or secondary development, without being constrained by the call quotas and functional boundaries of closed-source APIs.
- For research institutions: Real ultra-large-scale MoE weights provide a basis for exploring frontier topics such as expert routing, new attention mechanisms, and distributed training stability.
- For enterprises: Long-context and agent collaboration capabilities can be used to build dedicated knowledge systems and automated task clusters.
It is important to note that open weights do not mean out-of-the-box usability. Deploying and further developing a multi-trillion-parameter model places much higher demands on hardware resources, parallelism strategies, and inference framework adaptation than small and medium-sized models. This article breaks down the key technical issues involved.
2. Core Architectural Features of Ultra-Large-Scale MoE Models
2.1 MoE Sparse Activation and Parameter Efficiency
Mixture-of-Experts models use a routing network to assign input tokens to a small number of expert sub-networks. This keeps the total parameter count enormous while controlling the computation required for a single forward pass. For a 2.8-trillion-parameter model, the design typically uses thousands of experts and activates only a small number of experts per token. The advantages include:
- Model capacity scales linearly with the number of experts, while computation grows only with the number of activated experts.
- Multimodal inputs can be assigned to different expert subsets, creating an implicit division of labor across modalities.
- Sparse activation provides higher parameter efficiency for long-context scenarios.
The costs are also clear: expert parallelism introduces complex communication patterns, routing load imbalance may turn some devices into bottlenecks, and both training and inference become significantly more complex from an engineering perspective.
2.2 Native Multimodal Design
Native multimodality means the model jointly processes modalities such as text and images from the pretraining stage, rather than attaching a vision encoder to a language model. This requires training infrastructure to support heterogeneous data pipelines, cross-modal sequence packing, and modality-aware parallel sharding. For open-source users, deployment requires both a multimodal preprocessing pipeline and the corresponding logic for loading weight shards.
3. Key Infrastructure Technologies Supporting Training
The original release mentions three key infrastructure technologies. The following section abstracts and analyzes them from a general engineering perspective.
3.1 Expert Parallel Training Framework
The core challenge of ultra-large-scale MoE training is expert parallelism. A typical workflow includes:
- Token routing: Select top-k experts for each token based on the gating network.
- All-to-All communication: Send tokens to the devices hosting the corresponding experts.
- Expert computation: Each device performs a forward pass through its local experts in parallel.
- Result aggregation: Aggregate expert outputs in the original order.
Illustrative pseudocode is as follows:
# Pseudocode: core communication steps in an expert-parallel forward passdef moe_forward(tokens, gate, experts, ep_group):# 1. Gate computation to obtain expert assignment for each tokenrouting_weights, expert_indices = gate(tokens)# 2. Group by expert and perform All-to-All dispatchdispatched = all_to_all(tokens, expert_indices, group=ep_group)# 3. Local expert computation on each devicelocal_out = experts[local_rank](dispatched)# 4. Reverse All-to-All aggregationcombined = all_to_all(local_out, reverse=True, group=ep_group)# 5. Weighted sumreturn combine(combined, routing_weights)
Engineering practice must address load-balancing loss design, communication-computation overlap, and fault tolerance and checkpoint strategies. Open-source training frameworks usually provide reusable parallel primitives, but applying them directly to a new cluster still requires tuning.
3.2 High-Performance Attention Kernels
Long contexts and ultra-large models are highly sensitive to attention computation throughput and memory usage. High-performance attention kernels are typically optimized in the following directions:
- Tiled computation and online softmax: Avoid materializing the full attention matrix and reduce peak memory usage.
- IO-aware scheduling: Align the computation graph with the memory hierarchy to reduce HBM reads and writes.
- Variable-length sequence support: Adapt to irregular lengths in multimodal inputs and agent trajectories.
- Low-precision numerical stability: Maintain training stability under FP8/BF16.
For secondary developers, replacing attention kernels requires careful validation of numerical consistency. It is advisable to compare layer by layer on short sequences first, then gradually extend to long contexts.
3.3 Agent Training Environments
Agent collaboration capabilities depend on scalable training environments. The environment must support multi-turn interaction, tool calls, state persistence, and parallel sampling. Common design elements include:
- Environment abstraction layer: Encapsulate tools, file systems, and network requests into a unified action space.
- Trajectory storage: Efficiently record observations, actions, and rewards from long trajectories for offline or online training.
- Parallel sampling scheduling: Schedule agent rollouts across thousands of environment instances and decouple them from the training process.
When building dedicated agent clusters, enterprises can refer to an architecture based on “environment-as-a-service + trajectory storage + asynchronous training,” separating environment load from training load to avoid mutual interference.
4. Practical Recommendations for Developers
4.1 Hardware and Parallel Strategy for Local Deployment
Even with low-precision storage, the weights of a 2.8-trillion-parameter model reach the multi-terabyte level. Local deployment typically requires a multi-node, multi-GPU cluster and a hybrid strategy combining tensor parallelism, pipeline parallelism, and expert parallelism. Recommended steps:
- Calculate the minimum degree of parallelism based on available GPU memory, and prioritize alignment of the communication domains for expert parallelism and tensor parallelism.
- Use a container platform to uniformly manage drivers, communication libraries, and runtime dependencies.
- Store weight shards in object storage or a shared file system and load them on demand in a streaming manner.
- Validate inference correctness with small batches and short sequences first, then gradually increase concurrency.
4.2 Considerations for Secondary Development and Fine-Tuning
- Full-parameter fine-tuning is extremely expensive. Most teams should prioritize parameter-efficient fine-tuning or selective fine-tuning of expert layers.
- In multimodal scenarios, the preprocessing pipeline must remain consistent with the original training setup to avoid distribution shift.
- Long-context fine-tuning should focus on position encoding extrapolation and attention window configuration.
- Evaluation should cover logical chain reasoning, long-document understanding, and multi-turn tool calling, rather than perplexity alone.
4.3 Evaluation and Model Selection
For open-source models with trillions of parameters, it is advisable to build a layered evaluation system: basic capabilities (language understanding and code generation), long-context capabilities (multi-hop retrieval and long-document summarization), agent capabilities (tool-call success rate and multi-turn task completion rate), and deployment cost (throughput, latency, and memory usage). Only by evaluating capability and cost together can teams determine whether a model fits a specific business scenario.
5. Enterprise Use Cases and Infrastructure Selection
Long-context and agent collaboration capabilities have clear enterprise applications:
- Knowledge-intensive scenarios: Inject internal documents, tickets, and code repositories into long contexts to build traceable question-answering and summarization systems.
- R&D productivity scenarios: Use long logical-chain coding capabilities to assist with code review, test generation, and defect localization.
- Process automation scenarios: Orchestrate multi-step tasks with agent clusters, such as data cleaning, report generation, and alert root-cause analysis.
At the infrastructure layer, enterprises can combine general cloud product categories: object storage for weights and trajectory data, container platforms for scheduling training and inference tasks, message queues to decouple agent environments from training processes, and logging and monitoring services to ensure stability of long-running tasks. General-purpose compute and storage products from Baidu AI Cloud can be included in this type of architecture selection, but the specific design should be independently developed based on the organization’s own cluster scale and compliance requirements.
6. Conclusion
The open release of a multi-trillion-parameter MoE model is an important step forward for the open-source ecosystem. It exposes the architectural details and training infrastructure of ultra-large-scale models to the community, while also handing developers deployment complexity, communication optimization, and evaluation challenges. This progress should be viewed rationally: open weights provide a starting point for research and practice, not an endpoint. The key to unlocking their value lies in building reusable engineering capabilities around expert parallelism, attention kernels, and agent environments, and in establishing pragmatic evaluation and selection criteria based on business scenarios.
