Analyzing Open-Source Ultra-Large MoE Models: Architecture and Local Deployment Practices
This article explores the architecture and local deployment challenges of open-source, ultra-large Mixture of Experts (MoE) language models, specifically focusing on a recent 2.8T parameter model. It breaks down the MoE architecture, multimodal and agent-oriented training, memory and storage requirements, and practical hardware and inference frameworks for local deployment, helping developers make informed infrastructure decisions.
Introduction: The Era of Open-Source Ultra-Large Models
In recent years, open-source large language models (LLMs) with parameter scales exceeding the trillion level have continuously emerged. This marks a significant trend: leading laboratories are no longer keeping their largest flagship models entirely behind APIs but are choosing to open their weights to the community. A recently discussed open-source model is a typical representative: with a total parameter count of 2.8T, a weight file size of approximately 1.56TB, and only about 104B parameters activated per inference, it employs natively multimodal and Agent-oriented training.
The emergence of such models allows researchers and enterprises to access capabilities close to those of closed-source flagships locally. On the other hand, it poses unprecedented challenges to deployment infrastructure. This article will analyze what it truly means to “run a 2.8T model locally” from four dimensions: model architecture, memory estimation, hardware solutions, and inference optimization.
1. How 2.8T Parameters Are Composed: MoE Architecture Breakdown
1.1 The Basic Idea of Sparse Activation
This model utilizes the Mixture of Experts (MoE) architecture. Unlike dense models where every token passes through all parameters, MoE replaces the feed-forward network layers with a large number of parallel “expert” subnetworks and introduces a Router module to dynamically select a small number of experts for each token.
Specifically, the key structural parameters of this model include:
- 896 routing experts per layer;
- Only 16 experts selected per token for computation;
- 2 additional shared experts that are constantly active for all tokens, used to carry general knowledge;
- Total parameters of about 2.8T, with approximately 104B activated parameters per token.
This means the activation ratio is about 3.7%. This “large capacity, small activation” design brings two direct benefits: first, the inference computation (FLOPs) is roughly equivalent to a dense model at the ten-billion level, keeping inference speed controllable; second, the model capacity is massive, allowing different experts to specialize in knowledge across different domains, languages, and task patterns, thereby achieving stronger comprehensive capabilities in multilingual, code, reasoning, and multimodal dimensions.
1.2 Natively Multimodal and Agent-Oriented Training
Unlike patchwork approaches that “first train a language model, then attach a visual encoder,” this model is natively multimodal from the pre-training phase, jointly modeling visual, textual, and other information within the same training framework. The advantage is more natural cross-modal alignment, leading to better consistency in tasks like chart understanding, document parsing, and visual reasoning.
Simultaneously, the model was continuously trained on top of the previous generation base, with a focus on strengthening Agent capabilities, such as long-chain task planning, tool calling, and multi-turn environmental interactions. Such capability improvements typically rely on synthesizing large-scale Agent trajectory data and reinforcement learning, enabling the model to “do things itself” in real workflows rather than just answering questions.
2. What 1.56TB Weights Imply: Memory and Storage Estimation
2.1 The Origin of Weight Size
If 2.8T parameters were stored in FP8 (1 byte per parameter), the theoretical size would be about 2.8TB. The official weights are around 1.56TB, indicating that some parameters use lower-bit quantized storage schemes (e.g., mixed precision or more aggressive quantization formats). This is a common practice for current ultra-large open-source models to lower the barrier for downloading and loading.
2.2 Inference Memory Requirement Estimation
The required inference memory can be roughly estimated using the following formula:
Total Memory ≈ Weight Memory + KV Cache + Activations and Runtime Overhead
- Weight Memory: Even loading quantized weights requires about 1.5~1.6TB of memory (if no further quantization is applied);
- KV Cache: Cannot be ignored in long-context scenarios; calculating with concurrent requests of tens of thousands of tokens may consume an additional hundreds of GB;
- Runtime Overhead: Includes temporary tensors for expert routing, CUDA graph caches, etc., usually reserving about 10%.
Therefore, the conclusion of “8 GPUs” is no coincidence: calculating with high-performance accelerator cards offering around 141GB of memory per card, 8 cards total about 1.1TB, which requires more aggressive quantization (like 4-bit level) and KV cache compression to barely fit. With cards offering 80GB of memory, 8 cards only provide 640GB, necessitating deeper quantization or multi-node parallelism. In other words, the so-called “runnable on 8 GPUs” is a minimum configuration based on high-memory models plus quantization schemes, not a comfortable setup.
3. Hardware and Engineering Solutions for Local Deployment
3.1 Feasible Hardware Paths
For enterprises or research teams, common deployment paths fall into three categories:
- Single-node multi-GPU: Using 8 high-memory accelerator cards combined with Tensor Parallelism (TP), suitable for latency-sensitive, low-concurrency scenarios;
- Multi-node parallelism: Splitting the model across multiple servers via high-speed interconnects, combining Tensor Parallelism and Expert Parallelism (EP) to handle higher throughput, though introducing network communication overhead;
- GPU + CPU hybrid offloading: Offloading cold experts to RAM or even disk, keeping only hot experts in GPU memory. Because MoE is naturally sparse, some inference frameworks support scheduling at the expert granularity, trading latency for feasibility when memory is severely constrained, suitable for offline batch processing.
3.2 Key Capability Requirements for Inference Frameworks
To run such a model, the inference framework needs to support at least:
- Combined partitioning of Tensor Parallelism, Pipeline Parallelism, and Expert Parallelism;
- Efficient de-quantization and matrix multiplication for FP8 or lower precision weights;
- Continuous batching and paged KV Cache management to improve throughput;
- Attention optimization for long contexts (such as sparse attention or sliding window strategies).
Below is an illustrative startup configuration (parameters are generic examples, refer to the actual framework documentation):
serve --model /path/to/weights \--tensor-parallel-size 8 \--enable-expert-parallel \--quantization fp8 \--max-model-len 131072 \--gpu-memory-utilization 0.92
3.3 Balancing Cost and Benefit
The investment in building an 8-card high-memory node (including procurement, power, and maintenance) is substantial. For most teams, a more realistic strategy is tiered decision-making: daily R&D and evaluation can use APIs or cloud-based pay-as-you-go inference services; investing in a self-built cluster is only worthwhile when there are rigid needs for data sovereignty, private compliance, or long-term high-concurrency calls. Alternatively, deploying inference services on container platforms, combined with object storage for weight management and a monitoring/alerting system to ensure availability, can form a scalable private inference platform. For instance, on mainstream cloud platforms like Baidu AI Cloud, such inference environments can be quickly set up using high-performance computing instances and managed container services, avoiding heavy upfront capital expenditures.
4. Conclusion
The architecture of 2.8T total parameters, 104B activated, and 896 experts choosing 16 represents the current engineering pinnacle of open-source models: using sparsity to balance capacity and inference efficiency, and using native multimodal and Agent training to expand capability boundaries. However, “open-source weights” does not mean “runnable by everyone”—the 1.56TB size dictates strict requirements for hardware, inference frameworks, and engineering capabilities. When embracing such models, developers should first clarify their scenarios’ concurrency, latency, and compliance needs, and then make rational choices among local clusters, cloud computing, and API services. For teams capable of implementation, this is not just a model deployment, but a practical exercise in building large-scale inference infrastructure capabilities.
