Ultra-Large Sparse Multimodal MoE Models: Open Weights and Local Deployment
This article explains how to interpret ultra-large sparse multimodal MoE models with roughly 2.8 trillion total parameters, about 1.56 TB of weights, and roughly 104 billion activated parameters per token. It covers the distinction between total and activated parameters, the 896-expert routing design with 16 selected experts and 2 shared experts, why local deployment usually requires multiple GPUs, and the tradeoffs among parallelism, quantization, offloading, and runtime memory. It also provides a practical checklist for evaluating whether such a model can be loaded, run, and used in production.
1. Distinguish Total Parameters from Activated Parameters
A recently published class of ultra-large models uses a Mixture of Experts (MoE) architecture. Its total parameter count is about 2.8 trillion, and the weight files are about 1.56 TB. However, not all parameters participate in computation for every input token. The number of parameters actually activated in a single forward pass is about 104 billion.
These numbers refer to different concepts:
- Total parameters: All trainable parameters stored in the model, including all expert networks, shared modules, attention layers, embedding layers, and output layers.
- Activated parameters: The parameters that actually participate in the forward computation for a given token based on the routing result.
- Weight footprint: The disk space occupied by model parameters after they are saved in a specific data type. This is not the same as VRAM requirements.
- Inference VRAM: In addition to model weights, this includes runtime caches, temporary computation tensors, communication buffers, and framework overhead.
Therefore, it is incorrect to assume that because only 104 billion parameters are used per token, only 104 billion parameters need to be stored. An MoE model still needs access to all expert weights unless expert offloading, layer-wise loading, or other sparse storage mechanisms are used.
2. How the 2.8-Trillion-Parameter Model Is Structured
According to public descriptions, the model contains 896 experts. Each token selects 16 experts through a routing mechanism, and there are also 2 shared experts.
A MoE layer can be abstracted as follows:
Input hidden states│▼Router network computes expert scores│▼Select the top 16 experts│├── Expert 1├── Expert 2├── ...└── Expert 16│▼Weighted fusion by router weights│▼Merge with shared-expert outputs│▼Output new hidden states
Having 896 experts does not mean that every token passes through 896 complete feed-forward networks. The router scores experts based on the hidden representation of the current token and then selects a limited number of experts. This allows the model to expand capacity while controlling the computation per token.
Shared experts handle relatively stable base transformations. Regular experts provide specialized capabilities for different domains, semantic patterns, or task distributions, while shared experts provide a common base representation transformation for all tokens. This design helps reduce instability caused by overly dispersed routing and allows different experts to retain some general capability.
It is important to note that the number of experts, the number of experts selected per token, and the number of shared experts only describe the architectural organization. They cannot by themselves be used to infer accuracy, throughput, or actual model quality. Model quality is also affected by training data, training objectives, routing策略, parallelism methods, context length, and inference implementation. [Note: “routing策略” should be “routing strategy.”]
3. Why the Weight Footprint Is About 1.56 TB
Model parameters are usually stored as floating-point numbers or in low-precision formats. Assuming a total of 2.8 trillion weights and a low-precision format of about half a byte per parameter, the theoretical size is approximately:
2.8 trillion × 0.5 bytes ≈ 1.4 TB
Actual files usually also contain tensor sharding information, metadata, alignment space, and a small amount of additional structure, so a final size of about 1.56 TB is not difficult to understand.
However, weight file size is not equal to the minimum VRAM requirement during deployment. The main reasons include:
Runtime caches
Autoregressive generation requires a key-value cache, or KV Cache. The longer the context and the more concurrent requests, the higher the cache usage.Temporary tensors and intermediate results
Matrix multiplication, routing computation, expert aggregation, and communication all produce temporary buffers.Data type conversion
Some weights may be stored in one format but need to be converted to another format for computation, so multiple copies of data may exist at the same time for a short period.Framework and communication overhead
Multi-GPU execution requires communication buffers, VRAM management space, and additional memory for operator execution.Sharded loading strategies
Some deployment frameworks can load weights in batches or by expert, but this introduces scheduling latency and storage access pressure.
Therefore, 1.56 TB is better used as a reference for “weight storage scale,” not as a number to mechanically match against total GPU VRAM.
4. Why Local Deployment Usually Requires Multiple GPUs
If a single accelerator has tens of GB of available VRAM, even a rough estimate based on weight size alone requires many devices to host the model. For example, with about 200 GB of available VRAM per GPU, at least 8 GPUs are needed just to hold 1.56 TB of weights, but this is only a theoretical lower bound for capacity.
In actual deployment, space should also be reserved for:
- KV Cache;
- Expert routing and aggregation caches;
- Runtime temporary tensors;
- Communication buffers;
- System and driver overhead;
- Additional memory demand from concurrent users.
Therefore, “8 GPUs” is better understood as a starting reference for a specific hardware configuration, not a fixed answer that applies to all environments. If each GPU has less VRAM, more devices may be needed. If more aggressive quantization or weight offloading is used, the VRAM threshold may be lowered, but the cost is usually increased precision loss, reduced speed, or greater engineering complexity.
Before deployment, at least the following formula should be calculated:
Required VRAM≈ weight VRAM+ KV Cache+ intermediate tensors+ communication buffers+ safety margin
The safety margin should not be fully compressed. When VRAM is exactly full, the model is prone to allocation failures during long-context processing, larger batches, or request bursts.
5. Computational Advantages and Engineering Challenges of MoE
The core advantage of MoE is that “parameter capacity” and “per-token computation” can be partially decoupled. Compared with having all parameters participate in every computation, sparse activation can reduce part of the computation in a single forward pass while maintaining a large model capacity.
However, it is not simply “fewer parameters, faster speed.” There are several clear engineering challenges.
1. Uneven Expert Load
If a large number of tokens concentrate on a small number of experts, some devices will be busy while others are idle. Overall speed is often determined by the slowest expert shard. Training and inference systems usually need routing regularization, capacity limits, or dynamic scheduling to alleviate this problem.
2. High Expert Communication Cost
In a multi-GPU environment, tokens may need to be sent from the current device to experts on other devices, and the computation results then sent back. This process is called expert-parallel communication. Network bandwidth, communication latency, and topology directly affect throughput.
3. Limited Utilization for Small-Batch Requests
When the number of requests is small or the context is short, expert matrix multiplications may not form a sufficiently large batch, reducing hardware utilization. In this case, the theoretical advantage of activated parameters may not fully translate into actual speed gains.
4. More Complex VRAM Management
Different experts are accessed at different times. The deployment system needs to handle resident weights, sharded loading, cache eviction, and migration between devices. If experts are frequently swapped in and out, storage bandwidth may become a new bottleneck.
6. Parallelism Strategies to Consider During Deployment
Ultra-large models usually combine multiple parallelism strategies.
Tensor parallelism splits a computation layer across multiple GPUs. It is suitable for layers with a large number of parameters, but it increases synchronization communication between devices.
Pipeline parallelism assigns different layers of the model to different devices or device groups. It can reduce pressure on a single GPU, but it requires handling transfers between stages and pipeline idle time.
Expert parallelism distributes different experts across different devices so that tokens can access the corresponding experts based on routing results. For MoE models, this is an important way to reduce expert storage pressure on a single GPU.
Actual deployment may also combine data parallelism to serve more independent requests. The choice of combination depends on VRAM capacity, interconnect bandwidth between devices, context length, concurrency scale, and target latency.
7. Quantization, Offloading, and Local Runtime Strategies
If the goal is a personal workstation or a small server, running an ultra-large model locally usually requires tradeoffs among precision, speed, and hardware cost.
Quantization
Quantization reduces storage and VRAM usage by lowering the precision of weight representation. Common approaches include converting high-precision weights to lower-bit formats. Quantization does not only affect model size; it may also affect output quality, operator compatibility, and inference speed. For multimodal and tool-calling scenarios, stability on complex tasks should be specifically validated.
Weight Offloading
Some experts or temporarily inactive weights can be placed in host memory or local storage and transferred to the accelerator when needed. This method can alleviate insufficient VRAM, but data movement between the device and host may significantly increase latency, making it unsuitable for scenarios with strict real-time requirements.
Layer-Wise Loading
If the inference framework supports reading expert weights on demand, some experts can be dynamically loaded based on routing results. This approach is sensitive to storage device throughput and cache hit rate. During deployment, attention should be paid to first-request latency and stability under continuous requests.
8. A Practical Deployment Evaluation Checklist
Before actually downloading and deploying the model, evaluate it in the following order:
- Confirm the weight format, number of shards, and actual file size;
- Measure available VRAM per GPU, not just nominal VRAM;
- Estimate KV Cache based on context length;
- Confirm whether the inference framework supports MoE routing and multi-GPU parallelism;
- Check interconnect bandwidth and topology between devices;
- Evaluate whether host memory and local storage are sufficient;
- First validate functionality with a single user and short context;
- Then gradually increase context, concurrency, and output length;
- Record first-token latency, generation speed, peak VRAM usage, and communication usage;
- Test stability separately for long text, multimodal input, and tool calls.
9. How to Interpret “Runs Locally”
“Can run locally” usually only means that inference can be completed under certain hardware conditions, with a suitable inference framework and proper configuration. It does not mean that an ordinary computer can use the model smoothly.
For ultra-large MoE models, local deployment has at least three levels of meaning:
- Can load: The weights can fit into device or host memory;
- Can infer: Operators, routing, and multi-GPU communication can execute correctly;
- Can be practical: Latency, throughput, and stability meet the requirements of real scenarios.
If the goal is only to verify the model format, lower speed and higher latency may be acceptable. If the goal is to provide multi-user service, concurrency scheduling, cache isolation, fault recovery, logging and monitoring, and resource quotas must also be considered.
Overall, 2.8 trillion parameters, about 1.56 TB of weights, and about 104 billion activated parameters reflect a technical route that trades sparse activation for extremely large model capacity. It reduces part of the per-token computation pressure, but it does not eliminate the system costs of weight storage, multi-GPU communication, and runtime caching. For developers, the real deployment challenge is not only “how many GPUs to prepare,” but establishing a complete evaluation system covering weight format, VRAM budgeting, parallelism strategy, and service stability.
