2.8T Open MoE Model Deployment: A Practical Guide to Starting on 8 GPUs
This article explains how to evaluate and deploy a newly released 2.8T-parameter open MoE model with 104B activated parameters per token. It covers the model architecture, license considerations, GPU memory and bandwidth requirements, parallelization strategies, benchmark interpretation, and a checklist for deciding whether to self-host or use managed inference.
From downloading 1.56TB to generating tokens reliably: a deployment guide for a very large MoE model
Overview
On the evening of November 4, a China-based large-model team released the full weights of a 2.8T-parameter model on a major model hosting platform. The files were available for direct download without an application process and totaled about 1.56TB. The team described it as the strongest openly available weight model at the time.
Whether that claim holds can be left to benchmark discussions. For developers, a more practical question is: after downloading the weights, how can you make the model generate tokens reliably on one or more servers?
This article compresses the pre-release process into a short background section and focuses on what happens after the weights are available:
- What is inside the model;
- How the license and access constraints are defined;
- How to calculate GPU memory, bandwidth, and parallelism;
- How to interpret the vendor-reported benchmark results;
- How a team should evaluate and adopt the model.
1. Background: a weight release that delivered on its promise
The model did not appear out of nowhere:
- On October 24, it first became available through an API and web interface. At the same time, the team promised to release the full weights before November 4.
- On November 2, the hosted repository was created. It remained closed on the morning of November 4 and was officially opened that evening.
- The repository contains 112 files, including 96 weight shards, totaling about 1.56TB. The files are intended for direct download; where supported mirrors are available, they may help improve synchronization.
Three facts are worth remembering:
- The release kept its schedule. About 11 days passed between the API launch and the open-weight release.
- No application was required. The weights were not released under a controlled-access process; anyone with the download link could obtain them.
- The scale raises the bar. It is described as the largest open-weight model released so far, which directly raises the memory threshold required for a server to run it.
The size itself reflects a broader trend: competition among frontier models is moving from leaderboard scores to deployment cost. Open weights solve availability, but usability depends on whether the hardware, license, and inference stack are all ready.
2. Model anatomy: 2.8T total parameters and 104B activated parameters
According to the official model card, this is a native multimodal agent model, further trained from the team’s previous generation.
Table 1: Structure and quantization overview
| Item | Value | Engineering implication |
|---|---|---|
| Total parameters | 2.8T | Determines weight storage size and minimum GPU memory |
| Activated parameters per token | 104B | Determines the number of parameters involved in each forward pass |
| Expert structure | 896 routed experts + 2 shared experts; 16 selected per token | Sparse activation decouples compute from total model size |
| Layers | 93 layers; 24 full-attention layers replaced with sparse attention | Saves compute and memory for long context |
| Context length | 1 million tokens | Constrained by memory, parallelism, and scheduling |
| Vision encoder | Proprietary vision encoder | Native image-reading capability |
| Weight precision | 4-bit floating point with block scaling | Weight size is roughly one quarter of BF16 |
| Activation precision | 8-bit floating point with block scaling | Balances precision and compute efficiency |
Compared with the previous generation, the official structural change is limited to one point: 24 layers that originally used full attention were changed to sparse attention. Most other parts remain unchanged, including the MoE experts and million-token context. This small-step structural change serves two purposes:
- It reduces training and inference risk;
- It concentrates optimization on the most compute-intensive part, attention.
1. The activation ratio is about 3.7%, decoupling compute from model size
For each generated token, the model activates about 104 billion parameters, or roughly 3.7% of the total parameter count. In other words, 2.8T represents capacity, while 104B is closer to the amount of compute used at each step. This is the value of sparse MoE: more parameters provide knowledge capacity, while sparse routing keeps per-step compute within an acceptable range.
However, compute is not the entire cost:
- All expert weights must remain resident in GPU memory; otherwise, switching to another token would require reloading experts from external storage.
- The experts selected for a token may be spread across different GPUs, requiring cross-GPU data exchange at every step.
Therefore, the bottleneck of MoE deployment is often not compute, but GPU memory capacity and inter-GPU bandwidth.
2. Multimodal and agent positioning
Native multimodal means the vision encoder is not an external adapter. Images are encoded and processed together with text tokens during training and inference, making mixed text-image input a first-class input type.
The agent positioning means the model is intended for multi-turn tool calling and long-trajectory tasks. This places higher demands on long context, structured output, and instruction following.
3. Only one quantized version was released
The team released only one version: 4-bit weights and 8-bit activations. There is no BF16 or FP8 version. This choice has a clear engineering logic:
- It lowers the minimum memory threshold. In BF16 storage, the weights alone would require about 5.6TB of memory. With 4-bit storage, they require about 1.4TB, bringing the model into the range of a single 8-GPU machine with high-memory GPUs.
- It uses quantization-aware training to control precision loss. Quantization error is considered during training rather than added as a post-deployment compression step.
The trade-offs are equally clear:
- If you want to deploy the model, you must accept 4-bit precision; there is no fallback option.
- The fine-tuning toolchain must support a low-bit base model and validate precision on its own.
- The quantized release is more coherent as a product, but offers less flexibility.
3. The two conditions behind “open”: license and hardware
The two conditions are: first, the license terms; second, the hardware threshold.
(1) License conditions: open weights do not mean no constraints
Open weights only mean that the model files can be downloaded. They do not mean the model can be used without restrictions. Common limitations include:
- User-scale clauses may trigger stricter terms. Some model licenses require separate authorization once a product exceeds a certain daily active user threshold.
- Redistribution and attribution obligations. There may be specific requirements for redistributing weights, naming derivative models, or including accompanying statements.
Before deployment, make sure to:
- Read the license file and model card in the repository line by line, and confirm the scope of commercial use, user-scale clauses, and output-use restrictions;
- Evaluate legal and compliance risks;
- Confirm whether redistribution and attribution obligations can be met;
- If the model is integrated as a foundation into your own product, pay attention to the product terms and output ownership rules.
(2) Hardware conditions: 8 GPUs as the practical starting point
The roughly 1.56TB of weights must be fully resident in memory during inference. This is a hard lower bound. Table 2 compares different memory configurations using a unified set of assumptions:
Table 2: Minimum GPU count under different memory configurations
| Per-GPU memory (illustrative) | Common configuration | Theoretical capacity of one node | Minimum number of GPUs (illustrative) |
|---|---|---|---|
| 80GB | 8-GPU node | 640GB | 20 |
| 141GB | 8-GPU node | 1.13TB | 12 |
| 192GB | 8-GPU node | 1.54TB | 12 |
| ≥200GB | 8-GPU node | ≥1.6TB | 8 |
Note: The values are used to illustrate scale; actual numbers vary by hardware and framework.
Only when per-GPU memory reaches 200GB or more can the weights fit into a single 8-GPU machine. In other words, “8 GPUs as the starting point” is not a performance recommendation; it is a physical lower bound. If memory is insufficient, parallelism must be extended from a single node to multiple nodes.
Even if you have enough memory, the inference stack must still pass the following checks:
- The inference engine must support the model. Sparse attention requires corresponding support in the inference engine. Otherwise, it may fall back to full-attention computation, and long-context memory and compute overhead will increase significantly.
- Streaming output is required. For agent scenarios, use a calling method that supports streaming output to avoid holding a connection open while waiting for a complete response.
- Structured output constraints are required. Structured output constraints, such as JSON Schema, help ensure that tool-call parameters can be parsed.
- Prompt and parsing specifications must be complete. Tool definitions, call formats, and error-retry policies must be aligned with the model’s capabilities.
For most teams, a more practical approach is to first validate the model through an API or cloud inference service, then decide whether to self-host based on data compliance, cost, and usage volume.
4. How to read the vendor-reported benchmarks
All benchmark numbers in the model card are vendor-reported: on a terminal-task benchmark, the model scored 65.7%, slightly below 66.4% for an international flagship closed model and above 58.2% for another flagship model. On a software-engineering task, it scored 51.7%, above 46.3% for the comparison model. The vendor also emphasized that it leads on 16 of 37 benchmarks.
However, it does not lead across the board:
- On scientific terminal tasks, it scored 38.1%, compared with 63.3% for the comparison model;
- On a comprehensive hard-reasoning evaluation, it scored 39.4%, compared with 60.9% for the comparison model.
A third-party composite index gives it 55 points, ranking fourth globally. Summarizing the numbers, the conclusion can be expressed in eight words: strong in coding and agents, weaker in science and some reasoning.
Table 3: Representative benchmark comparison (vendor-reported)
| Capability area | This model | International flagship closed model | Takeaway |
|---|---|---|---|
| Terminal tasks | 65.7% | 66.4% | Roughly on par, slightly lower |
| Software engineering tasks | 51.7% | 46.3% | Ahead |
| Scientific terminal tasks | 38.1% | 63.3% | Clearly behind |
| Hard reasoning evaluation | 39.4% | 60.9% | Clearly behind |
| 37 benchmarks | Leads on 16 | — | Mixed results |
The data comes from vendor release materials. When using it, pay attention to four points.
1. Vendor-reported data needs third-party validation
Self-reported results can be affected by evaluation settings, prompt templates, and sampling strategies. A third-party composite index can only serve as a reference; it cannot replace your own testing.
2. Look at the distribution, not just the total score
Leading on only 16 of 37 benchmarks indicates that the model has uneven capabilities. When selecting a model, prioritize the capabilities most critical to your business rather than focusing on a single aggregate score.
3. Weaknesses can be compensated with system-level techniques
- For scientific reasoning and knowledge-heavy tasks, retrieval-augmented generation can help, as can integrating computation and retrieval tools;
- For complex reasoning, external solvers or multi-step verification can be introduced;
- For task-oriented dialogue, examples can be provided at the prompt level, and result-validation steps can be added.
4. Build your own evaluation set
Create an evaluation set of 100 to 300 samples based on real business data, covering typical and long-tail scenarios. Compare open-weight models, closed APIs, and smaller models in terms of quality, latency, and unit cost. Use your own data, not leaderboards, as the basis for decisions.
5. Rough calculations before deployment: memory, compute, and bandwidth
1. Weight size: the order of magnitude matches
At 4-bit storage, 2.8T parameters theoretically require 2.8 × 10^12 × 0.5 bytes, or about 1.4TB. The actual size of 1.56TB is slightly higher. The difference usually comes from:
- The vision encoder and vocabulary;
- Unquantized embedding and normalization layers;
- Metadata such as quantization scale factors.
2. Peak memory: weights are only the lower bound
The memory usage of one inference process can be broken down into four parts:
Peak GPU memory ≈ weights (including unquantized parts) + KV cache + activations + communication buffers + framework overhead
KV cache can be estimated as follows:
KV cache per token = number of layers × number of attention heads × head dimension × 2 (K and V) × bytes per valueTotal KV cache = KV cache per token × sequence length × number of concurrent requests
Assuming KV cache is stored in BF16 (illustrative):
| Context length | KV cache per request (illustrative) |
|---|---|
| 8K tokens | About 12GB |
| 32K tokens | About 48GB |
| 128K tokens | About 192GB |
This explains why sparse attention is necessary. Every layer must maintain a full key-value cache. For a 93-layer model, this overhead can quickly approach or even exceed the memory footprint of the model weights in long-context scenarios.
3. Parallel strategy: make good use of 8 GPUs
For a single 8-GPU node, the weights can be split along three dimensions:
- Tensor parallelism. Split attention projections and feed-forward layers across GPUs;
- Expert parallelism. Distribute the 896 experts across GPUs; with 8 GPUs, each GPU would carry 112 experts;
- Shared experts. These can be split with tensor parallelism or replicated across data-parallel groups.
A common rule of scheduling is: the higher the bandwidth of the interconnect, the more communication-intensive parallelism it should carry.
- Tensor parallelism involves collective communication at every layer and should be limited to the high-bandwidth domain inside a node;
- When expert parallelism spans nodes, all-to-all exchanges are constrained by network bandwidth and can become a throughput bottleneck;
- Longer context means more communication;
- Routing balance also matters: uneven expert load can make the GPU hosting a “hot expert” the bottleneck, causing waiting.
Actual performance must be validated through small-scale load testing. Do not copy someone else’s parallel configuration blindly.
4. Memory estimation script
GiB = 1024 ** 3 # Use GiB consistently to avoid GB/GiB mismatchesdef estimate_memory(total_params=2.8e12, # Total number of parametersweight_bits=4, # Number of bits per weightactivated_params=104e9, # Activated parameters per token (affects compute only, not separate weight memory)layers=93, # Number of layersheads=64, # Number of attention heads (illustrative; use the actual configuration)head_dim=128, # Head dimension (illustrative)kv_bytes=2, # KV cache precision: 2=BF16, 1=FP8seq_len=128_000, # Context length per requestconcurrency=8, # Number of concurrent requestscard_gib=192, # Memory per GPU (illustrative)num_cards=8, # Number of GPUskv_reserve_ratio=0.10, # Ratio of memory reserved for KV cache):"""Roughly estimate the minimum per-GPU memory and maximum supported context length."""# 1) Weight memory: bytes per parameter + 8% metadata overhead, divided across GPUsweight_gib = total_params * weight_bits / 8 * 1.08 / GiBper_card_weight = weight_gib / num_cardskv_budget = per_card_weight / kv_reserve_ratio * (1 - kv_reserve_ratio)# 2) KV cache per tokenbytes_per_token = layers * heads * head_dim * 2 * kv_bytes# 3) Total supported context tokens, evenly divided by requestsmax_tokens = kv_budget * GiB / bytes_per_tokenprint(f"Weight memory is approximately {weight_gib/1024:.2f} TiB, {per_card_weight:.1f} GiB per GPU")print(f"KV cache budget is approximately {kv_budget:.1f} GiB per GPU, "f"supporting {max_tokens/concurrency/1024:.0f}K tokens per request "f"({concurrency} concurrent requests)")# 4) Activation size is used only to assess compute demand; it does not occupy separate weight memoryprint(f"Activated parameters per token: approximately {activated_params/1e9:.0f}B")if __name__ == "__main__":estimate_memory()
What the script means:
- Weight memory is calculated from total parameters × bytes per parameter, plus an 8% metadata overhead, then divided across GPUs;
- KV cache budget is inferred from a 10% reservation ratio;
- The maximum supported context is calculated from the per-layer cache size;
- Activation size is reported separately and is used to assess compute demand.
Actual values will vary with the inference engine, parallel strategy, and batching policy. It is recommended to calibrate with real workloads before going live and leave 10% to 20% of memory headroom.
5. Key inference mechanisms
- Continuous batching. New requests can join a decoding batch at any time, significantly improving throughput.
- Prefix caching. In agent scenarios, system prompts, tool definitions, and conversation history are carried repeatedly. A prefix-cache hit can reuse existing KV cache and reduce time to first token.
- Prefill/decode separation. Long-context prefill is compute-intensive, while decode is memory-intensive. Assigning these workloads to different resource pools can prevent interference.
- KV cache quantization. Storing the cache in FP8 can nearly halve memory usage, but precision loss should be evaluated.
6. Example inference configuration
// Key inference-service settings (illustrative; actual fields depend on the framework){"parallel": {"tensor_parallel_size": 8, // Keep tensor parallelism within the high-bandwidth domain of a single node"expert_parallel_size": 8, // Distribute the 896 experts across GPUs"enable_expert_parallel": true},"quantization": {"weight_bits": 4, // Only 4-bit weights are provided"activation_bits": 8},"context": {"max_model_len": 131072, // Set based on needs; do not default to 1 million tokens"sparse_attention": true // Requires engine support; otherwise long-context overhead grows sharply},"memory": {"kv_cache_dtype": "fp8", // Storing KV cache in FP8 can reduce memory usage"gpu_memory_utilization": 0.90},"scheduling": {"enable_chunked_prefill": true, // Execute long prefills in chunks to avoid blocking decode"enable_prefix_caching": true, // Reuse KV cache for system prompts and conversation history"max_num_seqs": 8 // Determine through load testing based on memory and latency goals},"output": {"structured_output": "json_schema" // Constrain tool-call outputs to a parseable structure}}
6. Deployment evaluation checklist
Hardware
- Per-GPU memory is sufficient for the weights to fit on a single 8-GPU node; otherwise, plan for multi-node deployment
- The node provides high-bandwidth GPU interconnect
- The cross-node network provides high bandwidth and low latency, and supports large-scale collective communication
- Storage throughput and network bandwidth match model download and loading requirements. At 100MB/s, a full 1.56TB download takes about 4.5 hours; at 1GB/s, it takes about 27 minutes
Software
- The inference framework supports this MoE structure, sparse attention, and low-bit quantization
- The fine-tuning toolchain supports a 4-bit base model, such as quantized low-rank adaptation, and can validate precision
- Structured output, streaming output, and tool-call parsing are supported
- Metrics monitoring and logging are in place. Key metrics include: time to first token, output latency per token, throughput, GPU memory usage, expert routing distribution, and error rate
- Rate limiting, circuit breaking, and fallback mechanisms are available, such as switching to a smaller model or a backup channel
Compliance
- License terms have been reviewed line by line, including commercial-use and user-scale restrictions
- Redistribution, attribution, and output-use obligations have been addressed
- If providing a public generative service, complete any applicable regulatory filings and safety assessments in the target jurisdiction
- Input and output content moderation mechanisms are in place
Evaluation
- Build a 100 to 300 sample business evaluation set
- Compare open-weight models, closed APIs, and smaller models under the same conditions
- Load-test at the target context length and concurrency
- For weak areas such as scientific reasoning, validate the compensation effect of retrieval augmentation and tool calling
7. Self-hosting or managed inference?
Self-hosting and managed inference are not mutually exclusive. They can be layered by scenario.
Self-hosting or private deployment is suitable for the following scenarios:
- Data is sensitive and cannot leave the environment;
- Call volume is high and stable;
- Deep customization is required.
At the same time, you must bear hidden costs such as GPU procurement and depreciation, data-center power, driver and framework upgrades, and troubleshooting. Total cost of ownership should be calculated over a 2 to 3 year usage period.
Cloud-hosted or API services are suitable for the following scenarios:
- The project is in a validation stage;
- Traffic fluctuates significantly;
- The team lacks experience in large-scale inference operations.
The advantages are elastic scaling, pay-as-you-go pricing, and freedom from hardware operations. Whether data can leave your environment should be judged by the enterprise according to compliance requirements.
For enterprises that self-host, cloud infrastructure can support the common parts:
- Object storage can host model weights, and a content delivery network can accelerate downloads;
- A container platform can schedule inference services and scale them elastically;
- Monitoring, alerting, and logging services can provide operational visibility;
- Elastic bare-metal or GPU cloud instances can host inference nodes.
Baidu AI Cloud provides such general cloud-product capabilities and can be considered a reference option for enterprises building their own inference platform, subject to regional availability and commercial terms. Model selection and architecture design should still be based on business evaluation results, not parameter scale.
If an 8-GPU machine is not affordable for now:
- Choose a smaller model first. A MoE model with about 30B activated parameters requires about 20GB of memory after INT4 quantization. A dense model with about 20B parameters requires about 40GB in BF16. Either can run on a single consumer or professional GPU.
- Use quantization and distillation. Distilling a smaller model from a larger model’s outputs can often achieve similar results on vertical tasks at lower cost.
- Plan a hybrid architecture. Use a small model for high-frequency simple requests and a large model for complex tasks, with both accessed through APIs or private deployment in a layered manner.
Conclusion
2.8T parameters, 1.56TB of weights, and an 8-GPU starting point together show that open-model competition has moved from “can it be opened?” to “can we afford to use it?”
The value of this weight release is that it puts frontier capabilities into developers’ hands. But the license and hardware thresholds are equally real:
- The license determines whether you can use it commercially;
- Memory determines whether you can run it;
- Engine support determines whether you can run it efficiently;
- Evaluation determines whether it fits your business.
A recommended path is: first validate the model through an API or cloud service, build your own evaluation set, then decide whether to self-host based on data compliance, call volume, and total cost of ownership, while keeping a smaller-model fallback option in your model routing.
Ultimately, the value of a model is never determined by parameter count alone, but by how much real-world value it can deliver at what cost.
