Deploying a 1.56TB MoE Open-Weight Model: 2.8T Parameters, 8 GPUs, and Practical Evaluation
This article analyzes a new generation of ultra-large open-weight Mixture of Experts models, using a model with about 2.8T total parameters and 1.56TB of weights as a representative case. It explains why “being able to run” such a model is not a single metric, and separates total parameters from activated parameters, weight storage from compute, VRAM capacity from memory bandwidth, and offline inference from online serving. It also covers quantization, parallelism, KV-cache management, expert offloading, self-reported benchmark interpretation, and a practical deployment checklist for engineering teams.
Introduction
A new generation of ultra-large open-weight models has appeared. Two numbers are frequently cited: about 2.8T total parameters and about 1.56TB of weight files. In community discussions and social media posts, these models are often reduced to a single sentence, such as “2.8T parameters, 1.56TB, runs on eight GPUs locally.”
What is usually omitted is the most important engineering question: what does “runs” actually mean? It can mean offline inference at only a few tokens per second, or it can mean meeting the throughput and latency requirements of online serving. These are not the same thing. The hardware and configuration requirements for the two cases can differ by an order of magnitude.
This article does not focus on one specific model. Instead, it treats the model as a representative example of a new generation of ultra-large sparse Mixture of Experts models, and answers three questions:
- Starting from 2.8T total parameters and 1.56TB of weights, what capacity and bandwidth calculations are needed to run it on eight GPUs?
- How should quantization, parallelism, KV-cache management, and expert offloading be chosen?
- How should vendor-reported coding, agent, knowledge, and reasoning benchmark results be interpreted, and how can they be validated in your own environment?
1. Three Key Concepts: Total Parameters, Activated Parameters, and Weight Size
1.1 Sparse MoE: Memory Is Determined by Total Parameters, Compute by Activated Parameters
A model with 2.8 trillion total parameters is almost certainly a sparse Mixture of Experts model. MoE splits feed-forward layers into many parallel “experts.” For each token, a routing network selects only a small number of experts, such as Top-2 or Top-4. The other experts do not participate in that forward pass.
This creates two concepts that must be understood separately:
- Total parameter count: determines the model’s knowledge capacity and the total amount of weights that must remain resident in VRAM or system memory.
- Activated parameter count: determines the compute and memory-bandwidth read volume required to process one token. With the same hardware, a smaller activated parameter count means a higher theoretical decoding speed limit.
This explains a common confusion: how can a 2.8T-parameter model still provide usable generation speed on a limited number of accelerators? Compute is determined by the activated parameters, but the capacity threshold is still determined by all the weights.
1.2 Estimating from Precision: What 1.56TB Means
The approximate weight size of a 2.8T-parameter model under different storage precisions is:
| Storage precision | Bytes per parameter | Estimated weight size |
|---|---|---|
| BF16/FP16 | 2 | About 5.6 TB |
| FP8 | 1 | About 2.8 TB |
| INT4, excluding quantization metadata | 0.5 | About 1.4 TB |
| Publicly reported weight size | About 0.56 | About 1.56 TB |
The 1.56TB figure is not equal to the roughly 5.6TB required by half precision, nor is it equal to the roughly 2.8TB required by FP8. It is slightly higher than the theoretical value of pure 4-bit quantization. A reasonable explanation is:
The public release contains quantized weights, with most parameters stored at around 4 bits and additional space used by some parameters or quantization metadata.
Which precision is used for computation, for example 4-bit storage followed by high-precision dequantization, depends on the inference engine and hardware capabilities. The official technical report should be treated as the final reference. However, the engineering conclusion is clear:
- The weight release is designed for deployment after quantization.
- Loading the same model at half precision requires a hardware threshold roughly 3.6 times higher than 1.56TB.
1.3 Can Eight GPUs Hold It? Capacity and Bandwidth
First, consider capacity. If 1.56TB is spread evenly across eight GPUs, each GPU must hold about 195GB, before counting KV cache, runtime buffers, and framework overhead.
The following table compares common eight-GPU memory configurations, using decimal GB:
| Total VRAM across 8 GPUs | Per-GPU capacity | Can it hold 1.56TB of weights? | Engineering implication |
|---|---|---|---|
| 384 GB | 48 GB | No | Must rely on expert offloading |
| 640 GB | 80 GB | No | Expert offloading is mandatory |
| 1128 GB | 141 GB | No | Some experts must be offloaded |
| 1536 GB | 192 GB | Barely | Very little headroom; context compression or light offloading may be needed |
| 2304 GB | 288 GB | Yes | Enough headroom for long context and concurrency |
In a 64×64 routing grid, a 1.56TB footprint can be viewed as 64 expert blocks of about 24GB plus 4 parameter blocks of about 15GB. Expert blocks can be distributed across cards, while some parameter blocks may need to be split, for example about 6GB per GPU, or kept resident on a single GPU. Therefore, whether a “single node with eight GPUs” configuration works depends on the specific expert layout strategy.
The conclusion has three layers:
- Capacity: whether eight GPUs can hold the weights depends on whether each GPU has at least 192GB of memory.
- Bandwidth: the bottleneck appears during decoding.
- Offloading: on GPUs without enough memory, the excess must be handled by host memory through PCIe, which limits bandwidth.
2. VRAM Budget: Turning “Can Run” into a Calculation
2.1 Calculation Formula
Inference-time VRAM demand can be broken into four parts:
Total VRAM demand ≈ weights + KV cache + runtime buffer + activations and temporary tensorsKV cache per request = 2 × num_layers × num_kv_heads × head_dim × context_length × kv_bytes
The coefficient 2 in the KV cache formula corresponds to Key and Value. For models using grouped-query attention, the number of KV heads is much smaller than the number of attention heads, so this term can be reduced significantly.
The following is an example estimate. The values below are illustrative and only demonstrate the calculation method:
def estimate_vram(total_params_B, bytes_per_param, num_layers, num_kv_heads,head_dim, max_context, concurrency, kv_bytes=2, buffer_ratio=0.2):"""Estimate single-instance VRAM requirements (unit: GiB, 1 GiB = 2^30 bytes)."""gib = 1024 ** 3weights = total_params_B * 1e9 * bytes_per_param / gibkv_per_token = 2 * num_layers * num_kv_heads * head_dim * kv_bytes / gibkv_cache = kv_per_token * max_context * concurrency# buffer_ratio: runtime overhead for memory fragmentation, communication buffers, temporary tensors, etc.runtime = (weights + kv_cache) * buffer_ratioreturn {"weights": weights, "kv_cache": kv_cache, "runtime": runtime,"total": weights + kv_cache + runtime}report = estimate_vram(total_params_B=2800, bytes_per_param=0.56, num_layers=60,num_kv_heads=8, head_dim=128, max_context=32768,concurrency=4, kv_bytes=2)print(report)
weights ≈ 1462.88kv_cache ≈ 8.05runtime ≈ 294.19total ≈ 1765.12 # GiB
2.2 Interpreting the Results
The KV cache is small. Under the assumptions above, the KV cache per token is about 0.25MiB. A 32K context for one request uses about 8GB. Even with concurrency increased to 64 requests, it is only about 512GB. Weight storage dominates. KV-cache management is helpful, but it is not the decisive threshold.
Runtime overhead cannot be ignored. A 20% buffer adds about 294GB, which is equivalent to about 1.6 additional 288GB-class accelerators. Setting GPU memory utilization too aggressively can cause fragmentation or even service interruption. Capacity planning must leave room for this overhead.
The real bottleneck is memory bandwidth. The weights exceed 1.4TiB. Even when tensor parallelism splits the weights across eight GPUs, each GPU must hold roughly 183GiB, leaving little room for cache and runtime overhead. In practice, weight capacity and memory bandwidth dominate, so cache headroom alone cannot be treated as service capacity.
The following table estimates single-stream output speed under different conditions. Values use decimal GB/TB.
| Parallelism and precision assumption | Weights read per token | Available bandwidth | Single-stream upper limit |
|---|---|---|---|
| 8-GPU tensor parallelism; about 3 TB/s per GPU; 4-bit weights read at 0.5 bytes per parameter | About 182GB per GPU | About 24 TB/s | About 130 token/s |
| 4-GPU expert parallelism; about 1 TB/s per GPU; activated parameters assumed to be 15% of total parameters | About 105GB per GPU | About 4 TB/s | About 38 token/s |
| Single node with 8 GPUs plus host-memory expert offloading; PCIe 5.0 x16 unidirectional bandwidth about 64 GB/s; activated parameters assumed to be 15% of total parameters | About 59GB must be swapped in through PCIe per token | About 64 GB/s | About 1 token/s |
Three practical conclusions follow:
- With the same bandwidth, using more GPUs reduces the weight each GPU must read, so single-stream speed improves nearly linearly.
- The ratio of activated parameters to total parameters, which reflects routing sparsity, directly determines the theoretical speed limit.
- Expert offloading can make the model “run,” but at the bandwidth level it is nearly “unable to run.” It is suitable only for offline, low-concurrency, latency-insensitive scenarios.
Actual speed is also affected by collective communication, routing imbalance, attention computation, and scheduling overhead. Real performance must be measured on the target hardware, target context length, and target concurrency level.
3. Parallelism Strategies: Choosing TP, EP, and PP
| Parallelism method | Partition target | Main communication | Suitable topology | Main risk |
|---|---|---|---|---|
| Tensor Parallelism, TP | Individual weight matrices | Multiple all-reduces per layer | High-bandwidth intra-node interconnect | Frequent communication; cross-node deployment significantly slows decoding |
| Expert Parallelism, EP | Different experts | All-to-All dispatch and return | Intra-node, or multi-node with RDMA network | Routing skew causes load imbalance and tail-latency jitter |
| Pipeline Parallelism, PP | Different layer segments | Point-to-point transfer between adjacent stages | Inter-node | Pipeline bubbles; requires enough micro-batches to fill the pipeline |
The selection principle can be summarized as “prefer intra-node first, then inter-node.”
Keep TP within a node whenever possible. TP communication is directly related to the number of layers. More layers mean more communication. If TP crosses nodes, decoding must pay a network round-trip cost for every generated token.
EP requires attention to routing balance. If a small number of experts are selected by a large number of tokens, the corresponding devices become long-tail bottlenecks. Overall speed is determined by the slowest device.
PP is suitable for cross-node scaling. However, it needs enough micro-batches to fill the pipeline. Otherwise, bubbles will significantly reduce throughput.
A proven progressive path is:
- If capacity is the main constraint, prioritize capacity-focused measures rather than blindly adding parallelism.
- If a single GPU cannot hold the model, first split it with TP within the node.
- If a single node cannot hold the model, add EP or PP across nodes.
4. Quantization and KV Cache: Two Key Switches
4.1 Weight Quantization
Weight quantization: common approaches include 4-bit group quantization, activation-aware weight quantization, and FP8 per-tensor scaling. Aggressive quantization can significantly reduce VRAM and bandwidth requirements, but may cause accuracy loss.
MoE-specific risk: some experts may not receive enough training. They may be more sensitive to quantization errors. Average metrics may hide the problem of individual experts collapsing.
Validation requirement: a quantized model must be re-evaluated on the target tasks. It is not safe to assume that performance is identical before and after quantization.
4.2 KV Cache Optimization
GQA: multiple query heads share one group of KV heads, directly reducing KV cache usage.
Low-rank compressed attention: some architectures, such as Multi-head Latent Attention, jointly compress Key and Value with low-rank projections, further reducing cache size.
KV cache quantization: reduce the KV cache from half precision to lower precision such as FP8.
Paged cache management: split the cache into fixed-size blocks and allocate them on demand. This reduces fragmentation and improves batching density.
For a 2.8T-class model, these optimizations do not change the fundamental judgment that “weights determine the threshold.” However, they determine how many concurrent requests can be supported under long context, and whether VRAM headroom can be converted into throughput.
4.3 Expert Offloading
Principle: keep experts that are temporarily not needed in host memory or local storage, and swap them into VRAM on demand.
Common implementations:
- Layer-based placement: early layers stay resident in VRAM, while other layers reside in host memory. During the forward pass, layers are swapped in one by one. This is simple, but each swap waits for PCIe transfer.
- On-demand caching: recently used experts are cached in VRAM, using temporal locality for cache hits. If locality is weak, the hit rate can drop significantly.
- Speculative prefetching: use a lightweight proxy to estimate which experts will be selected in the next layer, then issue asynchronous copies in advance to overlap computation and transfer.
- Distributed parameter server: expert parameters are managed centrally by CPUs or remote nodes.
Fundamental limitation: when weights are much larger than VRAM, generating one token requires swapping in at least almost one set of activated parameters through PCIe. If activated parameters are 15% of the total, the activated parameter count is about 420B. At 0.5 bytes per parameter, one token requires about 210GB of transfer. PCIe 5.0 x16 provides about 64 GB/s unidirectional bandwidth, so a single swap-in takes more than three seconds. In the worst case, if hot caches miss across all layers, the access amplification can reduce speed to well below one token per second.
Therefore, expert offloading is a fallback mechanism: usable, but slow.
- It is suitable for offline batch processing, single-machine demonstrations, or verifying weight integrity.
- It is not suitable for online interactive use or long agent tool-calling loops. A call loop of dozens of rounds may take hours.
5. Understanding Self-Reported Benchmarks
5.1 Interpreting the Results
Vendor-reported results show a mixed picture. The following are vendor self-reported numbers:
| Dimension | This model | Main competitors | Interpretation |
|---|---|---|---|
| Terminal agent tasks | 65.7% | 66.4% / 58.2% | Roughly on par with the first tier |
| Long-horizon software engineering tasks | 51.7% | 46.3% | About 5.4 percentage points ahead |
| Science-plus-terminal tasks | 38.1% | 63.3% | About 25.2 percentage points behind |
| PhD-level hard knowledge reasoning | 39.4% | 60.9% | About 21.5 percentage points behind |
| Overall | Leads in 16 of 37 items | — | Mixed results |
According to a third-party evaluation organization’s intelligence index, the model scores 55 and ranks fourth globally.
These results suggest a clear profile:
Coding and agent capabilities are in the first tier, while science and some deep reasoning capabilities show a clear gap.
This can be understood from the training side. Coding and terminal tasks have naturally verifiable feedback signals, such as compilers, unit tests, and command execution results. They are well suited to reinforcement learning, execution-based feedback, and synthetic data. Scientific knowledge and multi-step reasoning rely more heavily on high-quality corpora and long-tail knowledge coverage. Public data is limited, and such gaps are difficult to close in a short period.
For engineering practitioners, this suggests that weaknesses do not necessarily become bottlenecks. They can often be compensated with retrieval augmentation, tool calling, and other engineering methods.
5.2 Six Common Interpretation Mistakes
Treating self-reported results as definitive. Vendors often select favorable tasks, prompt templates, decoding parameters, and test conditions. Results should be treated as an upper-bound reference. Final judgment should come from third-party evaluations or your own evaluation sets.
Treating decimal-level differences as decisive. Differences such as 65.7% versus 66.4%, or 58.2% versus 51.7%, often fall within sampling variation when the number of tasks is limited. A more reasonable conclusion is “coding capability is comparable” or “there is a generational gap in science capability.”
Ignoring scaffolding differences. Agent benchmarks, such as terminal operation or end-to-end software engineering tasks, depend heavily on the toolset, prompt templates, retry policies, timeout settings, and test environment. Changing the scaffolding can significantly change the same model’s score. When comparing models, use the same scaffolding.
Ignoring data contamination. Public evaluation sets may have entered the training corpus, causing inflated scores. It is recommended to cross-validate with custom task sets, temporal split test sets, or lightly rewritten tasks.
Looking only at average scores. Vendors may claim that the model “beats the opponent in 16 of 37 items,” but averages can be distorted by extreme items. Weighting by business-relevant dimensions is more meaningful than looking only at total rankings.
Confusing scores with cost. Benchmarks do not reflect token pricing, first-token latency, or concurrency capacity. Open-weight models that can be privately deployed have value in data residency and deep customization that closed APIs cannot easily replace.
5.3 Reproduction and Acceptance Checklist
| Control item | Common practice | Risk and recommendation |
|---|---|---|
| Evaluation target | Use the latest weights and record the commit or checksum | Mixing versions makes results non-reproducible |
| Decoding parameters | temperature=0 or fixed seed, top_p=1 | Average over multiple samples to remove randomness |
| Generation length | Unified max_tokens and fixed timeout | Truncated generation may be misjudged as failure |
| Scaffolding | Unified toolset, prompt template, and retry policy | Comparing models without comparing environments |
| Task source | Custom set plus public set for cross-validation | Prevent data contamination |
| Metrics | pass@1, pass@k, success rate, first-token latency, per-token latency, concurrent throughput, cost per 1K tokens | Evaluate multiple dimensions, not a single metric |
6. Deployment Recommendations for Production
6.1 Three Deployment Options
Local bare-metal deployment
- Approach: a single node with eight high-VRAM GPUs, or a two-node sixteen-GPU cluster with high-speed RDMA interconnect between nodes.
- Advantages: data does not leave the domain, customizable, no per-call fee, and allows deep tuning of parallelism and quantization.
- Challenges: high upfront investment, requires operations capability for weight loading and version upgrades, and a larger fault domain for multi-GPU clusters.
Cloud GPU resources
- Approach: request single-node multi-GPU or dual-node multi-GPU instances on cloud platforms such as Baidu AI Cloud. Weights can be distributed through object storage, services can be orchestrated with container platforms, and observability can be supported by monitoring, alerting, and logging services.
- Advantages: elastic scaling, no hardware operations, and pay-as-you-go usage.
- Notes: VRAM and interconnect specifications must be derived from the weight size, not from experience with conventional 7B or 70B models. Storage throughput must be sufficient to load weights quickly.
API calls
- Suitable for teams with a single task type and data compliance requirements that allow cloud usage. Advantages include no operations burden and good elasticity.
A pragmatic approach is often a hybrid model: “validate in the cloud, then deploy in production on-premises or in a private environment.” First run a complete load test in the cloud using the target precision, parallelism, and concurrency. Confirm that VRAM and throughput meet requirements, then decide whether to purchase hardware or build a private cluster.
6.2 Recommended Process
Calculate VRAM correctly: weights + KV cache + runtime overhead must fit within per-GPU capacity × number of GPUs, with at least 15% to 20% headroom reserved.
Choose precision and parallelism: prefer official quantized weights. Keep TP within a node. If there are many layers and cross-node scaling is needed, add PP.
Benchmark decoding speed: measure tokens per second under the same hardware, context length, and concurrency. Do not rely only on vendor-reported throughput numbers.
Control the KV cache: estimate the cache upper bound based on GQA or low-rank compressed architectures. Configure paged cache management and KV cache quantization.
Decide whether to offload experts: enable expert offloading only in offline, low-concurrency, high-latency-tolerant scenarios.
Establish an evaluation baseline: use a custom task set covering coding, agent capabilities, and knowledge reasoning. Run evaluation both before and after quantization.
Monitor in production: track first-token latency, tokens per second, VRAM utilization, routing load balance, and error rates.
6.3 Quick Decision Table
| Scenario | Recommendation |
|---|---|
| 8 GPUs, per-GPU VRAM ≤141GB | Expert offloading is required. Suitable only for offline batch processing. For interactive use, choose a smaller model or a larger cloud GPU instance. |
| 8 GPUs, per-GPU VRAM 192GB | All weights can be loaded, but KV cache headroom is limited. Context compression and cache optimizations are recommended. |
| 8 GPUs, per-GPU VRAM ≥288GB | Can support a 32K context and moderate concurrency. Suitable as a primary private deployment scenario. |
| Dual-node, 16 GPUs | Weight sharding is sufficient. Focus on cross-node communication and EP load balancing. |
| Only coding and agent capabilities are needed | Prioritize evaluating quantized versions of comparable open-weight models, or use a cloud API. |
| Scientific knowledge reasoning is required | The model alone may still have weaknesses. Add retrieval augmentation and tool calling. |
Conclusion
A 2.8T-parameter open-weight MoE model with 1.56TB of weights is one signal that the gap between open and closed models is narrowing. However, for engineering teams, the real question is not simply “can it run?”
The more important questions are:
- Given the available hardware, how fast can it run?
- Under how long a context and how much concurrency can it serve stably?
- How does it perform on the target business tasks?
Accurately calculating VRAM and bandwidth, and aligning evaluation methods with business scenarios, is more valuable than chasing benchmark rankings.
