Trillion-Parameter MoE Open Weights: Technical Breakdown and Deployment Practices
This article examines a newly released Mixture-of-Experts model with about 2.8 trillion total parameters and roughly 104 billion activated parameters per inference. It explains the meaning of open weights, how MoE separates model capacity from per-token compute, why compute and memory must be evaluated separately, and the main engineering challenges of deploying trillion-parameter MoE models. It also covers leaderboard interpretation, hardware planning, quantization, serving strategies, agent-specific safeguards, memory estimation, and acceptance testing.
1. Why Open Weights Keep Gaining Momentum
On November 4, a leading Chinese large-model team delivered on an earlier public commitment by releasing the weights of a Mixture-of-Experts (MoE) model with approximately 2.8 trillion total parameters and about 104 billion activated parameters per inference.
After the weights were made available, the model topped a public Arena evaluation focused on agentic capabilities. It also ranked first among domestic models in a subsequent November leaderboard, with a composite score of 61.46. Third-party aggregate evaluation data indicated that it was about 14 points ahead of the previously strongest open model.
First, it is important to clarify the terminology. “Open weights” means the provider publishes model parameter files that users can download, load, and run in their own environments. It does not necessarily mean that training code, training data, and the full training pipeline are also open.
For engineering teams, access to weights provides three practical benefits:
- Private deployment: Inference can run in self-managed or rented compute environments, so sensitive data does not need to leave the enterprise boundary.
- Domain customization: Teams can perform continued pre-training, supervised fine-tuning, or parameter-efficient fine-tuning on top of general-purpose weights to adapt the model to domain-specific terminology and business workflows.
- Greater control: Teams can choose their own inference framework, quantization precision, and scaling strategy, without being locked to a single provider’s product roadmap.
However, “topping a leaderboard” and “performing well in production” are not the same thing. In practice, three major hurdles often stand between benchmark performance and usable business value: memory capacity, inference performance, and effectiveness validation. The following sections explain the architecture and provide practical deployment guidance.
2. Technical Breakdown: Understanding “2.8 Trillion Total Parameters / 104 Billion Activated Parameters”
2.1 Basic Structure of a MoE Model
The core idea of Mixture of Experts (MoE) is to replace the dense feed-forward network in a decoder architecture with multiple parallel “expert” sub-networks. A gating network, also called a router, decides which experts each token should be processed by:
- Each expert is usually an independent set of feed-forward layers, and their parameter capacities are not shared.
- The router computes a matching score between each token and each expert, then selects the top-scoring experts. A common design is Top-2 routing.
- A token is only processed by the selected experts during the forward pass; the remaining experts do not participate in that computation.
- Attention layers and other shared components generally remain dense. The main component being replaced is the feed-forward part, which usually accounts for the largest share of parameters.
This “large capacity, sparse activation” design fundamentally decouples model capacity from per-token compute.
2.2 A Key Distinction: Compute and Memory Must Be Evaluated Separately
In the industry, model parameter count is often directly equated with compute demand. For MoE architectures, however, these two dimensions must be assessed separately:
- Compute is determined by activated parameters. The forward computation per token is approximately 2× the number of activated parameters, or about 2.08 × 10¹¹ FLOPs. Compared with a dense model with the same activated parameter count, the per-token compute cost is in the same order of magnitude.
- Memory is determined by total parameters. All expert weights need to reside in memory. Assuming BF16 storage, 2.8 trillion parameters require approximately 5.6 TB, which is close to the memory requirement of a trillion-parameter dense model.
- Memory bandwidth is the key constraint during decoding. During token-by-token generation, each step only reads the activated expert weights, approximately 104 billion parameters, or about 208 GB in BF16. If the total effective memory bandwidth of the cluster is X TB/s, the theoretical decoding speed ceiling is approximately X × 1000 / 208 tokens/s. For example, if X = 24, the upper limit is about 115 tokens/s. Latency-sensitive deployments must perform capacity planning based on this constraint.
A practical conclusion follows: when evaluating whether a trillion-parameter MoE model can be deployed, the first question is not “Do we have enough compute?” but “Can the weights fit in memory, and can the experts be read fast enough?”
3. Deployment Challenges: Four Engineering Difficulties at Trillion-Parameter Scale
3.1 Memory Capacity and Parallel Splitting
Using BF16 as an example, the model itself requires about 5.6 TB of memory. Actual usage will be higher once activations, temporary buffers, and the KV Cache are included. A single server usually cannot hold the model, so multiple forms of parallelism must be combined:
| Parallelism | What is split | Main purpose | Cost and considerations |
|---|---|---|---|
| Tensor Parallelism (TP) | Weight matrices within a single layer | Reduces per-GPU weight and compute pressure | Requires frequent intra-layer communication; best limited to high-speed interconnects within a server |
| Expert Parallelism (EP) | Different experts | Distributes large numbers of experts across GPUs and is the main scaling method for trillion-parameter MoE | Relies on all-to-all communication to exchange tokens |
| Pipeline Parallelism (PP) | Model layers | Enables scaling across multiple servers | Too many pipeline stages can increase pipeline bubbles |
| Data Parallelism (DP) | Request replicas | Improves overall throughput | Each replica must hold the full model, so memory cost scales linearly |
In practice, TP should generally be kept within a single machine, while EP should scale across machines and depend on high-speed networking. If the deployment mode uses one request to occupy an entire group of devices, the number of concurrent requests the cluster can serve is limited, so resource pools must be planned in advance based on expected business peaks.
3.2 All-to-All Communication and Load Imbalance
- Communication-intensive: A MoE layer usually involves two all-to-all operations: first sending tokens to the devices hosting their target experts, then gathering the results after computation. This overhead grows with both the number of layers and batch size.
- Load imbalance: Real traffic often contains “hot tokens,” which can cause some experts to form longer queues and slow down the whole batch. During training, auxiliary load-balancing losses are often used to constrain the routing distribution. During inference, teams should monitor the actual load across devices.
- Capacity limits: Some implementations limit the number of tokens each expert can accept per batch. Tokens exceeding the limit may be dropped or sent through a fallback path. Teams must evaluate whether significant capacity overflow occurs under peak scenarios.
3.3 KV Cache and Long Context
The KV Cache grows linearly with concurrency × sequence length. The table below shows illustrative values for different decoder configurations (BF16, per token):
| Layers | KV heads | Head dimension | KV per token | 8K context per sequence |
|---|---|---|---|---|
| 32 | 8 | 128 | 128 KB | 1 GB |
| 64 | 8 | 128 | 256 KB | 2 GB |
| 80 | 8 | 128 | 320 KB | 2.5 GB |
Agentic applications often require multi-turn tool calls and long historical context. The KV Cache can easily become the concurrency bottleneck. The following techniques can help:
- Paged KV Cache management and prefix reuse;
- KV Cache quantization;
- Hard limits on context length;
- When trimming historical messages, preserving key conclusions and structured summaries.
3.4 Storage, Loading, and Operations
Weight files can easily reach hundreds of gigabytes or even several terabytes. Downloading, verification, distribution, and version management should all be included in automated workflows. Teams should verify file integrity, use read-only shared storage or local caches to avoid repeated downloads, and connect version information to monitoring and alerting systems.
4. Leaderboard Interpretation: How to Read “Topping the Charts” and “14 Points Ahead”
Leaderboards contain dense information, but their value depends on whether they are interpreted correctly. Four dimensions are recommended:
- Distinguish static benchmarks from live battles. Evaluations based on fixed question sets are objective and reproducible, but they can be vulnerable to targeted optimization or even data contamination. Arena-style evaluations based on real-user blind comparisons are closer to subjective user experience, but they are affected by question distribution, voter demographics, and sampling randomness. Rankings can therefore fluctuate. The two types of leaderboards should be used to cross-validate each other.
- Check whether the evaluation target aligns with the business target. Code generation, mathematical reasoning, and multi-step tool use have different leaderboard emphases. Agentic scenarios require particular attention to planning, tool calling, and error recovery, which cannot always be captured by a single composite score.
- Interpret score gaps in context. On a 100-point scale, a difference of about 14 points is significant. If a leaderboard uses an ELO-style rating, the gap should be converted into an expected win rate according to its statistical methodology. It should not be treated directly as a multiple of capability.
- Use your own business dataset as the final benchmark. Authoritative leaderboards should only serve as an initial screening step. Before production adoption, the model should be retested on desensitized real samples and evaluated against business metrics, such as:
- Task success rate;
- Human acceptance rate;
- First-token latency;
- Cost per request.
5. Deployment Practices: From Weight Files to Stable Services
5.1 Hardware and Parallelism Strategy
Planning should begin with three questions: target concurrency, acceptable latency, and precision requirements. For example, for an online chat scenario, teams can first define measurable targets such as “first-token latency below 1 second” and “at least 20 tokens/s output per user,” then work backward to determine the number of devices and parallelism configuration.
For trillion-parameter MoE models, it is recommended to prioritize inference engines that support expert parallelism and to benchmark them with a small set of real requests. Testing should focus on three metrics:
- Throughput (tokens/s);
- First-token latency;
- Per-token output latency.
When tuning, the following order can be used as a reference:
- Adjust the maximum batch size;
- Adjust the KV Cache usage limit;
- Adjust the expert parallelism degree.
The first two changes generally improve throughput at the cost of latency. The third mainly affects communication overhead and expert load distribution.
5.2 Quantization: Trading Lower Precision for Higher Throughput
Quantization methods such as FP8, INT8, and group-wise INT4 can reduce memory usage and improve memory access efficiency. For MoE models, additional considerations are needed:
- Dynamic routing: Different requests may hit different experts, so the calibration set must cover as diverse a set of real requests as possible.
- Item-by-item validation: Before and after quantization, results should be compared on the business evaluation set, while monitoring both perplexity and task-level metrics.
- Mixed precision: Attention layers can remain at higher precision, while only expert weights are quantized to lower bit widths to control accuracy loss.
5.3 Scheduling and Serving
- Use continuous batching so that new requests can join a batch that is already being decoded.
- Separate prefill and decode pools: long-document summarization requests can be concentrated in prefill, while chat requests focus on decode, reducing interference between them.
- When capacity approaches the watermark, enable queuing, rate limiting, and timeout-based degradation, prioritizing the completion of existing requests.
- Run the server side with container orchestration, configure health checks and autoscaling, and keep the inference layer stateless to make horizontal scaling and fast rollback easier.
5.4 Agent-Specific Safeguards
Agents can call external tools and execute multi-step tasks, so their risk surface is larger than that of ordinary chat applications:
- Enforce least-privilege access control for tool calls;
- Introduce human confirmation for high-risk actions;
- Isolate and validate model outputs and external content such as web pages and documents to mitigate prompt injection;
- Record end-to-end call logs to support auditing and troubleshooting.
6. Example Code: Memory Estimation and Acceptance Checks
6.1 Pre-Deployment Memory Estimation
The following code only demonstrates the estimation logic and does not include any framework-specific interfaces. Constants can be adjusted according to the actual model configuration:
def estimate_memory(total_params: float, # Total number of parametersactive_params: float, # Number of parameters activated per forward passlayers: int, # Number of layerskv_heads: int, # Number of KV headshead_dim: int, # Dimension per headweight_bytes: int = 2, # Bytes per parameter: BF16=2, FP8=1, INT4=0.5kv_bytes: int = 2, # Bytes per KV Cache elementconcurrency: int = 32, # Number of concurrent requestsctx_len: int = 8192, # Context length per requestoverhead: float = 1.2, # Overhead factor for activations and temporary buffers):"""Roughly estimate weight memory, KV Cache memory, and decode speed ceiling."""gib = 1024 ** 3weights_gib = total_params * weight_bytes / gibbytes_per_token = 2 * layers * kv_heads * head_dim * kv_bytes # One copy each for K and Vkv_gib = bytes_per_token * concurrency * ctx_len / gibpeak_gib = (weights_gib + kv_gib) * overheadread_bytes = active_params * weight_bytes # Weights actually read per tokenreturn {"weights_gib": round(weights_gib, 1),"kv_cache_gib": round(kv_gib, 1),"peak_gib": round(peak_gib, 1),"decode_tok_s_per_tb_bandwidth": round(1e12 / read_bytes, 1),}if __name__ == "__main__":result = estimate_memory(total_params=2.8e12, # Approximately 2.8 trillion total parametersactive_params=1.04e11, # Approximately 104 billion activated parameterslayers=80,kv_heads=8,head_dim=128,concurrency=32,ctx_len=8192,)for k, v in result.items():print(f"{k}: {v}")
With these parameters, the output is roughly as follows:
weights_gib: 5208.3kv_cache_gib: 20.0peak_gib: 6274.0decode_tok_s_per_tb_bandwidth: 4.8
This shows that if all weights remain resident in memory using BF16, peak memory usage is about 6.1 TiB, while the KV Cache for 32 concurrent requests with an 8K context is only about 20 GiB. Weights are the real capacity bottleneck. Switching to FP8 storage reduces weights to about 2.6 TiB; using group-wise 4-bit quantization reduces them further to about 1.3 TiB.
In addition, based on 1 TB/s of memory bandwidth, the estimated decode speed ceiling is only about 4.8 tokens/s. This means high throughput must rely on larger batches to amortize the cost of reading weights, and lower per-token latency requires higher aggregate memory bandwidth.
6.2 Production Acceptance Checklist
[ ] Functionality: Standard inference APIs work correctly, including streaming output and structured output[ ] Performance: Stress tests cover 1.5× peak concurrency, recording first-token latency and per-token output latency[ ] Accuracy: Scores on the business evaluation set are within an acceptable gap from the baseline, with item-by-item comparison before and after quantization[ ] Stability: The service runs continuously for 24 hours without memory leaks or restarts[ ] Fault tolerance: Traffic automatically shifts when a single node fails, and rate limiting/degradation works under overload[ ] Security: Input/output filtering, tool-call permission checks, and prompt-injection protection pass test cases[ ] Cost: Cost per request is calculated based on actual utilization and incorporated into capacity planning
7. Cost and Governance: From “Usable” to “Well Used”
The compute cost of trillion-parameter MoE models is in the same order of magnitude as dense models, but memory usage is higher. These models are well suited to businesses with high concurrency, heavy capacity needs, and domain customization requirements, such as intelligent customer service, code assistance, document understanding, and multi-step agents. For low-frequency, short-text, simple tasks, smaller and medium-sized models are often more economical.
From an engineering perspective, teams should do three things:
- Establish cost accounting by token, tenant, and business, preventing resources from being occupied for long periods by individual long-running tasks;
- Use model routing, sending simple requests to lightweight models and escalating only complex requests to the flagship model;
- Version-control weights, fine-tuning data, and prompts, while supporting canary releases and fast rollback so that the impact of each upgrade can be measured.
8. Conclusion
The release of these weights once again shows that open models are shifting from “single-point benchmark competition” to “engineering deployment competition.” For developers, the key is not to chase a particular leaderboard position, but to build a reusable evaluation and deployment methodology:
- Estimate compute using activated parameters;
- Estimate memory using total parameters;
- Validate effectiveness with real business data;
- Plan capacity using stress-test results.
Whether deploying privately on local infrastructure or using cloud platforms that provide compute, object storage, container orchestration, and monitoring/alerting capabilities, the goal should be to let model capabilities serve business objectives in a way that is measurable, operable, and scalable.
