Local Deployment of Trillion-Parameter MoE Open-Weight Models: A Complete Engineering Guide
A 2.8-trillion-parameter MoE agent model with open weights has been publicly released, reigniting the debate over whether it is truly "open source." Instead of arguing labels, this article breaks the question into concrete engineering judgments. It dissects the model's Mixture-of-Experts architecture, sparse attention, and quantization-aware training; shows how to read vendor-reported benchmarks; clarifies the difference between open weights and open source along with the constraints of the custom license; and works through the VRAM math, GPU sizing, parallelism strategies, and three deployment scenarios. It closes with a risk list and a decision checklist to help teams determine whether local deployment is worth the investment.
Introduction: When “Open Weights” Becomes a Systems Engineering Problem
On the evening of November 4, a model R&D team released the full weights of an agent model with roughly 2.8 trillion parameters on an open model hosting community, just as it had previously promised. According to its model card, the model supports image input and million-token context, making it the largest publicly downloadable model to date. Almost at once, two camps emerged: those saying “it can run locally too” and those insisting “this isn’t open source at all.”
Rather than stay stuck on the “open source or not” argument, it is more useful to break the question into three engineering judgments: what it can do, where its limits are, and what local deployment requires. The sections below analyze the model structure, capability benchmarks, license terms, VRAM ledger, and deployment architecture in turn, and end with a decision checklist.
1. From API to Weights: The Event and the Boundary of “Open Source”
The timeline is straightforward. In late October, the model first launched as an API and a web interface, and the team promised to release the full weights before November 4. On November 2, the weight hosting repository was created. On the morning of November 4 it was still closed; that evening it officially opened. The weights comprise 112 files, including 96 safetensors shards totaling about 1.56TB, downloadable directly without any application, with other community mirrors also providing synced copies.
Two points are worth noting. First, “delivering on a promise on schedule” is itself informative in a release cadence where models routinely slip by quarters. Second, and more importantly: what was released are the weights—all the numerical parameters obtained after training—not “open source.” The two terms are often used interchangeably, but they differ substantially in engineering and legal contexts:
| Dimension | Open Weights | Open Source |
|---|---|---|
| What you get | Model parameters, configuration, and the files needed for inference | Parameters + full training code + training data (or a way to obtain it) + license and build instructions |
| Can you self-host? | Yes | Yes |
| Can you fine-tune / distill / redistribute? | Depends on the license terms | Depends on the license, but usually governed by permissive terms |
| Can you reproduce training? | No; in general you can only continue pre-training or fine-tune | In principle, yes |
| Typical license form | Custom license with added usage-scope or commercial-scale restrictions | Common permissive licenses (e.g., MIT, Apache-style) |
The prevailing open-source definition emphasizes three points: source code must be obtainable, the freedom to modify and redistribute must be guaranteed, and the license must not discriminate against specific use cases or groups. Measured against that standard, the added clauses with scale thresholds mean these weights are not “open source” in the legal sense. As the sections below show, this new entrant has its own answer for every column in that table.
2. Model Anatomy: Where the 2.8 Trillion Parameters Come From
2.1 MoE: Total Parameters Are Not the Same as Compute
The model uses a Mixture-of-Experts (MoE) architecture. Its key structural parameters are:
| Item | Value |
|---|---|
| Total parameters | ~2.8 trillion |
| Activated parameters per token | ~104 billion (~3.7% of the total) |
| Expert configuration | 896 routed experts, 16 selected per token; plus 2 shared experts |
| Layers | 93, of which 24 use sparse attention in place of full attention |
| Context length | 1,000,000 tokens |
| Vision encoder | A proprietary vision encoding module supporting image input |
| Training origin | Continued training from the previous generation |
The idea behind MoE is to split the feed-forward layers of a dense Transformer into a large number of “expert” sub-networks, with a gating network deciding which experts to invoke per token. Here each token routes through 16 routed experts plus 2 shared experts: the shared experts carry the general computation that every token needs, while the routed experts provide specialized capability.
This design yields a rule that is easy to overlook: total parameters decide “whether it fits,” while activated parameters decide “how fast it computes.” The 2.8 trillion sets the VRAM floor; the 104 billion sets the compute demand. By activated volume, its online compute is roughly comparable to a hundreds-of-billions-scale dense model. But expert routing can make some experts hotspots while others sit idle, so measured throughput also depends on load-balancing strategy, batching, and expert-parallel efficiency.
2.2 Sparse Attention: A Second Compute Ledger for Long Context
Beyond MoE, the model structure differs from the previous generation in only one place: 24 of the 93 layers switch from full attention to sparse attention.
Full-attention compute scales quadratically with sequence length. At million-token context, this overhead quickly dominates the entire inference process, and the first thing to hit a wall is often not the GPU’s compute (FLOPS) but memory bandwidth and capacity. Sparse attention no longer has every token interact with all historical tokens; instead, some selection strategy limits each token to attending to only a portion of them. It trades a little precision for a large drop in compute, and is a general technique for cutting the cost of long-context models.
But this ledger is incomplete. Sparse attention saves the compute cost of attention; the VRAM occupied by the KV cache does not disappear along with it, unless the model additionally compresses the cache via cross-layer KV sharing, linear attention, and similar structures. A hybrid sparse design also means different layers may need to take different compute paths, so the actual speedup depends heavily on the specific implementation. For long-context scenarios, this is a point that must be validated under load testing.
2.3 Quantization-Aware Training: The Trade-off of Shipping Only One Version
The weights use the Microscaling (MX) floating-point format: weights in 4-bit floating point (MXFP4) and activations in 8-bit floating point (MXFP8). Both are formats defined by an open standard. Microscaling divides values into small blocks, each sharing a scaling factor, with elements within a block represented in low-bit-width floating point. Compared with per-tensor quantization, this adapts better to MoE, where numerical ranges vary widely across different experts.
The team shipped only this one version, with no higher-precision or 8-bit weight release. Behind this is Quantization-Aware Training (QAT): the model simulates rounding and truncation errors at low bit-widths during training, learning to preserve its performance at low precision rather than being “compressed after the fact” post-training.
This route has clear trade-offs:
- Advantages: VRAM and bandwidth usage are roughly half that of 16-bit weights, and inference cost and per-token energy consumption drop accordingly; because it is adapted during training, the precision loss is typically smaller than post-training quantization.
- Cost 1: The precision ceiling is locked in by the 4-bit representation. For error-sensitive tasks such as numerical computation and long chains of reasoning, regression validation is required—you cannot assume equivalence with a higher-precision version.
- Cost 2: There is no high-precision fallback. For strictly regulated domains such as finance and healthcare, if internal policy requires running at a specific precision, this model fails the requirement outright and must be ruled out in advance.
3. Capability Benchmarks: How to Read Vendor Self-Reported Numbers
3.1 Overview: Strong at Agents and Coding, Weak at Scientific Reasoning
A note on methodology first: every figure in the table below is vendor self-reported and has not been independently reproduced by a third party. The comparison target is a leading international closed-source flagship model (referred to below as “the benchmark model”).
| Benchmark | Focus | This model | Benchmark model | Relative result |
|---|---|---|---|---|
| Terminal-based coding benchmark | Completing coding tasks in a real terminal environment | 65.7% | 66.4% | Slightly lower |
| Long-horizon software engineering benchmark | Cross-step, long-duration engineering fixes | 51.7% | 46.3% | Leads by ~5.4 points |
| Another closed-source frontier model series | Terminal coding | 58.2% | — | This model leads |
| High-difficulty multidisciplinary QA benchmark (science subset) | Science problems requiring deep reasoning | 38.1% | 63.3% | Clearly behind |
| High-difficulty multidisciplinary QA benchmark | Cross-disciplinary hard problems | 39.4% | 60.9% | Clearly behind |
A third-party independent evaluator also gave the model a composite intelligence index of 55, placing it fourth globally.
The conclusion can be summed up in three sentences:
- Structure matches capability. Agent capabilities such as coding, terminal operation, and long-horizon task repair are close to—or in some cases exceed—the international flagship.
- The gaps are obvious. On high-difficulty science and reasoning tasks the gap exceeds 20 percentage points, a clear weakness.
- Headline counts are misleading. The team states that 16 of 37 benchmarks beat the benchmark model, but a “number of wins” hides the size of the gaps. The shortfall on weak items may be far larger than the lead on strong ones.
3.2 Three Interpretation Traps
Good-looking leaderboards do not equal business readiness. Watch for three things when reading vendor self-reported data.
First, benchmark contamination. Public problems and reference answers on the internet have very likely already entered the training corpus. The same model may perform very differently on public leaderboards versus private problems.
Second, evaluation protocol and scaffolding differences. Agent benchmarks are extremely sensitive to prompt templates, tool-calling methods, retry logic, and result-verification logic. Swap the evaluation scaffolding and the same model’s score can differ by several points, yet vendors typically do not publish the full evaluation code and configuration.
Third, sampling and variance. There are only a limited number of hard problems, and fluctuations of around 1% often fall within the influence of random seeds, temperature parameters, and sampling counts (pass@1 vs. pass@k). A single run is not enough to serve as a basis for decisions.
3.3 A More Reliable Approach: Let Your Business Benchmarks Speak
Teams should build their own evaluation set that covers at least the following dimensions:
| Dimension | Example metrics | Notes |
|---|---|---|
| Task completion | End-to-end success rate, human-takeover rate | Closest to business value |
| Tool-calling ability | Argument accuracy, multi-turn error correction | Core to agent scenarios |
| Long context | Needle-in-a-haystack hit rate, multi-document synthesis accuracy | Validates the real payoff of million-token context |
| Long-horizon stability | Performance decay curve across steps | Judges whether it can handle complex tasks |
| Hallucination rate | Spot checks on factual consistency | The bottom line for high-risk scenarios |
| Safety and compliance | Jailbreak success rate, sensitive-content block rate | Must be verified before launch |
| Engineering metrics | Time to first token, throughput, per-request cost | Determines capacity and budget |
When evaluating, be sure to fix the temperature, random seed, and prompt template, and to keep complete request logs so that conclusions are reproducible.
4. License: Open Weights Are Not Open Source
4.1 What the License Text Actually Says
The repository ships a custom license (labeled “other/custom” on the hosting platform), not a standard open-source license like MIT or Apache. According to a public summary of the license text, its structure is word-for-word identical to the previous generation’s license, with only the name changed.
The main body is indeed close to an MIT-style grant: it permits free use, copying, modification, distribution, fine-tuning, and commercial use. On top of this, it adds two restrictions that target only large-scale external business:
- Revenue threshold: A company that offers this model externally as a Model-as-a-Service (MaaS) offering and whose trailing 12-month cumulative revenue exceeds USD 20 million must sign a separate agreement with the R&D team before commercial use.
- Monthly active user threshold: A commercial product with more than 100 million monthly active users likewise needs to obtain separate authorization.
Both numbers point somewhere specific:
- USD 20 million in revenue is roughly what a mid-sized SaaS company earns in a year—right at the point of “moving from startup to scale.” The intent is clear: small-scale commercial use is entirely free, while monetization at scale requires negotiation.
- 100 million MAU is extremely rare among users of open-weight models; the primary targets of this clause would be super-apps on large platforms.
As a result, most small and mid-sized teams face no real obstacle in using these weights. The genuine restrictions fall on two types of actors: service providers that resell model capabilities via API and already earn in the tens of millions of dollars annually, and platform operators with enormous user bases. For them, the issue to solve before commercial use is not technical but a matter of license negotiation and legal review.
4.2 Enterprise Compliance Checklist
Before adoption, the legal and architecture teams should jointly confirm the following:
- Whether the use case touches a threshold: Are you offering MaaS externally? Is projected 12-month revenue approaching USD 20 million? Does the product reach the 100-million-MAU level?
- Redistribution boundaries: Can you package fine-tuned weights into your own product or service and distribute them externally?
- Output ownership: Who holds the copyright to model-generated content, and how is liability allocated?
- Naming and branding of derivatives: Is source attribution required? Are there restrictions on using the original model’s name or trademarks?
- Export control and cross-border data: For cross-border deployment, assess both compute-hardware controls and data-transfer requirements.
- Change tracking: A custom license may be modified; lock the hash of the downloaded version and retain a snapshot of the license as audit evidence.
4.3 Should You Choose It, or a Truly Open-Source Model?
If what the team values is “complete freedom with no added conditions,” choose a model under a standard permissive license. If what it values is “the strongest current capability, with acceptable commercial constraints,” then this model is a reasonable candidate—provided the compliance assessment is completed in advance.
5. Hardware Ledger: Why Local Deployment Starts at 8 GPUs
5.1 Three VRAM Ledgers
The feasibility of local deployment is, first of all, a VRAM arithmetic problem. VRAM must simultaneously hold three parts:
- Weights: on the order of ~1.4TB;
- KV cache: proportional to context length, batch size, and cache precision, and able to balloon quickly under long context;
- Runtime overhead: activations, temporary buffers, communication buffers, framework reservations, and so on, usually reserved as a certain percentage of total VRAM.
5.2 Let the Code Do the Math
The script below estimates the number of GPUs required (illustrative code; you must add the imports and handle division by zero yourself):
MODEL_BYTES = 1.56e12 # Total bytes of the weight files (decimal)def required_gpus(weights_bytes, gpu_mem_gb, kv_budget_ratio=0.2):"""Estimate the minimum number of GPUs needed to hold the weights.gpu_mem_gb : Per-GPU VRAM (GB, decimal, 1GB = 1e9 bytes)kv_budget_ratio: Fraction of VRAM reserved for KV cache and runtime overhead"""budget = 1 - kv_budget_ratiousable = gpu_mem_gb * 1e9 * budgetreturn math.ceil(weights_bytes / usable)def kv_cache_bytes(num_layers, kv_heads, head_dim, seq_len,batch=1, bytes_per_elem=1):"""Estimate KV cache VRAM: K and V are cached per token, per layer."""return 2 * num_layers * kv_heads * head_dim * seq_len * batch * bytes_per_elemfor gpu_mem in (80, 141, 192, 288):print(f"Single GPU {gpu_mem:>3}GB -> at least {required_gpus(MODEL_BYTES, gpu_mem)} GPUs")
Using the conservative assumption that “KV cache and runtime take at least 20% of VRAM,” the results are roughly as follows:
| Single-GPU VRAM | Usable per GPU | Min. GPUs required | Practical recommendation |
|---|---|---|---|
| 80GB | 64GB | 25 | Needs 4+ nodes; network becomes the main bottleneck |
| 141GB | ~112.8GB | 14 | Two nodes with 7+ GPUs each; significant inter-node communication overhead |
| 192GB | 153.6GB | 11 | Two-node setup; still needs high-speed interconnect |
| 288GB | 230.4GB | 7 | A single 8-GPU node is more than enough |
This table explains the hardware threshold behind the “8 GPUs to start.” Splitting 1.56TB of weights across 8 GPUs means each GPU must hold about 195GB on average, which implies per-GPU VRAM on the order of at least 200GB—and that class of hardware is itself scarce and expensive to procure. Ordinary 80GB-class cards would require 20-plus units, and they would need to be networked across nodes. So the question of “can an ordinary person run it” comes down to what hardware the “ordinary person” has, and what “run” actually means.
The calculation above only guarantees that “the weights fit.” Long-context scenarios require a second calculation for the KV cache, for example:
# Using 93 layers and assuming 8 KV heads with head_dim 128 (illustrative# assumptions only; refer to the actual model configuration)num_layers, kv_heads, head_dim = 93, 8, 128for seq_len in (32_000, 128_000, 1_000_000):for prec, bpe in (("8bit", 1), ("4bit", 0.5)):gb = kv_cache_bytes(num_layers, kv_heads, head_dim,seq_len, bytes_per_elem=bpe) / 1e9print(f"{seq_len:>7} tokens | KV {prec} | ~{gb:.1f} GB per sequence")
With an 8-bit cache, a single sequence needs about 5.8GB, 23GB, and 181GB of VRAM at 32K, 128K, and 1M tokens of context respectively; a 4-bit cache halves those. In other words, a genuine million-token request could, on the KV cache alone, consume VRAM equivalent to two or three 100GB-class cards. “Supports million-token context” is a capability ceiling, not a service guarantee. With multiple concurrent sequences this overhead multiplies further—which directly explains why the next section calls for P/D separation.
5.3 Parallelism Strategy: How to Shard the Model
Mainstream inference frameworks typically offer several parallelism methods, and real deployments usually combine them:
- Tensor parallelism (TP): Splits each layer’s weight matrices across multiple GPUs, with communication required every layer. Because communication is frequent, it demands high bandwidth and low latency, and is best confined within a single machine’s high-speed interconnect.
- Expert parallelism (EP): Distributes the 896 experts across different GPUs, with tokens dispatched cross-GPU according to routing results. For MoE this usually scales better than TP, but it depends on inter-node network bandwidth and latency.
- Pipeline parallelism (PP): Splits the model into segments by layer and places them on different GPUs or nodes. Communication volume is small, making it suitable across nodes, but it introduces pipeline bubbles that lower utilization.
A typical single-machine 8-GPU configuration is TP=8, keeping the whole model resident inside one machine and avoiding any cross-node communication. If constraints force a two-node deployment, TP is usually kept within the number of GPUs in one machine, the nodes are connected via expert parallelism or pipeline parallelism, and RDMA-class high-speed networking between nodes should be ensured.
6. Deployment Options: How to Choose Among Three Scenarios
6.1 Scenario A: Capability First, Use a Hosted API
The need is “get this model into use as soon as possible,” but the bar for self-hosting is too high. This is essentially the choice between “buying only the weights” and “buying a service.” When calling a model of comparable scale through a cloud provider’s AI platform service, pay particular attention to:
- The Service Level Agreement (SLA) and rate-limiting policy;
- Whether the data is used for training;
- The billing basis (pricing separately for input/output tokens);
- Context limits and concurrency limits.
The advantage is zero hardware investment, elastic scaling, and pay-as-you-go. The cost is that data must leave your premises, so data-sensitive scenarios need careful evaluation.
6.2 Scenario B: Data Stays In-House, Private Deployment
The need is that data cannot leave the internal network, or that deep customization and fine-tuning are required. Taking a single 8-GPU machine as an example, a common service configuration looks like this:
# Illustrative config; field names differ across inference frameworksmodel: /data/models/moe-2.8ttensor_parallel_size: 8 # Single-machine 8-GPU tensor parallelismquantization: mxfp4 # Keep the official 4-bit weightskv_cache_dtype: fp8 # Drop KV cache to 8-bit, essential for long contextmax_model_len: 32768 # Set to actual business needs; do not max it out blindlygpu_memory_utilization: 0.90 # Leave room for the KV cacheenable_chunked_prefill: true # Chunked prefill for long prompts, easing long-request blocking
A few rules of thumb:
- Set context length on demand. Every additional context tier raises the KV cache reservation significantly, and maxing it out blindly can cause concurrency to fall off a cliff.
- Enable cache reuse for repeated prefix content such as prompts and tool definitions, which can substantially reduce prefill computation.
- Turn on runtime logs and monitoring metrics to leave a trail for tuning.
6.3 Scenario C: High-Concurrency Production
When QPS (queries per second) climbs into the tens or even hundreds, a single machine is no longer the constraint, and the architectural focus shifts to prefill/decode (P/D) separation.
The prefill stage processes the entire input prompt and is compute-intensive; the decode stage generates tokens one at a time and is memory-access-intensive. The two workloads have very different hardware needs, and mixing them causes interference: prefilling a long prompt can block other requests’ output, causing time-to-first-token jitter. After separating them:
- Prefill node group: Handles long-context inputs;
- Decode node group: Handles the KV cache and token-by-token generation;
- Routing layer: Schedules requests to different node groups based on request length and cache-hit status.
At the same time, introduce an asynchronous task queue and result caching, turning the “long thinking” phase common in agent scenarios into asynchronous execution. The online layer can also deploy a smaller, faster model for intent routing, forwarding only complex tasks to the large model. This layered architecture can significantly reduce the load on the large model.
This class of architecture can be built on general-purpose cloud capabilities: object storage (for weights and artifacts), container platforms (elastic scheduling of inference instances), message queues (asynchronous task decoupling), and logging plus monitoring/alerting (observability). Cloud platforms such as Baidu AI Cloud offer the above product categories and can serve as one reference option; the specific selection should be decided based on actual needs and cost estimates.
6.4 Quantization Trade-offs and Pitfalls
- Do not use more aggressive quantization to force-fit a single GPU. The official weights are already a QAT product, and further compression tends to amplify errors.
- Do not sacrifice concurrency for context length. Most businesses genuinely need a window far smaller than a million tokens.
- Do not ignore uneven expert load. Long-tail experts may become “storage black holes” (occupying VRAM yet rarely called), so monitor routing distribution.
- Do not skip regression testing. After changing versions, frameworks, or quantization parameters, rerun a full pass on your business evaluation set.
7. Risk List and Decision Checklist
Risk list:
- License risk: Whether the custom license covers your scaling path requires legal assessment up front.
- Capability-mismatch risk: Science and deep reasoning are weak spots; do not place this model in the chain of high-risk decisions.
- Hardware-availability risk: Supply of 200GB-class VRAM cards is limited; plan procurement and leasing timelines ahead of time.
- Cost risk: The electricity and depreciation of occupying 8 GPUs full-time may far exceed API costs.
- Version-drift risk: The team ships only one quantized version, so precision changes when upgrading to a new version must be re-validated.
Decision checklist:
- Is the business scenario mainly coding, agents, and tool calling? If so, capability fit is high.
- Does it involve scientific computing and deep reasoning? If so, add supplementary evaluation or consider another model.
- Must data stay within the internal network? If so, lean toward private deployment.
- Could the relevant 12-month revenue approach USD 20 million? If so, contact them for authorization in advance.
- Can you obtain compute of the form single-GPU 200GB-class VRAM × 8? If not, evaluate hosted API or a multi-node approach.
- Have you built your own evaluation set and regression process? If not, build the evaluation set before selecting a model.
Conclusion
The significance of this weight release lies not in the word “open source” itself, but in turning the “strongest open-weight model” from a marketing claim into an engineering object you can measure with calipers. 2.8 trillion parameters, 104 billion activated, 1.56TB of files, 8 GPUs to start, a custom license—every number is saying the same thing: the capability is real, and so is the barrier.
For most teams, the rational path is to first validate value with a hosted service, then confirm fit with your own business evaluation set, and finally—once data compliance, cost, and license terms all add up—decide whether to invest in a private cluster. Technology selection is never about “who is strongest,” but “who is the best fit.”
