Trillion-Parameter Open-Weight Model: A Practical Guide to Local Deployment

    After a team released the weights of a 2.8-trillion-parameter, native multimodal agent model, this guide covers the engineering questions that matter once the download finishes: what is actually inside the weights, whether the license allows your use case, whether an 8-GPU node can run it, and whether running it is worth the cost. It walks through the model architecture (896 routed experts plus 2 shared experts, 93 layers, MXFP4 weights, MXFP8 activations, and sparse attention on 24 layers), how to read the vendor's benchmark claims, the custom license, the real memory/bandwidth/cost math behind the "8-GPU minimum," parallelism and inference optimization, and a deployment checklist to help you choose between self-hosting, managed inference, and API access.

    1. What Was Released: Delivering on the Promise of Open Weights

    On November 4, a team fully published the weights of a native multimodal agent large model (referred to below as “the model”) as previously promised, and the vendor called it “the strongest open model you can download today.” But “downloadable” and “runnable” are two different things: the weights are about 1.56TB, and running them locally starts at 8 GPUs.

    The timeline looked like this:

    • October 24: The model first became available via API and a web interface, and the team promised to release the full weights before November 4;
    • November 2: The weight repository was created, and as of the morning of the 4th it was still closed;
    • Night of November 4: The repository opened officially, downloadable without any application, containing 112 files and 96 weight shards, totaling about 1.56TB.

    This is the largest open-weight model ever publicly released, far larger than any comparable release before it. The vendor already delivered an impressive scorecard at launch, and this article will not repeat that evaluation. Instead, it focuses on the real engineering questions that come after the weights land: what is actually inside, whether the license permits use, whether 8 GPUs can run it, and whether running it is worth it.

    2. Architecture: How 2.8 Trillion Parameters Are Organized

    Model Positioning and Core Parameters

    According to the model card, the model is a native multimodal agent model, further trained from the previous generation’s flagship model. Key parameters:

    Item Value
    Total parameters 2.8 trillion
    Activated parameters per token 104 billion
    Expert structure 896 routed experts + 2 shared experts, 16 selected per token
    Layers 93
    Context length 1M tokens
    Vision encoder Proprietary vision encoder (can understand images)
    Weight precision MXFP4 (micro-scaled 4-bit floating point)
    Activation precision MXFP8 (micro-scaled 8-bit floating point)

    The Only Structural Change: Sparse Attention

    Compared with the previous generation, the architecture changed in only one place: 24 layers of full attention were replaced with sparse attention (DSA).

    The compute cost of full attention grows quadratically with sequence length, and the prefill stage (the phase before the first token is generated) for million-token contexts easily becomes the bottleneck. Sparse attention lets each token attend only to a filtered subset of key tokens, reducing the compute and memory-access pressure of long contexts. However, only about a quarter of the layers were replaced, so this optimization does not eliminate all of the overhead of long contexts.

    Quantized Only: MXFP4 Weights + MXFP8 Activations

    The model is produced directly through quantization-aware training (QAT). Weights are stored in MXFP4, at about 0.5 bytes per parameter, plus a small amount of overhead for shared scaling factors and similar. Activations are computed in MXFP8 during inference, balancing memory footprint against precision loss.

    Note carefully: the vendor released only this one version, with no BF16 or FP8 variant. This means the inference framework must natively support this micro-scaling format. It also means that if you want to do downstream training such as supervised fine-tuning, you cannot start from an official high-precision set of weights; you can typically only do low-rank adaptation directly on these quantized weights.

    Two Kinds of Parameters, Two Cost Profiles

    An MoE model requires distinguishing between two kinds of parameters, whose cost profiles are completely different:

    • Total parameters (2.8 trillion): These set the “does it fit” threshold. No matter how few experts are activated at once, all these parameters must remain resident in memory, which determines the minimum hardware configuration;
    • Activated parameters (104 billion): These set the “can it compute” overhead. Each token passes through only 16 routed experts plus the 2 always-active shared experts, so the compute is roughly on the level of a dense model with 104 billion parameters, but the memory-bandwidth cost is still significant (see Section 6).

    In addition, because this is a native multimodal model, the vision encoder and its activations must also be counted in the memory budget. Long contexts further require reserving space for the KV cache. These two items are often underestimated and are a direct cause of many “the memory should be enough on paper but it won’t run” cases.

    3. Reading the Benchmarks: The Limits of the Vendor’s Scorecard

    The vendor published a set of comparison data spanning terminal coding, long-horizon software engineering, scientific reasoning, and hard knowledge reasoning. The conclusion: the model beat a leading international closed-source flagship model on 16 of 37 benchmarks, with strong coding and agent capabilities but weaker science and some reasoning abilities. One independent evaluation organization gave it a composite intelligence index of 55, ranking it fourth globally.

    A Total Score Can’t Answer “Is It Right for Me”

    All of these numbers are self-reported by the vendor, and there are at least three layers of interpretive bias:

    1. Systematic bias in self-reported benchmarks: Generation parameters, few-shot settings, and tool-calling methods are often chosen by the vendor itself. Even when done fairly, they only represent performance “under a specific configuration”;
    2. Single scores hide capability gaps: On science-reasoning benchmarks, the model trails the flagship model noticeably. If your work needs rigorous scientific computation or very hard reasoning, the total-score ranking has limited reference value;
    3. Third-party indices are composite scores: Different benchmarks carry different weights, and the same model can rank differently across evaluations. Rankings are a useful reference, but they cannot replace your own evaluation.

    The Right Approach: A/B Testing with a Business Eval Set

    The recommended approach is to build a small but realistic eval set close to your actual business:

    1. # Illustrative code: model A/B evaluation framework (choose dependencies yourself)
    2. import json, time
    3. def evaluate(requests, engine):
    4. """requests: [{"id","prompt","tools"}] covering real tasks such as coding, agent, and Q&A
    5. engine: wrapped inference interface that returns {"text","latency"}"""
    6. records = []
    7. for req in requests:
    8. t0 = time.perf_counter()
    9. out = engine(req["prompt"], tools=req.get("tools"))
    10. records.append({
    11. "id": req["id"],
    12. "output": out["text"],
    13. "latency": out["latency"],
    14. })
    15. return records
    16. def compare(records_a, records_b, judge):
    17. """judge: returns a 0~1 score using automated unit tests or human-defined rules"""
    18. sa = sum(judge(r["output"]) for r in records_a) / len(records_a)
    19. sb = sum(judge(r["output"]) for r in records_b) / len(records_b)
    20. print(f"Baseline: {sa:.3f} | Candidate: {sb:.3f}")
    21. # Also compare P50/P95 latency so you don't look at quality alone
    22. return {"baseline": sa, "candidate": sb}
    23. # Fix prompts, generation parameters, and tool definitions; use the exact same eval set for both sides

    When evaluating, fix the prompts, generation parameters, and tool definitions, and use the exact same eval set and scoring criteria on both sides. Beyond answer quality, also record P50/P95 latency and cost per token. Only by weighing all three can you support a selection decision.

    4. Licensing: Open Weights Are Not the Same as Open Source

    This is the most important point to discuss separately from this release: the repository is under a custom license, marked as an “other” license on the hosting platform, rather than a standard open-source license like MIT or Apache.

    Broad Core Terms, but Read the Appendices Closely

    The license is word-for-word identical to the previous generation’s flagship model license, only with the name changed. The main body adopts MIT-style permissive terms: free use, copying, modification, distribution, fine-tuning, and commercial use are all allowed. For the vast majority of enterprises and developers, this already covers common needs.

    But a “custom license” means you shouldn’t take anything for granted. Before deployment, carefully check the appendix terms, including:

    • attribution (credit and notice) requirements;
    • disclaimers and usage restrictions on generated content;
    • additional conditions for large-scale commercial use or offering a service to others;
    • how historical versions are traced if the license changes.

    These terms vary by version, so always rely on the license text shipped inside the weight repository.

    “Renamed but Unchanged” Calls for Extra Caution

    A license that changed its name but not its terms invites two kinds of misperception:

    • Assuming the new version carries over the old terms. In this case that is indeed true, but this is only an empirical inference, and every upgrade should be re-checked;
    • Treating permissive terms as zero obligation. Commercial use does not mean there are no additional responsibilities.

    A Note on Terminology

    “Open weights” only means “you can obtain the parameters”; whether it meets each of the freedoms in the open-source definition still depends on the specific license. In internal company communication and external materials, prefer the term “open-weight model” over “open-source model” to avoid compliance misunderstandings.

    5. Resource Math: Is 8 GPUs Really Enough?

    Start with Unit Conversions

    • 1.56TB = 1560GB ≈ 1.42TiB;
    • The vendor’s “8 GPUs” is a minimum threshold, meaning “can load and complete inference,” not “can stably serve at high concurrency.”

    The Memory Allocation Breakdown

    Based on the model architecture, a single instance’s memory consists roughly of four parts:

    Component Basis for estimate Order of magnitude
    Weights 2.8 trillion parameters, ~0.5 bytes/param in MXFP4, plus shared scaling factors ~1.42TiB
    KV cache Grows linearly with “concurrency × context length” Tens to hundreds of GiB
    Activations and vision encoding Per-batch forward intermediate tensors Tens of GiB
    Runtime overhead Communication buffers, memory fragmentation, framework reserve ~10%

    (Note: the figures above are order-of-magnitude estimates derived from public architecture parameters; validate against your actual deployment measurements.)

    Can 8 GPUs Carry It?

    Per-card memory 8-card total Can it fit the weights (1.42TiB)? Notes
    80GB ~0.58TiB No Needs 20+ cards and weight offloading enabled
    140GB ~1.02TiB No At least 12 cards, or fall back on host memory
    192GB ~1.41TiB Borderline short Needs 8+ cards and must rely on offloading
    200GB and above ≥1.46TiB Barely fits Headroom is only enough for small-batch inference
    1. # Illustrative code: estimate the minimum GPU count from per-card memory
    2. CARD_CAPACITY = 200 * 1024**3 # Per-card memory capacity (bytes)
    3. KV_BUDGET = 40 * 1024**3 # KV cache budget
    4. ACTIVATION = 20 * 1024**3 # Activations and vision encoding
    5. RESERVE = 0.10 # Reserve ratio for framework and fragmentation
    6. WEIGHTS = 1_420_000_000_000 # ~1.42TB weights (bytes)
    7. # Assumes balanced load across cards; actual demand is higher when expert routing is uneven
    8. usable = CARD_CAPACITY * (1 - RESERVE)
    9. gpus = -(-(WEIGHTS + KV_BUDGET + ACTIVATION) // usable)
    10. print(f"At least {gpus} GPUs") # Outputs 9 under the above assumptions

    The conclusion is clear:

    • Only when per-card memory reaches the 200GB class does an 8-card node barely “fit”;
    • At 192GB per card, the 8-card total is about 10GiB smaller than the weight size, so it cannot fit at all without offloading;
    • For common cards at 140GB and below, 8 cards can’t even hold the weights.

    So “8 GPUs to start” should be understood more as the minimum form factor for a single-node 8-GPU machine. In practice, choose a machine with 200GB or more per card, keep concurrency in a reasonable range, and reserve KV cache space for long-context scenarios.

    6. Deployment Paths: How to Choose Among Three Options

    Path Typical setup Best-fit scenario Main considerations
    Self-hosted Single-node 8-GPU large-memory machine / multi-node cluster Data cannot leave the domain, stable high concurrency If utilization falls below the break-even point, it loses money continuously
    Managed inference Platform-hosted service for the open model Bursty traffic, short-term validation Watch context limits, concurrency quotas, and data policy
    API calls Vendor’s official interface Fast integration, low-frequency use High availability, but cost and data compliance depend on the terms of service

    The three paths can be combined: use the API to quickly validate results, use managed services to absorb bursty traffic, and migrate to a self-hosted cluster once utilization stabilizes.

    On Baidu AI Cloud, you can combine general-purpose capabilities such as GPU cloud servers and container services to build a self-hosted inference cluster, then pair them with object storage, logging, and monitoring/alerting to close the loop from weight hosting to runtime observability.

    The Core Formula for the Cost Decision

    Whether self-hosting pays off depends on whether resource utilization can cross the break-even point:

    1. Self-hosted unit cost = GPUs × per-GPU hourly cost × 24 × 30 / monthly effective output tokens
    2. API unit cost = input token price × input volume + output token price × output volume

    Take an 8-GPU machine as an example: the GPU resources are essentially billed around the clock, while business traffic usually has valleys. Only when utilization is high enough will the self-hosted cost fall below the API cost. First measure your historical peak-to-valley ratio and average daily token volume, then decide the card count and path.

    7. Parallelism Strategy and Inference Optimization

    Parallelism: Expert Parallelism Takes the Lead

    • Expert parallelism (EP): Distribute the 896 experts across cards. When routing is uneven, some cards overload while others idle, so pair it with a capacity factor or load-balancing strategy;
    • Tensor parallelism (TP): Split the attention layers and the 2 shared experts. The shared experts are always active and compute-intensive, so TP is a good fit for spreading that load further;
    • Single-node first: The all-to-all communication of expert parallelism is extremely sensitive to interconnect bandwidth, so prefer a single-node 8-GPU machine over an ad-hoc multi-node setup;
    • Cross-node deployment: If you use multiple nodes to get past the memory limit, rely on high-speed networking and overlap communication with computation to reduce communication overhead.

    In addition, the 24 sparse-attention layers and the remaining dense layers must be included in the compute graph at the same time, so the inference engine must support both attention kernels simultaneously. Verify the target framework’s model support list and operator coverage before deployment.

    Performance Optimization Checklist

    1. Continuous batching: Combined with paged KV management, let requests of different lengths share memory and be batched together dynamically;
    2. Context tiering: The default context doesn’t need to be maxed out at 1M; set a limit per business. The longer the context, the more KV cache it consumes;
    3. Prefix caching: Reuse KV for common prefixes such as system prompts, tool definitions, and long documents to lower time to first token;
    4. Chunked prefill: Split a long prompt into small chunks and interleave its scheduling with the decoding of other requests, so a long request doesn’t block short ones;
    5. Decode is bandwidth-bound: Every generated token requires reading the weights activated in that round, on the order of:
    1. 104B × 0.5125 bytes/param (incl. scaling factors) ≈ 55GB/token
    2. 8 GPUs aggregate memory bandwidth ≈ 27TB/s → theoretical ceiling ≈ 470 tokens/s

    Real throughput must also subtract communication and scheduling overhead. This shows that MoE saves compute, not bandwidth, which is also why “only 104 billion activated parameters” does not mean a single card can run it.

    Monitoring Metrics

    At minimum, monitor the following and set reasonable alert thresholds:

    • Memory usage: Break it down into weights, KV cache, and temporary activations to prevent long contexts from triggering OOM;
    • Expert load distribution: Detect routing skew;
    • Time to first token (TTFT): Mainly affected by prefill and queueing;
    • Per-token generation latency (TPOT): Mainly affected by memory-access bandwidth;
    • Throughput (tokens/s) and batch size: Assess resource efficiency;
    • Per-card compute utilization: If it stays low while bandwidth is saturated, the bottleneck is memory access rather than compute.

    8. Deployment Checklist

    1. Download and verify: Check the file count and shard hashes to confirm nothing is corrupted or missing;
    2. License review: Have legal review the license text and appendix terms to clarify the boundaries of commercial use and redistribution;
    3. Environment preparation: Confirm the inference framework supports this quantization format and sparse attention, and match the corresponding versions of the runtime and operator libraries;
    4. Memory budget: Calculate the card count using the formulas in Section 5 and reserve space for the KV cache;
    5. Baseline testing: Record TTFT, TPOT, throughput, and memory usage;
    6. Quality evaluation: Do an A/B comparison against your existing solution using a business eval set;
    7. Gradual rollout: Start with a small traffic share, ramp up gradually, and set a rollback plan.

    Conclusion

    The significance of this weight release lies not in a ranking, but in the fact that it puts an ultra-large-scale MoE model into developers’ hands in complete form for the first time. 2.8 trillion parameters, 1.56TB of weights, MXFP4 quantization, and sparse attention together define both the capability ceiling for coding and agent scenarios and a clear hardware threshold: 8 GPUs are only the entry ticket — the memory bill, the bandwidth bill, and the cost bill are the three accounts that really need to be worked out.

    For a development team, the sensible path is: first understand the architecture and license, then work out the resource math, validate results with a business eval set, and finally choose the most economical combination among self-hosting, managed inference, and API access.