2.8T-Parameter Open-Weight Model: A Complete Guide to Local Deployment

    An overview of a Chinese team's release of a 2.8T-parameter open-weight model (~1.56TB), and a practical look at the three things that actually matter after release: what is inside the weights, how the custom license works, and whether the model can run locally. It covers the sparse MoE and sparse attention architecture, a framework for reading vendor self-benchmarks, license obligations for large-scale use, a memory/compute/bandwidth budget showing why 8 GPUs is the practical floor, and a single-machine 8-GPU deployment checklist with risk and decision guidance.

    1. What Happened: A Trillion-Scale Weight Release, Right on Schedule

    On November 4, a Chinese model team released the full weights of an ultra-large-parameter model, as previously promised: 2.8T total parameters, roughly 1.56TB download size, described by the team and the press as “the strongest open-weight model currently available for download.” But this openness comes with two practical conditions—a commercial threshold written into a custom license, and a local deployment floor of 8 GPUs to start.

    The timeline was roughly:

    • October 24: The model first went live via API and a web interface, and the team promised to release the full weights before November 4;
    • November 2: The model repository was created on the hosting community;
    • November 4: The repository was still closed that morning, then formally opened that evening, with no application required and direct download available;
    • The repository holds 112 files, of which 96 are sharded weight files (the mainstream zero-copy tensor sharding format), totaling roughly 1.56TB.

    This is by far the largest set of public weights released to date—more than an order of magnitude larger than any prior open model. Rather than repeat the benchmark hype from launch week, this article focuses on the three things that actually matter once the weights land: what is inside, how the license works, and whether it can run.

    2. Breaking Down 2.8T: The Engineering Tradeoffs of Sparse MoE and Sparse Attention

    According to the official model card, this is a natively multimodal, agent-oriented model continued-training from the previous generation. The key architecture parameters can be summarized in a table:

    Dimension Configuration Engineering implication
    Total params / activated per token 2.8T / 104B Provision memory for 2.8T; estimate compute by 104B
    MoE expert config 896 experts, top-16 routing + 2 shared experts High sparsity; expert parallelism and load balancing are critical
    Layers 93 Long-context KV cache scales roughly linearly with depth
    Context length 1M tokens Prefill compute and KV memory pressure amplified significantly
    Vision encoder Proprietary Image+text input needs an extra image preprocessor and tokenizer config
    Weight/activation precision Weights 4-bit (MXFP4), activations 8-bit (MXFP8) Product of quantization-aware training, not post-training quantization

    Compared with the previous generation, only one thing changed in the architecture: 24 of the layers swapped full attention for a sparse attention mechanism, intended to compress the quadratic complexity of attention under million-token context. This deserves emphasis. At 1M context, KV reads/writes and the attention matrix dominate per-card bandwidth and compute; sparsifying lowers the marginal cost of long-text scenarios but introduces a new question—whether information coverage within the sparse window is complete—which needs dedicated validation on long-range retrieval tasks.

    Another fact to note: the team released only a single 4-bit weight version, with no BF16 or FP8 variant. This directly dictates deployment strategy—either rely on hardware and inference frameworks that natively support the 4-bit block quantization format (including block scaling factors), or dequantize to higher precision yourself, at the cost of inflating resident weight memory from about 1.5TB to the 5.6TB range (BF16). In other words, “can it run” is less a model problem than a question of whether the software stack supports this quantization format.

    3. How to Read the Vendor Self-Benchmarks: A Reusable Framework

    The comparison tables the team published are all self-reported and should be understood in layers of “verifiability.” The cleaned-up results are below (comparison models labeled A/B):

    Evaluation This model Model A Model B
    Terminal-operation benchmark 4.0 65.7% 66.4% 58.2%
    Long-horizon software engineering benchmark 51.7% 46.3% —
    Scientific terminal benchmark 38.1% 63.3% —
    High-difficulty comprehensive knowledge reasoning benchmark 39.4% 60.9% —

    The team’s framing is that the model beats Model A on 16 of 37 benchmarks; a third-party independent evaluation outfit gives it a composite intelligence index of 55, ranking fourth globally. Spread the table out, though, and the results are decidedly mixed: coding and agent tasks are close to or ahead of the competition, while scientific computing and some high-difficulty reasoning lag clearly behind.

    A more rigorous reading follows four principles:

    1. Separate by task type. Terminal operation and long-horizon code changes are “multi-step tool-calling + verifiable-feedback” tasks, where sparse MoE plus long context tends to pay off; scientific computing and competition-level reasoning depend more on depth of single-step thinking, where architectural sparsity brings no inherent advantage.
    2. Watch for self-report bias. Prompt wording, sampling temperature, max steps, retry policy, and whether self-verification is allowed all materially affect agent scores. A score published without its evaluation scripts and parameter settings is only an upper-bound reference.
    3. Look at the absolute capability gap. A gap on the order of 38.1% vs 63.3% is not something you can tune away; for technology selection, treat the corresponding capability as “not available.”
    4. Build a private evaluation set. General leaderboards do not represent your business distribution. At minimum, cover task success rate (pass@k), output format compliance, tool-call success rate, long-context retrieval hit rate, and regression stability.

    4. License: Open Weights ≠ Open Source, and Here It Especially Matters

    This is the most easily misread part. The repository’s license is labeled “other/custom,” pointing to a model license agreement owned by the team rather than a standardized open-source license like MIT or Apache. The agreement is word-for-word identical to the previous generation’s license terms, with only the name changed.

    The core is indeed MIT-style: it permits free use, copying, modification, distribution, fine-tuning, and commercial use. But two restrictions aimed at large-scale external business sit on top:

    1. If you offer the model “as a service” and rolling 12-month revenue exceeds $20 million, you must sign a separate agreement with the team before commercial use;
    2. If a commercial product reaches the order of a billion monthly active users, you likewise need prior written permission (exact thresholds and wording per the license text).

    The substance of these two limits is “tiered licensing”: small- and mid-scale commercial use is fully free, while very large traffic-driven/platform businesses go back to bilateral negotiation. Practical guidance for engineering and legal:

    • Maintain a license ledger. Record model version, weight hash, license version, and download date to prevent traceability problems if the license later changes.
    • Define what counts as “MaaS.” Whether API resale, subscription bundles, or free-tier acquisition count needs to be defined up front.
    • Define revenue-consolidation rules. The 12 months is a rolling window; whether affiliate revenue is consolidated needs to be settled.
    • Define the MAU counting method. Deduplication, statistical period, and cross-platform consolidation rules.
    • Retain attribution and license text. Ship the license and a NOTICE-type file with the weights on redistribution.
    • Constrain derivative models. When fine-tuned weights are treated as a derivative work, naming, distribution channels, and scope of re-licensing should be agreed on together.
    • Regional and export compliance. Availability in different regions must be checked separately; do not equate license terms with a compliance conclusion.

    5. Can It Run Locally? The Memory Math from 1.56TB to “8 GPUs to Start”

    The “8 GPUs to start” number is a simple addition problem. A rough estimate (illustrative code, for order-of-magnitude judgment only):

    1. # 1) Resident weight memory: 4-bit + block scaling factors
    2. TOTAL_PARAMS = 2.8e12
    3. BYTES_PER_PARAM = 0.5 # MXFP4
    4. SCALE_OVERHEAD = 1.0625 # extra overhead from block scaling factors
    5. weights_gb = TOTAL_PARAMS * BYTES_PER_PARAM * SCALE_OVERHEAD / 1e9
    6. print(f"Resident weight memory ≈ {weights_gb:.0f} GB") # ≈ 1490 GB ≈ 1.49 TB
    7. # 2) Minimum per-card memory for a single 8-GPU machine
    8. per_card = weights_gb / 8
    9. print(f"Per card needs at least ≈ {per_card:.0f} GB (excluding KV cache and runtime overhead)")
    10. # ≈ 186 GB/card → in practice 192GB-class cards, with almost no headroom

    The conclusion is clear: to actually fit into 8 cards, per-card memory must be in the 192GB range, at which point almost all remaining space gets eaten by the KV cache. The KV cache estimate is:

    1. def kv_cache_gb(layers, kv_heads, head_dim, tokens, bytes_per_elem):
    2. per_token = 2 * layers * kv_heads * head_dim * bytes_per_elem # one copy each for K and V
    3. return per_token * tokens / 1e9
    4. # Example assumptions: 93 layers, 8 KV heads, head_dim=128, KV stored in 8-bit
    5. print(kv_cache_gb(93, 8, 128, 1_000_000, 1)) # ≈ 190 GB / 1M token

    In other words, under the extreme 8-card configuration, million-token context can essentially not be fully enabled, and the usable length gets squeezed down to the tens of thousands to low hundreds of thousands. If long context is mandatory, there are only three paths: increase parallelism scale (cross-node, needing 400Gbps-class RDMA networking), reduce precision and quantize the KV cache, or tier offload some experts/weights (host memory, local NVMe)—but offloading sharply raises per-token latency.

    Then there are the compute and bandwidth constraints:

    1. # 3) Prefill: theoretical time for 1M-token input
    2. prefill_flops = 2 * 104e9 * 1_000_000 # 2x activated params forward compute
    3. peak_flops = 8 * 1.0e15 # 8 cards × 1 PFLOPS dense compute
    4. mfu = 0.35 # actual utilization taken as 35%
    5. print(f"Theoretical prefill ≈ {prefill_flops / (peak_flops * mfu):.0f} s")
    6. # 4) Decode: bandwidth bound at batch=1 for a single request
    7. bytes_per_step = 104e9 * 0.5 * 1.0625 # activated weights read per step
    8. agg_bw = 8 * 8.0e12 # 8 cards × 8 TB/s HBM bandwidth
    9. print(f"Per-step lower bound ≈ {bytes_per_step / agg_bw * 1000:.2f} ms")

    That is roughly 74 seconds of prefill (1M-token input) and a per-step decode lower bound of about 0.86ms—a theoretical ceiling of around 1100 tokens/s. Accounting for communication and scheduling overhead, real throughput usually lands at 30%–50% of the theoretical value. These two numbers show that this class of model is not suited to low-latency interactive scenarios; it fits “batch + long thinking” offline, task-oriented workloads, where pushing throughput is what makes it economical.

    Download and storage need budgeting too:

    1. size_tb = 1.56
    2. for gbps in (1, 2.5, 10, 25):
    3. print(f"{gbps:>4} Gbps → ~ {size_tb * 8 / gbps / 3600:.2f} hours")
    4. # 1 Gbps≈3.47h, 2.5≈1.39h, 10≈0.35h, 25≈0.14h

    Accounting for unpacking, format conversion, and cache copies, reserve 2.5TB or more of local NVMe; if you use object storage as the weight repository, pay attention to parallel shard fetching and resumable download.

    6. A Single-Machine 8-GPU Engineering Checklist

    1. Parallelism strategy. Prefer pure tensor parallelism (TP=8) over the in-box high-bandwidth interconnect; only introduce cross-node pipeline parallelism when a single machine cannot hold the model, and evaluate cross-machine all-to-all communication overhead alongside it.
    2. Expert parallelism and load balancing. The 896-choose-16 routing has natural hot experts; enable expert replicas/redundant experts and dynamic reordering, or you will get “individual cards overloaded, the whole set waiting.”
    3. Weight loading. Use zero-copy mapped sharded loading to avoid a startup-time bulk deserialization that causes cold starts of tens of minutes; put weights on a read-only mounted volume in the container image to cut down repeated copies.
    4. Sample startup parameters (exact flag names vary by inference runtime):
    1. --model-path $MODEL_DIR \
    2. --tensor-parallel-size 8 \
    3. --max-model-len 131072 \
    4. --gpu-memory-utilization 0.92 \
    5. --kv-cache-dtype fp8 \
    6. --enable-chunked-prefill \
    7. --enable-prefix-caching
    1. Long-context optimization. Enable chunked prefill so long requests do not monopolize the whole batch; use prefix caching for system prompts and tool descriptions—agent scenarios usually see the biggest gains.
    2. Scheduling and queuing. Bucket requests by input length, routing short and long requests to different instance groups; run async tasks through a message queue to absorb peaks, avoiding long prefill blocking interactive traffic.
    3. Multimodal preprocessing. Image resolution and tiling strategy directly determine the number of visual tokens, which amplifies prefill cost, so set an upper limit and cache.
    4. Observability. Cover TTFT, TPOT, tokens/s, KV cache hit rate, expert load skew, memory watermark, and request queue time with monitoring and alerting; retain request samples in logging to support regression evaluation.
    5. Orchestration and elasticity. A container platform hosts multi-instance deployment, with object storage persisting weights and artifacts. On platforms like Baidu AI Cloud, you can typically combine general capabilities such as GPU cloud servers, a container platform, object storage, and monitoring/alerting to build this pipeline, scaling elastically to business peaks.

    7. Risks and Decision Guidance

    Main risks: First, the precision loss from the single 4-bit weight version is irreversible, and part of the shortfall in science and complex reasoning may come from quantization—use a self-built evaluation set to quantify its impact. Second, “8 GPUs to start” in real business often becomes “8 GPUs only runs short context,” so capacity planning must be based on peak context length, not the average. Third, the license’s tiered restrictions directly affect business-model design—this is a legal up-front item, not after-the-fact compliance.

    Selection guidance:

    • Good for local deployment: compliance requirements where data cannot leave your domain, stable and predictable batch workloads, deep fine-tuning and private-data injection, and teams with long-term operations capability;
    • Good for hosted API: large peak fluctuation, limited compute budget, the need for cross-model comparison, tight timelines. Run a break-even analysis of self-hosting’s fixed cost against the API’s marginal cost per average daily token volume—that usually yields a better answer than a gut call.

    Overall, the significance of this weight release is not the “strongest” label but that it puts an engineering problem on the table: when weight size enters the TB range, context enters the millions, and licensing enters tiered authorization, the core skill of model selection has shifted from “can I call it” to “can I account for memory, bandwidth, compute, and compliance cost.” Work out those four ledgers, then decide whether to run locally or use a service—that is the basic competency for this era of ultra-large open-weight models.