2.8T Open-Weight Model Deployment: The 8-GPU Threshold for Local Inference

    A Chinese LLM vendor has released the full weights of a 2.8T-parameter flagship model as a directly downloadable, open-weight release. This article breaks down what that means in practice: the sparse MoE architecture (896 routed experts, 104B activated parameters per token, sparse attention, and a 1M-token context), the MXFP4/MXFP8 quantization that produces a ~1.56TB download, the custom license that makes "open weights" very different from "open source," and the three hard hardware constraints—memory, bandwidth, and communication—that put the practical floor at 8 GPUs. It also covers parallelism strategies, inference tuning, a self-host-versus-cloud decision checklist, and a pre-deployment self-test list.

    1. What Happened: Weights Released on Schedule

    In late October, a Chinese LLM vendor shipped a flagship model with 2.8T total parameters, first exposing it through an API and web interface with a commitment to release the full weights before November 4. On November 2, the corresponding weight repository appeared on a mainstream model-hosting community. The repo remained closed on the morning of the 4th and opened that evening—downloadable without any application or approval process. It contains 112 files and 96 weight shards totaling roughly 1.56TB.

    While most flagship models are API-only, fully releasing the weights of the largest model is uncommon. Publishing the weights changes things concretely:

    • You can deploy fully offline, so data never leaves your network;
    • You can fine-tune on your own data;
    • You can validate capabilities with your own evaluation sets;
    • Inference cost is no longer constrained by the vendor’s pricing strategy.

    But whether these advantages materialize depends on two things: what the license allows you to do, and what your hardware can support. The sections below break each down.

    2. Breaking Down the 2.8T: The Sparse Architecture’s Parameter Ledger

    According to the official model card, the model is a natively multimodal agent model, continued-trained from the previous generation in the same series. The main specifications are:

    Item Specification
    Total parameters 2.8T
    Activated parameters per token 104B (~3.7% activation rate)
    Expert structure 896 routed experts + 2 shared experts; 16 selected per token
    Layers 93
    Context length 1M tokens
    Vision encoder Native multimodal vision encoder
    Precision format Weights MXFP4, activations MXFP8
    Structural change vs. previous gen 24 layers switched from full attention to sparse attention

    2.1 Why It’s Called a “Sparse” MoE

    In a traditional dense model, every token passes through all parameters. A Mixture-of-Experts (MoE) model splits the feed-forward layers into multiple “experts,” and a router selects a few experts to compute each token while the rest “rest” for that token. Here the routing rule is: pick 16 of the 896 experts, plus 2 shared experts that always participate.

    This creates a distinction you have to keep clear:

    • Parameter count determines how much knowledge the model can hold, and how much GPU memory inference needs;
    • Activated parameters determine how many multiply-accumulate operations each generated token requires.

    2.8T and 104B differ by about 27x. As long as all the weights fit in memory, generation speed is governed mainly by the 104B of computation, not the 2.8T.

    2.2 Sparse Attention and the 1M Context

    Attention computation grows quadratically with context length. To ease the cost of long contexts, the model replaces 24 of its full-attention layers with sparse attention (DSA) while the remaining layers keep full attention, forming a hybrid “sparse + full-attention” structure.

    This design is common: sparse layers filter relevant information across a wide range at lower cost, while full-attention layers handle fine-grained modeling. The hybrid layout saves compute on long-text tasks but also changes the attention distribution. Tasks like long-document retrieval and cross-file code understanding need to be validated separately.

    2.3 How to Read the Official Results

    First, the vendor’s self-reported comparison data (below, “competitors” refers to two leading international closed-source models):

    Benchmark This model Competitor A Competitor B
    Terminal-Bench 4.0 (coding/terminal tasks) 65.7% 66.4% 58.2%
    SWE Marathon (long-horizon software engineering) 51.7% 46.3% —
    Terminal-Bench-Science (science tasks) 38.1% 63.3% —
    HLE-Diamond (hard reasoning) 39.4% 60.9% —

    The vendor claims it beats Competitor A on 16 of 37 benchmarks. The comparison reveals a clear profile: it excels at agent-type tasks—coding, terminal operations, and long-horizon software engineering—but lags clearly on scientific computing and some of the hardest reasoning. One independent evaluation body reportedly ranked it fourth globally on a composite intelligence index.

    Three cautions apply when reading data like this:

    1. Look at task type: If your core need is code generation, terminal operations, or multi-step tool calling, it’s in the top tier; if it’s scientific reasoning or complex mathematical proofs, you should run a small-sample validation first rather than copying leaderboard rankings directly.
    2. Look at how it was tested: Vendor self-reported results typically use specific inference configurations and sampling strategies that may not match real-world usage.
    3. Look at agent scenarios: Agent tasks are chained over multiple steps, so winning a single step doesn’t mean winning the whole chain. More meaningful metrics are task completion rate, average step count, tool-call accuracy, and per-task cost.

    3. Where the 1.56TB Comes From: Quantization Is the First Deployment Hurdle

    The official release ships only one version: MXFP4 weights with MXFP8 activations. This is the product of quantization-aware training (QAT)—simulating the error from low precision during training, rather than “compressing” a model after training. No BF16 or FP8 version was released.

    You can work backward from the size:

    • Stored in BF16 (2 bytes per parameter), 2.8T parameters would need about 5.6TB;
    • MXFP4 represents parameters in 4 bits and pairs with block-level scale factors in the microscaling (MX) format, so each weight takes about 0.5 bytes;
    • 2.8T × 0.5 bytes ≈ 1.4TB, plus the vision encoder, token embeddings, and other non-core parts, which roughly matches the ~1.56TB download.

    For deployment, this has three implications:

    • Compute-feasible doesn’t mean it fits: The 104B activations determine compute demand, while 1.56TB determines storage and memory demand. Consumer GPUs can barely handle the compute, but nowhere near fit these weights.
    • Low bit-width isn’t universal: MXFP4 is sensitive to activation distributions, and the model card notes it’s a QAT product. If you want to convert to another precision or format yourself, you’ll need to recalibrate or even do light fine-tuning, and the conversion pipeline must be tested separately.
    • There’s no “lite” fallback: The vendor didn’t release a smaller sibling model. To run on limited hardware, you can only rely on third-party community quantization, distillation, or pruning—and you have to validate the quality yourself.

    4. Licensing: Open Weights ≠ Open Source

    This is the point most worth examining closely. The repo uses the vendor’s custom license, labeled “other” on the hosting platform rather than a standard open-source license like MIT or Apache. The main terms are word-for-word identical to the previous generation’s, with only the name changed.

    The core is an MIT-style grant: free use, copy, modify, distribute, fine-tune, and commercial use, without the full clauses the open-source community typically requires (source-code disclosure, patent grants, and so on). The two additional restrictions apply only to “large-scale external business”:

    • By revenue: Companies offering the model as model-as-a-service (MaaS) externally, with cumulative revenue over $20M over any rolling 12 months, must sign a separate agreement before commercial use;
    • By user scale: Commercial products exceeding a certain monthly-active-user threshold (in the tens of millions) are likewise subject to a separate arrangement.

    From this you can extract general principles for open-source compliance:

    1. “Weights are downloadable” and “you can use the license freely” are two different things. To judge whether you can use it in a product, read the license text—not whether the download button requires approval.
    2. Look at obligations, not just prohibitions. Attribution, declaring modifications, whether downstream redistribution must use the same license, and naming restrictions on derivative models are all items to check one by one.
    3. Additional clauses are often tiered by scale. Once revenue or user volume crosses a threshold, you switch to a commercial-negotiation path. Bring legal review into the timeline during the technology-selection stage.
    4. Keep records and archive. Save the model version used, the license text, and evaluation records so you can respond to later audits and compliance inquiries.

    For most small and mid-sized teams and internal applications, the bar in this license isn’t high. What genuinely needs planning ahead is any product whose user scale or revenue might approach a threshold.

    5. 8 GPUs Is Just the Starting Point: Three Hard Constraints of Local Deployment

    “Local inference starts at 8 GPUs” is the conclusion; behind it are three quantifiable constraints.

    5.1 Memory: 8 GPUs Just Barely Fits, with Limited Headroom

    Spread 1.56TB of weights across 8 GPUs, and each card carries about 195GB—slightly above the 192GB-class HBM capacity of mainstream single cards. In real deployments, memory must also be reserved for the KV cache, runtime overhead, and easily overlooked modules like the vision encoder. Even at just a 25K-token context, a rough FP16 KV estimate adds more than 10GB per card. This means an 8-GPU setup has only two options: use higher-capacity accelerators, or trade off headroom through offloading and compression.

    5.2 Bandwidth: It Determines Single-Request Generation Speed

    During generation, each output token requires moving the selected experts’ weights for that token from memory into the compute units. With 104B activated parameters and MXFP8 weights (~1 byte), that’s at least ~100GB moved per token. At a combined 8-GPU HBM bandwidth of about 25TB/s, the theoretical ceiling is only about 250 tokens/sec. After accounting for communication and scheduling overhead, single-request throughput typically lands in the range of a few dozen to a bit over a hundred tokens/sec.

    Adding GPUs raises concurrent throughput but barely makes any single request faster. Decoding a large MoE model is a bandwidth-intensive workload: memory capacity determines “can it run,” and memory bandwidth determines “how fast it runs.”

    5.3 Communication: Even a Single 8-GPU Node Can’t Avoid It

    Once weights are split across 8 GPUs, every forward pass runs into two kinds of communication:

    • The non-expert parts require tensor-parallel (TP) reductions;
    • The experts selected by the router sit on different cards, requiring all-to-all communication to send tokens to the right experts and gather the results back.

    A single 8-GPU node communicates over high-speed inter-card links, so latency is acceptable. Once you scale to multiple nodes, cross-node all-to-all quickly becomes a bottleneck and demands high-speed networking.

    5.4 A Sample Memory Budget

    The script below demonstrates a top-down memory-budget calculation. Field names and structure can be swapped for your actual model card:

    1. import math
    2. # The following values are illustrative; for actual deployment, refer to the official model card and inference framework documentation
    3. TOTAL_PARAMS = 2.8e12 # Total parameters
    4. ACTIVATED = 104e9 # Activated parameters per token
    5. GPUS = 8
    6. BYTES_PER_WEIGHT = 1 # Approximate 1 byte, estimated from MXFP8 activations
    7. # 1. Weights: bytes of weights each GPU must hold
    8. weights_per_gpu = TOTAL_PARAMS * BYTES_PER_WEIGHT / GPUS
    9. print(f"Weights per GPU: {weights_per_gpu / 1e9:.1f} GB")
    10. # 2. KV cache: estimated from a typical decoder structure; use model-card values for specifics
    11. LAYERS, KV_HEADS, HEAD_DIM = 93, 8, 128
    12. def kv_cache_gb(seq_len, bytes_per_elem=2):
    13. # One copy each for K and V; under tensor parallelism, each KV head lives on a single GPU
    14. return 2 * LAYERS * KV_HEADS * HEAD_DIM * bytes_per_elem * seq_len / GPUS / 1e9
    15. for s in (32768, 262144, 1048576):
    16. print(f"Context {s:>7} tokens, per-GPU KV (FP16): {kv_cache_gb(s):6.1f} GB")
    17. # 3. Theoretical single-stream decode ceiling under bandwidth constraints (ignoring communication and scheduling overhead)
    18. HBM_BW_PER_GPU = 3.0e12 # Per-GPU HBM bandwidth (B/s)
    19. decode_budget = ACTIVATED * BYTES_PER_WEIGHT
    20. print(f"Single-stream decode ceiling: {GPUS * HBM_BW_PER_GPU / decode_budget:.0f} tok/s")

    Sample output (actual results vary with configuration):

    1. Weights per GPU: 175.0 GB
    2. Context 32768 tokens, per-GPU KV (FP16): 10.1 GB
    3. Context 262144 tokens, per-GPU KV (FP16): 80.5 GB
    4. Context 1048576 tokens, per-GPU KV (FP16): 322.1 GB
    5. Single-stream decode ceiling: 230 tok/s

    The conclusions are intuitive:

    • 32K context: ~10GB KV per card; with runtime overhead, 8×192GB is feasible;
    • 256K context: ~80GB KV per card; a 192GB single card is already tight;
    • 1M context: over 320GB KV per card; 8 GPUs can’t hold it, so you can only compress the KV cache, reduce context length, or add more GPUs.

    There’s also an easy-to-miss detail: 896 experts can’t be evenly distributed across 8 cards. Powers of two like 512 or 1024 divide evenly, but 896 split across 8 gives 112 per card, and the remainder must be assigned elsewhere, slightly increasing load on individual cards.

    6. Deployment Practice: Parallelism Strategies and Tuning

    6.1 Parallelism: The Trade-offs Among TP, EP, and Pipeline Parallelism

    Strategy What it splits Main communication Best-fit scenario
    Tensor parallelism (TP) Shards each layer’s weights horizontally Reduction in every forward/backward pass Within a single node’s high-speed interconnect
    Expert parallelism (EP) Places different experts on different cards Token-level all-to-all Many experts, or when scaling out
    Pipeline parallelism (PP) Splits layers into segments Point-to-point transfer between adjacent segments Cross-node, with limited interconnect bandwidth

    The common practice is a layered combination: TP with EP inside a node, PP between nodes. This way cross-node traffic carries only the activations at layer boundaries, and all-to-all stays confined within the high-speed interconnect domain, significantly reducing communication volume.

    6.2 Inference-Serving Tuning Checklist

    • KV quantization: Compress the KV cache to FP8 or lower; on long-context workloads this can save tens of GB per card, at the cost of a small precision loss that needs regression testing;
    • Paged KV management: Allocate memory in pages to reduce fragmentation and let different requests share common prefixes (system prompts, long documents);
    • Chunked prefill: Split very long prompts into small chunks and mix them into the decode schedule, so a long prompt doesn’t block subsequent requests;
    • Expert load balancing: Monitor expert-hit distribution across cards—hot experts will slow down the whole batch;
    • Speculative decoding: Use a small draft model to guess multiple tokens in parallel, then have the large model verify them in one pass. This eases the bandwidth bottleneck but needs extra memory and adds implementation complexity;
    • Batching strategy: Static batching is simple, but long-tail requests slow down the whole batch. Continuous batching substantially improves throughput but requires framework support.

    6.3 Capacity Planning: Concurrency First, Then Length

    Capacity planning must account for both concurrency and context length. The memory reserved for the KV cache, multiplied by maximum concurrency, cannot exceed the remaining memory. For example, with a 32K context ceiling, KV is about 10GB per card, so every 100 additional concurrent requests adds roughly 1TB (across all cards)—8 GPUs simply can’t hold it. High-concurrency, long-context scenarios therefore require:

    1. Using prefix sharing to eliminate duplicate KV;
    2. Quantizing and compressing the KV cache;
    3. Scaling out horizontally when necessary.

    6.4 Fine-Tuning and Customization

    • Parameter-efficient fine-tuning: LoRA/QLoRA train only a small set of new parameters while the base weights stay read-only, which 8 GPUs can support. The hard part is the MoE router: if you only update LoRA on the attention layers and freeze the router, the model still computes with its original expert division, so customization has limited effect. Whether to also train router-related parameters requires ablation experiments.
    • Continued pre-training: Extremely costly, needing massive high-quality corpora and long training cycles—usually beyond the reach of small and mid-sized teams.
    • Small-sample evaluation: MoE processing paths for the same input can be non-deterministic, so after fine-tuning, use a fixed evaluation set to confirm there’s no regression.

    7. Self-Host or Cloud: A Decision Checklist

    Scenarios that favor self-hosting

    • Sensitive data, requiring fully offline or private deployment;
    • Access to 8-GPU-class accelerators and an ops team familiar with distributed inference;
    • Stable long-term load, where self-hosting clearly amortizes cost.

    Scenarios better suited to the cloud

    • Lack of large-scale accelerator resources and inference-ops experience;
    • Traffic with clear peaks and troughs that need elastic scaling;
    • Wanting to quickly validate the model’s capabilities before committing to a long-term investment.

    If you choose the cloud, you can complete the whole deployment using the general capabilities offered by Baidu AI Cloud:

    1. Use object storage to hold weights, images, and datasets, and manage versions;
    2. Deploy inference services on a container platform, auto-scaling by queue depth;
    3. Expose a unified interface through a model service, with load balancing to distribute traffic;
    4. Use a logging service to record requests and routing distribution, and monitoring/alerting to watch memory usage, queue latency, and throughput.

    It’s worth emphasizing that cloud usage is still subject to the model’s own license terms on revenue, user scale, and similar clauses.

    8. Deployment Self-Test Checklist

    1. License: Confirm the license type and version, check each additional clause, and assess whether your revenue and user scale might cross a threshold;
    2. Storage: Reserve at least 2x the space for the 1.56TB of weights, for download, extraction, and caching. After downloading, verify integrity shard by shard against the official checksums to avoid loading corrupted weights;
    3. Memory: Budget by maximum context and target concurrency, leaving 10%–20% headroom for runtime;
    4. Communication: Use inter-card high-speed links within a single node; multiple nodes need high-speed networking, and you should validate the actual all-to-all throughput;
    5. Business evaluation: Compare “this model” against “your current production model” on your own task set, covering task completion rate, latency, and per-task cost;
    6. Long context: Test long-document retrieval and cross-file understanding separately to validate sparse attention’s real effect;
    7. Fine-tuning: Compare before and after on a fixed evaluation set;
    8. Monitoring: After launch, continuously track memory levels, expert-load distribution, queue latency, and per-token cost.

    Conclusion

    The deployment of this 2.8T open-weight model sends three clear signals:

    • Trillion-parameter sparse models are becoming the standard architecture for flagship models;
    • Low-bit quantization makes weight distribution of ultra-large models feasible, but the hardware bar remains real;
    • “Open weights” is becoming a conditional form of openness, so license review should proceed in step with technology selection.

    For developers, rather than chasing leaderboard rankings, it’s better to first clarify whether your business needs coding, reasoning, or long-text handling, then make the decision with a memory-budget sheet and your own evaluation set. If capabilities are sufficient, data is sensitive, and you have ops capacity, self-host; if you lack resources, need speed, and face fluctuating traffic, hand it to a cloud platform. Balancing technical judgment with compliance review is what turns this 1.56TB of weights into real productivity.