Deploying a 2.8T-Parameter Open-Weights Model: The 8-GPU Floor and the License Fine Print

    A Chinese model team has released full weights for a 2.8T-parameter model — about 1.56TB across 96 shards, downloadable without an application. This article skips the leaderboard talk and answers three engineering questions: what is actually inside the repository, what the custom license permits, and how to size hardware for local inference. It covers the MoE architecture (896 experts, ~104B active parameters per token), the 4-bit weight / 8-bit activation MX quantization format, why 8 x 192GB GPUs is only a theoretical floor, KV cache math for 8K to 1M contexts, parallelism strategies, cloud vs. self-hosting economics, and a capacity-planning template. It also explains how to read vendor-reported benchmark scores and the two revenue/user thresholds that trigger extra licensing for large-scale model-as-a-service providers.

    Introduction: Getting the Weights Is Just the Beginning

    On November 4, a Chinese model team made good on an earlier commitment and published the complete weights of a 2.8T-parameter model on a mainstream model hosting community. A single repository contains 112 files, including 96 weight shards, totaling roughly 1.56TB — openly downloadable, no application required. By the publisher’s account, this is the largest open-weights model released so far, considerably larger than any model whose weights had previously been made public.

    The release process itself is worth a note. The model was first served through an API and a web interface in late October, and the team committed at the same time to publishing the full weights by a fixed date. The hosting repository was created two days in advance, was still private on the morning of launch day, and opened that evening. The stated date, the weight volume, and the file structure all hold up to verification.

    Admiring the “world’s largest open weights” is easy; actually putting them to work is a different matter. This article does not discuss leaderboard placement. It answers three engineering questions:

    • What is actually inside the repository;
    • What the license allows you to do;
    • How to do the hardware math to run it locally.

    1. Model Anatomy: What Makes Up 2.8T Parameters

    Overall specifications: sparse activation, dense parameters

    According to the model card, this is a natively multimodal agent model, continued-trained from the previous generation. The core specifications are:

    Dimension Specification Engineering implication
    Total / active parameters 2.8T / ~104B per token Provision memory for the total, compute for the active subset
    Architecture Mixture of Experts (MoE) Large parameter count, but only a fraction is computed per forward pass
    Expert configuration 896 experts, 16 selected per token, plus 2 shared experts Requires expert parallelism; communication and load balancing are the hard parts
    Layers 93 24 of them use sparse attention
    Context 1M tokens Long context is a design goal and a primary source of memory pressure
    Vision Built-in vision encoder, understands images directly Multimodal input works out of the box

    For developers, the most important row is “2.8T / 104B.” Resource consumption for inference splits roughly into two parts:

    • Compute is tied mainly to active parameters. Each generated token passes through roughly 104B parameters’ worth of computation.
    • Memory is determined by all parameters, the KV cache, and runtime overhead. Even experts that are not selected for this token must keep their weights resident in memory.

    Hence one basic principle: parameter-dense, compute-sparse. Compared with dense models of similar parameter count, this class of model is far cheaper per unit of compute — but without enough memory, you cannot even get started.

    Structural evolution: surgery for long context

    Compared with the previous generation, only one thing changed in the architecture: 24 of the 93 layers that previously used full attention were replaced with sparse attention. The remaining 69 layers keep full attention unchanged.

    The cost and KV cache of full attention both grow with context length. At million-token context lengths, that overhead becomes the main driver of inference cost. Letting some layers use sparse attention reduces attention compute and memory traffic under long sequences, at the price that those layers no longer compute against every historical token.

    Two implications follow:

    • The inference framework must implement this sparse structure properly. Otherwise it either fails to load the weights correctly or falls back to a naive implementation, and the long-context speedup disappears with it.
    • When evaluating long-text capability, test with the longest real samples from your own workload rather than looking only at short-text results.

    Numerical precision: a quantized-only release

    Weights are stored in a 4-bit floating-point format and activations in an 8-bit floating-point format, both from the micro-scaling (MX) family that shares scale factors over small blocks. This configuration is the product of quantization-aware training: the low-bit quantization error was folded into optimization during training so the model “performs well even at low precision.” The publisher released only this one version; there is no higher-precision floating-point build.

    Two direct consequences for engineering practice:

    1. No self-quantization needed to deploy, which is very friendly to memory-constrained setups.
    2. Further training needs extra caution. If you plan continued pretraining or large-scale fine-tuning on these weights, first confirm two things: whether your target inference framework natively supports these formats, and whether the training pipeline can run stably against low-bit weights. If necessary, ask the publisher whether other precision versions are planned.

    2. Reading the Benchmarks: How to Handle Vendor-Reported Numbers

    The publisher supplied a comparison table against leading closed models. The table below keeps only the common premise — all figures are publisher-reported — and the comparison groups are labeled simply A and B.

    Benchmark category (illustrative) This model Group A Group B
    Coding benchmark 65.7% 66.4% 58.2%
    Long-horizon software engineering benchmark 51.7% 46.3% —
    Science benchmark 38.1% 63.3% —
    Hard knowledge-reasoning benchmark 39.4% 60.9% —

    The publisher’s claim is that the model beat Group A on 16 of 37 benchmarks. Reading row by row, this is not a “leading across the board” result:

    • Strengths: coding and agentic tasks. Terminal coding is slightly below A; long-horizon software engineering is clearly ahead.
    • Weaknesses: scientific reasoning and hard knowledge reasoning, where the gap to A exceeds 20 percentage points in both.

    Taken together, the capability profile leans toward “write code, operate tools, run long workflows” rather than “do research, crack hard problems.” Separately, one third-party evaluation organization reportedly gave it a composite intelligence score of 55, ranking fourth globally. Specific rankings shift with the set of participating models and the evaluation version, so treat that as reference only.

    Where self-reported numbers go wrong

    Vendor self-testing commonly suffers from four problems:

    1. Sample selection bias — showing the benchmarks that look good for you.
    2. Opaque test configuration — prompts, number of examples, sampling temperature, and whether tool use is allowed are often undisclosed. Scores for the same model can vary widely across configurations.
    3. Data contamination — evaluation items or their variants appeared in training data, inflating scores.
    4. Non-reproducible baselines — competitors’ scores may come from an older release or a different configuration.

    For selection decisions, your own evaluations are more trustworthy.

    Building your own evaluation harness

    1. Match your business distribution. Pull 200–500 representative tasks from real workloads, covering both common and long-tail cases, and have humans write reference answers or acceptance criteria.
    2. Standardize the inference configuration. Use identical prompts, sampling parameters, and tool environments for every candidate model.
    3. Measure engineering metrics alongside quality. Besides answer quality, record time to first token, per-token latency, throughput, and failure rate.
    4. Check for contamination. Embed distinctive markers in the prompt, or temporarily swap out some background details, and see whether the output “remembers” the original answer.
    5. Group multi-turn and long-text scenarios separately. That is where sparse attention and the million-token context design are actually tested.

    3. The License: Open Weights Are Not Open Source

    This is the aspect of the release most worth discussing on its own.

    1. Nature: a custom license, not a standard open-source one

    The repository attaches a license under the publisher’s own name, and the hosting community classifies it as “other” rather than as one of the standard permissive licenses the industry commonly uses. It is word-for-word identical to the license used by the previous-generation model, with only the name changed. In other words, this reflects the team’s consistent licensing policy rather than a last-minute tightening.

    2. Grant: the core is permissive

    From the publicly available information, the body of the agreement adopts a permissive-style grant:

    • Free to use, copy, modify, and distribute;
    • Fine-tuning and commercial use allowed;
    • Typically requires retaining copyright and license notices in use.

    Individual research, internal enterprise use, and embedding the model in a product for inference generally fall within the grant.

    3. Restrictions: two thresholds that only affect large-scale external services

    On top of the permissive grant, two additional restrictions apply only to parties that “provide service externally” and reach a scale threshold:

    • Model as a service: a company that offers the model as an external service and whose trailing 12-month cumulative revenue exceeds $20M must sign a separate agreement with the publisher before commercial use;
    • Commercial product user scale: once a commercial product’s monthly active users exceed a certain scale, separate authorization is also required (the original text of this clause is incomplete in the available material, so the exact threshold must be confirmed against the full license).

    The triggering precondition for both restrictions is “providing service externally” or “a commercial product reaching scale.” Below the thresholds, you can use the model normally under the core license terms.

    4. Compliance checklist

    1. Determine your own role. Are you selling model inference as a service, or embedding the model in your own product? If it is the former, you need to track trailing 12-month cumulative revenue continuously.
    2. Plan the commercial conversation early. Start talking to the publisher before revenue approaches the threshold. License approval usually involves a legal process, and starting at the last minute can threaten business continuity.
    3. Derivatives inherit the same license. Models fine-tuned from these weights generally remain bound by the original license, and distributing them requires including the license text.
    4. Keep the notices. State the model’s origin and license information in your documentation or product description.
    5. The full text governs. Final interpretation of the license terms rests with the full text published by the publisher; for major commercial decisions, involve legal counsel.

    For the vast majority of small and mid-sized teams and enterprises, this license is permissive enough — it constrains only the small number of extremely large-scale external service providers.

    4. The 8-GPU Floor: How Much Memory Local Inference Really Takes

    1. Start with a division

    Spread 1.56TB of weights across 8 cards and each card carries roughly 195GB — before accounting for the KV cache, framework overhead, and memory fragmentation. Mainstream data-center GPUs currently offer somewhere between 80GB and 192GB per card:

    • 80GB per card: 8 cards total about 640GB, enough for only about 40% of the weights — ruled out immediately;
    • 141GB per card: 8 cards total about 1128GB, still short of the full weight set;
    • 192GB per card: 8 cards total about 1536GB, essentially equal to the weight volume in theory.

    So a typical “8-GPU starting point” configuration amounts to roughly eight 192GB-class GPUs.

    2. Why 8 GPUs is only a theoretical floor

    • KV cache grows with context length and the number of concurrent requests;
    • Framework and communication buffers — the inference framework itself consumes memory, and expert parallelism needs extra communication buffers;
    • Fragmentation — after long uptime, memory fragmentation reduces usable capacity;
    • Training or fine-tuning — essentially no headroom left.

    The safer approach is to take “weights ÷ per-card memory,” round down, and then budget another 15%–25% headroom when planning the card count.

    3. Capacity estimates at different context lengths (illustrative)

    The snippet below estimates the KV cache. Assuming grouped-query attention with 8 KV heads and a head dimension of 128, the KV cache per token per layer is:

    • FP16: about 4KB;
    • FP8: about 2KB.

    Across all 93 layers, that comes to roughly 360KB of KV cache per token (FP16).

    1. # Illustrative code: estimate KV cache memory; actual values must come from the model config
    2. def kv_cache_gb(seq_len, n_layers, n_kv_heads, head_dim, bytes_per_elem):
    3. """Return the KV cache footprint of a single request in GiB"""
    4. per_token_bytes = 2 * n_layers * n_kv_heads * head_dim * bytes_per_elem
    5. return seq_len * per_token_bytes / 1024 ** 3
    6. BYTES_FP16, BYTES_FP8 = 2, 1
    7. N_LAYERS, N_KV_HEADS, HEAD_DIM = 93, 8, 128
    8. for ctx in (32_768, 128_000, 1_000_000):
    9. print(ctx, round(kv_cache_gb(ctx, N_LAYERS, N_KV_HEADS, HEAD_DIM, BYTES_FP16), 1))
    10. print(ctx, round(kv_cache_gb(ctx, N_LAYERS, N_KV_HEADS, HEAD_DIM, BYTES_FP8), 1))

    Under these assumptions:

    • 8K context: under 3GB of KV cache for a single request — plenty of headroom from the long-context design;
    • 128K context: about 44GB per request (FP16), or about 22GB (FP8);
    • 1M context: about 350GB per request (FP16), which already exceeds the entire memory of 8 GPUs.

    So on 8 GPUs, “supports 1M tokens” and “serves stably at 1M tokens” are two different things. A real deployment should work backward from the context length and concurrency the business actually needs to the minimum memory required.

    4. From “it runs” to “it runs well”: parallelism strategy

    The 896-expert structure shapes the parallelism strategy:

    • Tensor parallelism shards each layer’s matrices across cards and suits high-bandwidth intra-node interconnects;
    • Expert parallelism distributes the 896 experts across cards, so each token interacts with only a few experts after routing, at the cost of inter-card collective communication;
    • Pipeline parallelism splits by layer across machines, lowering the per-machine card count but increasing end-to-end latency.

    Active parameters per card are about 104B. Under the illustrative assumption of 8 cards each delivering a notional compute figure per precision class, theoretical decoding speed can reach hundreds of tokens per second. This again confirms the principle: compute scales with active parameters, memory with total parameters.

    5. Cloud or owned hardware: run the full numbers

    Dimension Cloud rental Self-built cluster
    Time to launch Fast, provisioned in hours Slow, procurement and delivery measured in weeks or months
    Cost structure Pay as you go; better when utilization is low Fixed investment; needs sustained high load to amortize
    Elasticity Peaks can be scaled out quickly Expansion is constrained by hardware supply
    Operational burden The cloud provider handles the underlying hardware You build your own failure-handling and spares process

    Self-hosting is usually more economical when inference utilization stays high over the long term; if the workload has pronounced peaks and troughs, or you are still validating, on-demand cloud is the better fit. Run a sensitivity analysis against average monthly utilization before deciding.

    5. Running It in the Cloud: A Reusable Reference Path

    For most teams, procuring an 8-plus-GPU cluster of high-end accelerators is not realistic. The more common approach is to deploy and scale on a cloud platform. Using Baidu AI Cloud as an example, the setup follows roughly these steps:

    1. Weight distribution. Upload the 1.56TB of weights to object storage and have inference nodes pull from the internal network, avoiding repeated cross-region downloads. Also keep the publisher’s official checksums and verify every shard after download to confirm completeness and integrity.
    2. Instances. Use GPU instances with multi-card high-speed interconnect. When a single instance is not enough, combine multiple instances with a distributed inference framework.
    3. Traffic ingress. Distribute requests through a load balancer, with support for long-lived connections and streaming output.
    4. Observability. Connect log and monitoring/alerting services, with thresholds set on time to first token, throughput, and failure rate.
    5. Evaluation. Use stateless compute such as function compute to run the evaluation set on a schedule, so quality regressions after model updates get caught.

    Capacity planning template for launch

    1. # Illustrative capacity plan; values must be calibrated against real load tests
    2. model:
    3. total_params: "2.8T" # sets the memory floor
    4. active_params: "104B" # sets the compute requirement
    5. context_window: 32768 # set to the business need, don't copy the maximum
    6. runtime:
    7. tensor_parallel: 4 # intra-node tensor parallelism
    8. expert_parallel: 2 # expert parallelism
    9. replicas: 3 # replica count, determined by target QPS in load tests
    10. gpu_memory_utilization: 0.85 # leave room for fragmentation and communication buffers
    11. slo:
    12. ttft_p99_ms: 1500 # p99 time to first token
    13. tpot_p99_ms: 40 # p99 per-token output latency
    14. availability: "99.9%"

    License self-check

    After deployment, work through the five compliance points from the previous section, especially:

    • the determination of your external-service role;
    • trailing 12-month revenue tracking;
    • including the license when distributing derivative models.

    6. Six Techniques That Make Large Models Actually Affordable

    Even with the hardware in place, an unoptimized super-large model still has a high per-token cost.

    1. Prefix caching. System prompts, tool definitions, and long documents often recur across requests. Caching that computation substantially reduces time to first token.
    2. Semantic caching. Cache frequent, similar question-answer pairs and return on a direct hit. Support and knowledge-QA scenarios with high hit rates benefit most.
    3. Continuous batching. Process multiple requests at once to keep GPU compute saturated and raise overall throughput.
    4. Speculative decoding. A small model proposes several tokens and the large model verifies them in one pass, noticeably speeding up output.
    5. Model routing. Send simple tasks to a small model and route only complex ones to the 2.8T model, cutting average cost sharply.
    6. Context trimming. Use retrieval augmentation to feed only the relevant fragments into the context instead of stuffing in whole documents.

    Conclusion: Open Weights Are Not Zero Barrier

    This release carries three layers of meaning:

    • Capability: clearly competitive on coding and agentic tasks, still behind on science and hard reasoning — match the model to the workload.
    • License: “open weights” rather than “open source.” The overwhelming majority of teams can use it with confidence; only large-scale external service providers need separate authorization.
    • Engineering: 8 GPUs is only a theoretical floor. Million-token contexts, high concurrency, and low latency all have to be delivered step by step through memory planning, parallelism strategy, and inference optimization.

    For developers and enterprises, the pragmatic path is to validate quality on a small scale first, then do capacity planning, and handle license compliance in parallel. Where useful, cloud capabilities such as object storage, GPU instances, containers, and monitoring/alerting can lower the deployment barrier and leave your energy for the business itself.