Deploying a 2.8T-Parameter Open-Weight MoE Model: What It Really Takes to Run Locally

    This article uses a 2.8T-parameter, natively multimodal open-weight MoE model as a case study to explain what it actually takes to deploy large open-weight models locally. It breaks down the architecture (104B active parameters, 896 routed experts, a 1M-token context, and MXFP4 weights), shows how to critically read vendor-reported benchmark scores, clarifies the license (open weights are not open source, with restrictions on hyperscale commercial use), and estimates deployment resources (8 flagship-class GPUs, KV cache sizing, prefill compute, and operations). It also compares three deployment paths and provides a go-live checklist, making it useful for developers and enterprises deciding whether to self-host large open-weight models.

    Introduction: With open weights, shipping matters more than releasing

    On November 4, 2026, a natively multimodal, ultra-large-parameter model released its full weights on an international model-hosting platform. The model totals 2.8T parameters, delivered as 112 files across 96 shards of roughly 1.56TB combined, and can be downloaded without any application. It is the largest openly available weight set to date.

    Behind that size sit two thresholds:

    • Licensing: open weights do not equal open source—the license adds restrictions aimed at very large-scale commercial use.
    • Hardware: based on the official 4-bit quantized weights, running the model locally requires at least eight flagship-class accelerator cards.

    For developers and enterprises, the real question isn’t “how capable is this model,” but “should we run it ourselves, can we run it stably, and are there compliance concerns.” This article does not benchmark any specific product. Instead, it treats this class of open-weight large model as a sample and walks through, in turn, its architecture, the credibility of its benchmark scores, the boundaries of its license, and its resource requirements—alongside an evaluation method you can apply directly.

    1. Availability: Downloadable does not mean freely usable

    The model’s online service and API went live before its weights were released. At the time, the developers committed to publishing the full weights on release day, and they delivered as promised. The repository was created several days in advance, remained private on the morning of release, and went public that evening.

    The download has these characteristics:

    • Stored in sharded safetensors format, with an index file for on-demand loading;
    • A single shard is typically on the order of tens of GB, so you need ample disk space and a verification step;
    • The hosting platform marks the license as license:other, rather than a standard open-source license such as MIT or Apache.

    This should signal to teams: treating “open weights” as “open source” leads to compliance misjudgment. Downloading the weights is only the first step—license review, hardware assessment, and inference optimization make up the bulk of the real work.

    2. Model breakdown: 2.8T total parameters, 104B active parameters

    The official model card describes it as a natively multimodal agent model, continued-trained from the previous generation. Its core parameters:

    Dimension Value
    Total parameters 2.8T
    Active parameters per token 104B
    Expert structure 896 routed experts + 2 shared experts, 16 selected per token
    Layers 93
    Context length 1M tokens
    Vision In-house vision encoder, supports image input
    Weight precision MXFP4 (4-bit, block-level shared scaling factor)
    Activation precision MXFP8
    Structural change Full attention in some layers replaced with sparse attention (DSA)

    2.1 Sparse experts: nearly 27× gap between total and active parameters

    The model uses a Mixture-of-Experts (MoE) architecture in which each token passes through only 16 routed experts and 2 shared experts. Active parameters are 104B, about 3.7% of the total. This leads to two implications:

    • Capacity decoupled from compute: parameter capacity is nearly 27× that of a dense model, yet a single forward pass computes on the order of hundreds of billions of parameters.
    • VRAM becomes a hard constraint: every expert must reside in GPU memory so that any token can select it.

    MoE inference is therefore a memory-bandwidth-bound workload. More importantly, batching effects come into play: for a single request, each layer activates only 16 experts per token; as concurrent requests grow, the number of experts hit per layer approaches the total expert count. Assuming expert selection within each layer is approximately independent and uniformly distributed, an 8-GPU single-node deployment processing 32 concurrent tokens sees:

    • Roughly 1.66× the per-token expert reads compared to a single request;
    • Roughly 2.34× when processing 64 concurrent tokens.

    The higher the concurrency, the heavier the per-token VRAM reads, so each card’s effective bandwidth sets the throughput ceiling.

    2.2 Long context: sparse attention cuts compute, but the memory bill remains

    A 1M-token context length places enormous pressure on attention: full attention grows quadratically with sequence length, and the KV cache expands linearly with length.

    The model replaces full attention with sparse attention (DSA) in some layers. The idea is to use a lightweight scoring mechanism so each query attends only to a few key tokens, reducing compute overhead on long sequences.

    What needs clarifying: sparse attention mainly reduces compute. KV cache storage still depends on the number of layers, KV heads, and cache precision—unless paired with cache compression or eviction. Whether a full 1M context can actually be used hinges on the KV cache budget. This also explains the overall design choices:

    • Some layers keep full attention to integrate global information;
    • The remaining layers use sparse attention to control compute on long sequences.

    2.3 Multimodal and agent capabilities: perception and execution in one pipeline

    The vision encoder lets the model read screenshots, charts, and interfaces directly, then use tool-calling to complete multi-step tasks. This pipeline suits automated testing, code repository analysis, and document processing—but it also brings new deployment requirements:

    • Image input must first pass through the vision encoder, so prefill overhead is noticeably higher than for text-only input;
    • Agent scenarios typically involve multiple rounds of tool calls, so time-to-first-token (TTFT) directly affects task completion time.

    2.4 MXFP4 quantization: the benefits and risks of native 4-bit

    Weights use MXFP4—4-bit floating point in the Microscaling format, where every 32 elements share a scaling factor, trading tiny metadata overhead for sufficient dynamic range. Because it is the product of quantization-aware training (QAT), weights and activations participate in low-precision compute during training, generally yielding higher fidelity than “train at full precision, then quantize.”

    However, the developers released only this single precision version—no BF16 or FP8 variant. This creates two constraints:

    • No precision comparison: third-party reproduction can only be done at FP4, with no high-precision version to compare against.
    • No fallback path: if the inference stack does not support native MXFP4 compute, dequantization consumes extra VRAM, and the 1.56TB of weights would be hard to fit on 8 cards.

    3. How to read the vendor’s reported results

    Some of the developer-reported results (all comparison targets are the reference models the vendor listed):

    Evaluation dimension This model Comparison
    Terminal command-line tasks (v4.0) 65.7% Slightly below reference model A’s 66.4%, above reference model B’s 58.2%
    Long-horizon software engineering 51.7% Above reference model A’s 46.3%
    Terminal science tasks 38.1% Below reference model A’s 63.3%
    High-difficulty general knowledge & reasoning (curated subset) 39.4% Below reference model A’s 60.9%
    Overall across 37 benchmarks 16 leads Fewer than half

    An independent evaluation organization gave it a composite intelligence score of 55, ranking it fourth globally. Overall, this class of model is “strong in coding and agents, relatively weaker in science and deep reasoning,” showing a clear mix of strengths and weaknesses, which calls for a level-headed reading:

    1. Separate self-reported from independently measured: vendor numbers are affected by evaluation framework, prompt templates, sampling parameters, and tool permissions, so directly comparing scores from different sources has limited meaning.
    2. Look at task distribution, not the overall score: using self-reported data, leading items account for about 43%, so selection should center on small-scale testing against your own business task distribution.
    3. Add independent verification: independent evaluators usually provide composite scores and rankings that can be cross-referenced with official data, but they still cannot replace a POC validated on your business scenario.

    4. License compliance: open weights do not equal open source

    This is the part that most needs separate scrutiny for this rollout. The hosting platform marks the license as license:other, a custom license identical word-for-word to the previous generation’s terms, with only the name replaced.

    MIT-style core grant. The license’s core grant is MIT-style, permitting free use, copying, modification, distribution, fine-tuning, and commercial use. For individual developers and the vast majority of SMBs, this openness is enough to support derivative development, on-premises deployment, and even commercialization.

    Two restrictions targeting hyperscale business. Signing a separate agreement with the model provider is required before commercial use if either of the following is triggered:

    • Offering the model as a service (MaaS) with cumulative trailing-12-month revenue exceeding US$20 million;
    • A commercial product with more than 100 million monthly active users.

    Key takeaways:

    • The restrictions target only “large-scale external business” and do not affect downloading, fine-tuning, or internal use;
    • Whether something counts as “model as a service” depends on business form—selling a self-built API externally is legally different from internal enterprise use;
    • The MAU metric must be monitored continuously against the product’s reporting period, not judged once at launch;
    • A change in the license name does not imply a change in content—compliance judgments should rest on the license text itself.

    Compliance checklist:

    1. Confirm that distributed or integrated artifacts fully retain the license text and copyright notice;
    2. Define the revenue definition and reporting period for inference services;
    3. Confirm whether fine-tuned derivatives are likewise bound by the original license;
    4. Have legal counsel characterize the boundary of “providing externally”;
    5. Retain audit records of weight versions, checksums, and license versions.

    5. Running locally needs 8 GPUs: the triple constraint of memory, compute, and operations

    5.1 Weights: about 195GB per card across 8 cards

    The model weights are about 1.56TB, so weights alone require total capacity equivalent to eight 200GB-class memory cards. For 512GB and 1.5TB-class inference accelerators, roughly 4 and 2 cards are needed respectively—but VRAM is not the only constraint: the more concentrated the weights per card, the more pronounced the expert-read amplification as batch size grows, and the point of 8 cards is to spread the VRAM and bandwidth pressure. Counting KV cache and framework overhead, actual usage will exceed 1.6TB.

    5.2 KV cache: long context is the biggest variable

    Precision Per token, per layer Per token across 93 layers 8K context 128K context 1M context
    16-bit 2048B ~0.19MB ~1.5GB ~24GB ~186GB
    8-bit 1024B ~0.09MB ~0.8GB ~12GB ~93GB

    Basis: KV heads × head dim × 2 (K and V) × bytes; these are general estimates—actual values follow the model configuration.

    Keeping only 8K context with a single concurrency leaves less than 50GB of headroom; once the full 1M context is used, a single concurrency needs over a hundred GB. Add FP8 cache, multiple concurrencies, and image tokens, and a single 8-GPU node can support only a limited number of simultaneous online tasks.

    5.3 Compute: prefill determines the experience

    Each MoE forward pass computes about 2× the active parameter count, i.e., roughly 2×10¹¹ floating-point operations per token. Generating the first token for a 128K context requires, very roughly, about 2.6×10¹⁶ floating-point operations in the prefill stage. At a whole-machine 4×10¹⁵ FLOPS of FP8 compute and 30% effective utilization, prefill alone takes over 20 seconds.

    This shows that the bottleneck in local deployment is often prefill rather than decoding, so you should prioritize platforms with strong prefill compute and high inter-card interconnect bandwidth, and adopt optimizations such as chunked prefill and prefix caching.

    5.4 Operations: observability and elasticity

    Loading 1.56TB of weights from cold takes time, so elastic scaling cannot be done in seconds. Recommended:

    • Keep 2 or more replicas, spreading traffic via load balancing;
    • Use log services and monitoring alerts to track VRAM usage, TTFT, token generation rate, and queue time;
    • Configure auto-scaling policies with a cold-start buffer.

    6. Choosing among three deployment paths

    Path Suitable for Threshold Cost profile
    Hosted inference service/API Individual developers, quick validation Low, pay per usage Low upfront, unit cost rises with heavy usage
    Elastic cloud self-deployment Teams with ops capability, data must stay in-region Medium-high, needs 8-GPU-class instances and a parallel inference framework Billed by the hour per whole machine, utilization must be assessed
    On-premises Large organizations with strict data compliance High, one-time hardware spend and data-center upgrades Large fixed investment, provisioned for peak

    Whichever path you choose, advance in three steps:

    1. POC validation: evaluate quality and latency on the target task set to decide whether self-deployment is worthwhile;
    2. Capacity load test: load test with realistic concurrency and context length to determine the required number of cards and replicas;
    3. Canary rollout: shift traffic gradually with monitoring alerts and auto-scaling.

    Taking cloud self-deployment as an example, a typical architecture looks like this:

    1. Users / business systems
    2. │ HTTPS
    3. [Load balancer] ──> [Inference service replicas × N] (each an 8-GPU-class instance, tensor/expert parallelism)
    4. │
    5. ┌──────────────┼────────────────┐
    6. [Object storage: weights & images] [Monitoring / log service] [Secret management: API credentials]

    7. Go-live checklist

    • License terms reviewed by legal, with clear monitoring for the revenue and MAU definitions
    • Weight checksums fully recorded and sources traceable
    • Inference framework confirmed to support native MXFP4/MXFP8 compute
    • VRAM budget covers weights, KV cache, and framework overhead, with 20%+ headroom reserved
    • POC and load testing completed against the business task set
    • Load balancing, auto-scaling, and monitoring alerts configured
    • Version upgrade and rollback plan in place

    Conclusion

    2.8T parameters, 1.56TB of weights, and 8 GPUs to start—landing this kind of open-weight large model is a comprehensive challenge. Open weights significantly lower the barrier to acquiring model capability, but licensing terms, hardware cost, and operational complexity remain three unavoidable hurdles for enterprises. Validate effectiveness with your business data first, then estimate resources in the four steps of “weights → VRAM → compute → operations,” and complete the license review early—only then can the model enter production reliably.