# Deploying a 2.8 Trillion-Parameter Open-Weight MoE Model: Architecture, License, and Hardware Thresholds

    This article uses the recent open-weight release of a ~2.8T-parameter multimodal MoE model as a case study for understanding what actually ships when a frontier model's weights go public. It covers what is inside the weight package, how to read its custom license, the three real costs of self-hosting (resident weights of ~1.56TB, KV cache at long context, and compute, memory bandwidth, and inter-GPU communication), why the release exists only in MXFP4/MXFP8 precision and what that means for inference and fine-tuning, and how to interpret vendor-reported benchmark numbers before choosing between local deployment and a managed inference service.

    Introduction: From “Available in the Cloud” to “Weights on Disk”

    On November 4, a multimodal Mixture-of-Experts (MoE) model with roughly 2.8 trillion total parameters published its full weights, as promised. The download is about 1.56TB, and local inference starts at eight GPUs.

    Publishing weights and offering cloud API access are fundamentally different things. When you consume a model through an API, you face a stable endpoint: compute, scheduling, and operations are the platform’s problem. When you download weights, you take on all of it:

    • You provision your own GPU memory, bandwidth, and storage.
    • You have to read and interpret the license.
    • You have to judge whether the vendor’s published scores apply to your workload.

    Using this release as a sample, this article breaks down four questions: what is inside the weight package, how to read the license, where the real barriers to local deployment sit, and how the published benchmark results should be interpreted.

    1. What Happened: A “Service First, Weights Later” Release Path

    1. October 24: The model launched first as a cloud API and web chat interface. The publisher committed at the same time to releasing full weights before November 4.
    2. November 2: A weight hosting repository was created but remained private.
    3. November 4: The repository was still closed in the morning and opened that evening, with no application or approval required. It contains 112 files, of which 96 are sharded weight files totaling about 1.56TB.

    This sequence is becoming the common release rhythm for extremely large open-weight models: absorb demand with a hosted service first, collect feedback, then publish weights to honor the openness commitment. For adopters, the real evaluation work starts only after the weights land.

    One point needs emphasis: what was published are the weights, not training code, data, or the full training pipeline. The later debate about whether this counts as “open source” traces back to exactly that.

    2. Model Breakdown: How 2.8 Trillion Parameters Are Structured

    2.1 Total Parameters vs. Activated Parameters: Two Numbers That Must Not Be Mixed

    The model has about 2.8T total parameters, but each token activates only about 104B during the forward pass — an activation ratio of roughly 3.7%, which is typical of a sparse MoE design. The two numbers map to different costs:

    • Total parameters determine deployment cost. All expert weights must be resident in GPU memory or system memory, because you cannot predict which experts the next token will be routed to.
    • Activated parameters determine compute cost. Per-token compute scales roughly with the number of activated parameters.

    Estimating inference cost from 2.8T directly, or estimating memory requirements from 104B, both lead to wrong conclusions. This is the most common confusion when assessing models of this class.

    2.2 Expert Structure: 16 of 896, Plus 2 Shared Experts

    The model has 93 layers. According to the model card, its MoE layers contain 896 routed experts, of which each token selects 16, plus 2 shared experts that always participate:

    • Each token effectively involves 18 experts per layer, roughly 2% of the expert pool.
    • Shared experts carry general knowledge, so every token does not have to relearn basic capabilities; routed experts handle specialization across domains.
    • The output is a weighted merge of routed-expert and shared-expert results — a mainstream design pattern for combining fine-grained experts with shared experts.

    2.3 Long Context: 24 of 93 Layers Switched from Full to Sparse Attention

    Compared with the previous generation, the architecture changed in one place only: in 24 of the 93 layers, full attention was replaced with sparse attention (DSA). The other 69 layers keep full attention unchanged.

    Full attention’s compute grows quadratically with context length. At a million tokens, retaining complete history for every token becomes expensive very quickly. Sparse attention uses learnable token selection so that each query attends only to a small set of the most relevant historical tokens, bringing complexity down to near-linear.

    There are two common reasons for a mixed design — some sparse layers, some full — rather than making every layer sparse:

    • Deep layers typically need global information integration across long ranges.
    • A small number of full-attention layers help preserve precise retrieval capability.

    The officially stated context window is 1 million tokens. But “supported” is not the same as “usable”: the real cost of long context is the KV cache, and the math is in Section 4.

    2.4 Multimodal and Agentic Capabilities

    The model card positions it as a natively multimodal agent model, trained further from the previous generation:

    • Native multimodality: visual tokens produced by the vision encoder (MoonViT-V2) are processed in the same backbone as text tokens, rather than bolting a vision module onto a text model.
    • Agentic capability: emphasis on multi-turn tool calls and long-horizon tasks, including writing and executing code, retrieving information, and chaining multiple steps toward a goal.

    This positioning partly explains the benchmark results in Section 6: capabilities related to coding and tool use are relatively strong, while foundational science and general reasoning are comparatively weak.

    2.5 Structural Comparison with the Previous Generation

    Dimension Previous Generation This Release
    Total / per-token activated parameters ~2.8T / ~104B Unchanged
    Experts and routing strategy 896 routed experts, top-16, plus 2 shared experts Unchanged
    Layers 93 Unchanged
    Attention mechanism Full attention in all layers 24 layers switched to sparse attention, rest unchanged
    Context window — 1M tokens
    Vision encoder — MoonViT-V2
    Published weight precision — MXFP4 weights / MXFP8 activations

    The structural changes are minimal, which indicates that this generation’s gains come mainly from continued pretraining, post-training, and data mixture, not architectural innovation. It also means existing deployment and fine-tuning experience from the previous generation carries over to a large extent.

    3. The Quantization Strategy: Why There Is Only a 4-Bit Version

    3.1 MXFP4/MXFP8: Low-Precision Formats with Block-Shared Scale Factors

    The publisher released a single weight version: MXFP4 weights with MXFP8 activations. There is no BF16 or FP8 variant.

    MXFP4/MXFP8 are microscaling floating-point formats. Their defining trait is that every 32 elements form a block that shares one 8-bit scale factor, while elements within the block are each represented in 4 or 8 bits. Compared with a single scale factor for the entire tensor, block-level scaling adapts better to the long-tailed distribution of activations. By that arithmetic, weights average about 4.5 bits per parameter.

    3.2 Quantization-Aware Training: The Error Was Absorbed During Training

    The MXFP4 weights are not a post-training compression of a higher-precision model; they are the result of quantization-aware training (QAT). The model learned with quantization noise during training to compensate for the precision loss of low-bit representations.

    Two conclusions follow:

    1. Higher precision cannot be recovered after release. Dequantizing back to BF16 only inflates size without restoring discarded information. At 2 bytes per parameter, 2.8T parameters dequantize to roughly 5.6TB.
    2. Shipping a single precision lowers distribution cost. Users do not need to quantize the model themselves and do not have to worry about precision differences caused by inconsistent quantization across toolchains.

    3.3 What the Single Version Means for Users

    • Inference frameworks: running this efficiently requires an inference engine that supports the MX family of formats. Otherwise you must dequantize first, or wait for the software stack to catch up.
    • Fine-tuning: training on 4-bit weights places higher demands on frameworks and memory optimizations. Parameter-efficient approaches in the QLoRA family are generally required; full-parameter fine-tuning is essentially unrealistic.
    • Precision validation: it is advisable to run targeted regression on your own business datasets for math-heavy computation, code generation, and long-document understanding — tasks that are more sensitive to quantization error.

    4. Deployment Thresholds: The Three Bills Behind “8 GPUs Minimum”

    4.1 First Bill: Resident Weights

    Where it lives Estimated capacity needed Notes
    GPU memory 1.5TB+ 96 shards totaling ~1.56TB
    Host memory 1.5TB+ Required when memory offloading is used
    Disk 2TB+ Download package, extraction, and cached copies

    At 4-bit-or-higher precision, 2.8T parameters need roughly 1.4TB of storage, which matches the 1.56TB download closely; the extra comes mainly from scale factors and a small number of higher-precision parameters. This scale has two direct consequences:

    • Single node, 8 GPUs: even with 96GB per card, eight cards total only 768GB. So “8 GPUs minimum” typically means one of two setups: cards with 128GB or more of memory each, or an offloading scheme that combines GPU memory with host memory.
    • Single GPU is out of the question: even counting only the ~104B activated parameters per token (about 52GB at 4 bits), all expert weights must be randomly accessible. This differs from the on-device approach of loading only “hot” experts.

    4.2 Second Bill: The KV Cache

    Beyond weights, the second major consumer of GPU memory is the KV cache. The standard estimation formula is:

    1. KV cache bytes = 2 × num_layers × num_kv_heads × head_dim × bytes_per_element × seq_len

    With 93 layers, an assumed 8 KV heads, a head dimension of 128, and an FP16 cache:

    • about 380KB per token;
    • roughly 38GB for 100K tokens;
    • roughly 380GB for 1M tokens.

    A single full-length context’s KV cache alone, then, equals the capacity of several high-memory accelerators. The following code makes the conversion quick:

    1. def kv_cache_gb(layers, kv_heads, head_dim, bytes_per_elem, seq_len):
    2. # The factor of 2 accounts for one copy each of K and V
    3. return 2 * layers * kv_heads * head_dim * bytes_per_elem * seq_len / 1e9
    4. print(kv_cache_gb(93, 8, 128, 2, 100_000)) # ~38.1 GB
    5. print(kv_cache_gb(93, 8, 128, 1, 1_000_000)) # ~190.5 GB with FP8-quantized KV

    Common mitigations include KV quantization (FP8 or lower), paged cache management, prefix caching reuse (multi-turn conversations and agent tasks can share system prompts and historical context), and loading only the historical tokens required by sparse layers on demand. When offloading to host memory or remote storage is involved, the latency cost also needs to be evaluated.

    4.3 Third Bill: Compute, Bandwidth, and Communication

    • Compute: estimating forward compute as roughly twice the activated parameters, each token needs about 2×10¹¹ floating-point operations. On the effective throughput of eight high-performance cards, theoretical per-token compute time lands in the millisecond range.
    • Memory bandwidth: generating each token in principle requires reading about 52GB of activated weights. Summed across eight GPUs, the theoretical ceiling for single-request generation speed is a few hundred tokens per second.
    • Prefill: processing 1M input tokens takes roughly 2×10¹⁷ floating-point operations. At an estimated 80 PFLOPS of aggregate effective compute across eight cards, backbone computation alone takes about 2.6 seconds — and in practice it takes longer.
    • Communication: 896 experts distributed across 8 GPUs works out to roughly 112 experts per card. Top-16 routing means most tokens activate experts spread over multiple GPUs, so All-to-All communication under expert parallelism readily becomes the bottleneck. High-speed inter-GPU interconnect matters more than single-card peak compute.

    Under concurrent requests, weights are read once to serve a batch of tokens, so throughput improves markedly — but memory and bandwidth pressure rise at the same time.

    Option Typical hardware Where weights live Suitable for Main cost
    Single node, high memory 8× 128GB-class accelerators Fully resident in GPU memory Local research, data-sensitive workloads High hardware investment
    Single node with memory offload 8× mainstream training GPUs + ≥1.5TB RAM GPU memory combined with host memory Low-concurrency validation Noticeably higher single-request latency
    Multi-node expert parallelism Multiple nodes, 8 GPUs each Shards resident in GPU memory Team-level production serving Operational complexity, dependence on fast networking

    For multi-node deployments, the following principles are a useful reference (field names are illustrative only; follow the documentation of the inference framework you use):

    1. # Single-node, 8-GPU example
    2. tensor_parallel_size: 8
    3. enable_expert_parallel: true
    4. max_model_len: 131072 # Set according to actual workload; avoid maxing out blindly
    5. kv_cache_dtype: fp8
    6. enable_prefix_caching: true # Reuse system prompts and multi-turn history
    7. # For multi-node, add pipeline parallelism on top (example: 2 nodes × 8 GPUs)
    8. # pipeline_parallel_size: 2
    9. # data_parallel_size: 1

    If you lack the hardware or operational capacity, a more realistic choice is a managed inference service: pay per token while the provider handles scaling and availability. That fits teams with spiky demand or those still in the validation stage.

    On the engineering side, general cloud capabilities are commonly used for:

    • Object storage to host and distribute the shards;
    • Container platforms to orchestrate the inference services;
    • Monitoring and alerting to track throughput, latency, and memory utilization;
    • Logging services to record requests and call traces.

    5. Licensing: Open Weights Are Not Open Source

    5.1 License Type: A Custom License, Not a Standard Open Source One

    The repository ships with a custom license, categorized as “other” on model hosting platforms rather than as MIT or Apache-2.0 — licenses recognized by open source governance bodies. Compared with the previous generation, the license text is word-for-word identical; only the name differs.

    The core grant is MIT-style, permitting free use, copying, modification, distribution, fine-tuning, and commercial use. That is also why it is widely described as “open weights.”

    5.2 Additional Conditions: Constraints Beyond the Permissive Grant

    Additional terms usually accompany the MIT-style core. For models of this scale, those terms typically cluster in two areas:

    1. Acceptable use compliance: requirements to follow an acceptable use policy, prohibitions on unlawful use, and separate authorization for very large-scale commercial use.
    2. Interpretation risk: custom text can be read multiple ways in non-standard scenarios — for example, whether a given deployment counts as “large-scale service,” and whether outputs may be used to train other models.

    To be clear: the definitive source is the license text bundled with the weight repository. This article is not a substitute for legal advice, and commercial use should be cleared with your legal team.

    5.3 Enterprise Compliance Checklist

    • The license version is tied to the model version; re-check the terms after upgrading the model.
    • Define the boundaries of commercial use, any user-scale thresholds, and attribution requirements.
    • Confirm distribution obligations for fine-tuned models, and whether distillation and data synthesis are permitted.
    • Fine-tuning data is subject to compliance constraints as well, including personal information, copyright, and lawful data sourcing.
    • Retain checksums of the weight package and a snapshot of the license text to support audits.

    6. Reading the Scores: Self-Reported Results vs. Independent Evaluation

    6.1 Summary of Vendor-Reported Results

    The table below is entirely vendor-reported and has not been independently reproduced:

    Benchmark (described generically) This model Comparison closed-source flagship Outcome
    Computer/terminal operation, aggregate 65.7% 66.4% Close
    Long-horizon software engineering 51.7% 46.3% Ahead
    Computer operation, science subset 38.1% 63.3% Clear gap
    Hard composite reasoning 39.4% 60.9% Clear gap

    The publisher claims superiority on 16 of 37 benchmarks — meaning more than half are not favorable. Separately, a third-party evaluation platform’s composite intelligence index scores it at 55, placing it fourth globally.

    Overall, the model’s strengths concentrate in coding and agent tasks, with a clear gap in science and general reasoning.

    6.2 How to Interpret These Results Properly

    1. Separate self-reported from reproduced: self-reported numbers reflect the evaluation configuration chosen by the vendor. Before reproducing, standardize prompt templates, decoding parameters, and evaluation scripts.
    2. Check the subset scope: a close aggregate score does not mean balance across subsets. The large science gap shows the capability profile is uneven.
    3. Assess fit between benchmark and use case: if your workload centers on code generation and tool calls, the corresponding benchmarks are more informative; if it centers on knowledge Q&A and scientific computation, rely on independently reproduced results.
    4. Validate on private datasets: a new model may have had indirect exposure to public evaluation data, so conclusions should rest on your own business datasets.

    7. Selection Checklist

    Your situation Recommendation
    Long documents, codebases, or agent workloads, with multi-GPU or managed resources available Worth evaluating first
    Only a single GPU or a few consumer-grade cards Not recommended for local deployment; prefer managed service or a smaller model
    Data must stay entirely inside your network Evaluate a high-memory single node or a memory-offload setup
    Latency-sensitive Load-test first and confirm prefill and decode latency meet requirements
    Full-parameter fine-tuning needed Not realistic with current 4-bit weights; consider parameter-efficient fine-tuning
    Before commercial use Complete the license compliance review

    Conclusion

    The real significance of this weight release is not the “largest open-weight model” title. It is a sample for observing how frontier models are engineered:

    • Architecture: capability gains came from continued training plus a minimal structural change (sparse attention in 24 layers).
    • Engineering: 4-bit quantization makes local deployment of a trillion-parameter model conceivable, but the threshold remains high.
    • Compliance: “open weights” and “open source” differ at the license level and warrant careful evaluation.

    For developers and enterprises, the rational approach is: work out the three bills — GPU memory, bandwidth, and license — then validate capability on your own private datasets, and only then decide between local deployment and a managed service.