2.8T MoE Model Deployment: GPU Memory and Licensing Checks First

    Deploying a 2.8-trillion-parameter MoE model requires more than checking its active parameter count. This guide examines the reported architecture, MXFP4 quantization, task suitability, and custom license, then walks through GPU memory sizing, KV cache estimates, download verification, storage planning, and cloud infrastructure checks before deployment.

    1. Architecture: 2.8T Total Parameters, About 104B Active per Token

    The model is further trained from its predecessor’s base model and positioned as a native multimodal agent model. It uses a mixture-of-experts (MoE) architecture with 2.8 trillion total parameters, but only about 104 billion parameters are activated for each token. This provides substantial model capacity while keeping per-token computation more manageable than the total parameter count might suggest.

    According to the publisher, the model has 896 expert networks. Its router selects 16 experts for each token, while two shared experts always participate to support consistent extraction of foundational features. The model reportedly has 93 layers, a one-million-token context window, and a separate vision encoder.

    These specifications come from the publisher’s public materials and have not been independently verified item by item. Check the model repository’s configuration files for the expert count, layer count, maximum context length, and related settings.

    One of the most significant changes from the predecessor is in the attention layers: 24 layers reportedly switch from full attention to sparse attention. Full attention has quadratic computational complexity with respect to context length. Sparse attention restricts each token to a local region or tokens at selected intervals, reducing computation and making million-token inference more practical.

    However, sparse attention has limits:

    • It primarily reduces computation, not necessarily KV cache memory. Cache savings depend on whether the inference engine retains only the KV data needed by the attention pattern.
    • Speedups depend on kernel implementation, batch size, and sequence length.
    • Evaluation should record output tokens per second, time to first token, and peak GPU memory, using a full-attention baseline at the same input length.

    2. Quantization: MXFP4 Weights and MXFP8 Activations

    Storing 2.8 trillion parameters in BF16 would require approximately 5.6 TB for weights alone. The publisher therefore uses low-bit quantization: MXFP4 for weights and MXFP8 for activations.

    MXFP4 uses 4 bits per element, with one shared 8-bit scale for every 32 elements. This averages approximately 4.25 bits per parameter:

    1. Average bits per parameter = 4 + 8 / 32 = 4.25

    At that rate, 2.8 trillion parameters would occupy roughly 1.49 TB, close to the reported 1.56 TB download size. The difference may reflect unquantized components or metadata, but this has not been verified.

    The publisher describes this version as the result of quantization-aware training (QAT), which introduces quantization noise during training. Post-training quantization (PTQ), by contrast, compresses a model after training. Their accuracy characteristics can differ; consult the model card for the specific method used.

    At the time described in the source materials, the official release includes only this quantized version, with no BF16 or FP8 alternative. A direct comparison against a higher-precision release is therefore unavailable. Alternative reference points include:

    • The predecessor’s base model on the same tasks.
    • Human-labeled reference answers.
    • Variation across repeated samples from the same model, as a baseline for output noise.

    Check Hardware and Framework Support

    MXFP4 support varies by chip. GPUs or ASICs with native support can compute directly in the format. Other hardware may need to dequantize weights to a higher precision, reducing the expected speed and memory benefits.

    Before deployment, confirm:

    1. Whether the target hardware and inference framework provide MXFP4 compute kernels.
    2. Whether dequantization is required, and whether it occurs during loading or at runtime.
    3. Accuracy, throughput, and peak GPU memory on your own evaluation set.

    The evaluation set should include both representative inputs and edge cases. Fix and record the prompt template, decoding settings—including temperature, top_p, and max_tokens—and the number of repetitions. Run each test case at least three times to observe output variability.

    3. Capabilities: Stronger in Coding and Agent Tasks, Weaker in Scientific Reasoning

    The publisher’s materials indicate stronger performance in coding and automated agent tasks. Expert specialization in an MoE architecture may help support capabilities such as code generation, tool use, and task orchestration.

    The reported performance is weaker than that of leading closed-weight models on scientific reasoning and multi-step mathematical reasoning.

    This article does not reproduce benchmark scores. Benchmark names, versions, evaluation protocols, and test dates can materially affect results, so unverified numbers should not drive model selection.

    Task category Relative performance reported by the publisher Selection guidance
    Code generation and repository analysis Stronger Include it as a candidate and validate it on internal code tasks
    Tool use and task orchestration Stronger Consider it for agent projects, with particular attention to tool-call reliability
    Long-document information extraction Potential advantage, subject to long-context testing Test recall and accuracy on real long documents first
    Scientific computation and multi-step mathematical reasoning Relatively weaker Do not use it as the sole model; require human review

    For reproducible evaluation, fix the dataset version, prompt template, decoding parameters, and random seed. Document the human-review sample size—for example, 20 cases per task category.

    Use the same scoring criteria across all candidate models. Otherwise, the results will not be directly comparable.

    4. Licensing: Custom Open Weights, Not OSI-Standard Open Source

    The model uses a custom license rather than a standard license such as MIT or Apache 2.0. Its core permissions resemble MIT-style terms, allowing use, copying, modification, distribution, fine-tuning, and commercial use.

    However, it imposes additional conditions on two categories of large commercial entities:

    • Companies providing the model as a service whose revenue exceeds a specified threshold over 12 consecutive months.
    • Entities whose commercial products exceed a specified monthly active user threshold.

    Commercial entities meeting either condition must enter into a separate agreement with the publisher before commercial use. The specific thresholds are not reproduced here; verify them in the applicable license.

    License Review Checklist

    1. Locate the threshold clauses. Find the revenue and monthly active user thresholds in LICENSE, and record the relevant clause numbers.
    2. Check the definition of model as a service. Determine whether your offering falls within it, including whether usage-based external access is covered.
    3. Confirm measurement periods and scope. Check whether revenue is measured over 12 consecutive months or another period, whether monthly active users use calendar months or rolling windows, and whether affiliates are included.
    4. Archive the license version. Record the publication date, version number, and SHA-256 hash of the downloaded license file.
    5. Obtain an agreement if required. If the additional conditions apply, contact the publisher and complete the agreement before commercial use.

    Small and midsize teams may not reach these thresholds, but they still need to review every applicable clause. Large internet companies and cloud providers should obtain legal approval before committing to the project.

    5. Deployment Sizing: Can Eight GPUs Run It?

    The answer depends on memory per GPU. The reported weight files total approximately 1,560 GB in decimal units. Fewer GPUs mean each device must hold a larger share of those weights.

    5.1 A Quick Weight-Memory Check

    The following table compares aggregate capacity with the reported weight size. It excludes KV cache, activations, and framework overhead.

    Memory per GPU Total with 8 GPUs Total with 16 GPUs Can the weights fit?
    80 GB 640 GB 1,280 GB Neither configuration fits; roughly 25 GPUs are needed with a 20% reserve
    141 GB 1,128 GB 2,256 GB Eight do not fit; 16 fit the weights
    192 GB 1,536 GB 3,072 GB Eight fall short by about 24 GB, with no headroom; 16 fit the weights
    288 GB 2,304 GB 4,608 GB Eight theoretically fit, leaving about 744 GB before runtime allocations

    Among these configurations, eight-GPU deployment requires the 288 GB memory tier. Even eight 192 GB GPUs cannot hold all the reported weights.

    Actual deployment also needs room for KV cache, activations, and framework overhead. A preliminary GPU-count estimate is:

    1. N ≥ W / [M × (1 − r)]

    Where:

    • N is the required GPU count, rounded up.
    • W is total weight memory.
    • M is memory per GPU.
    • r is the reserved fraction of GPU memory.

    A starting reserve of 15%–25% is reasonable for planning, but must be validated through testing. With a 20% reserve and 1,560 GB of weights, the estimates are:

    Memory per GPU Estimated minimum GPU count
    80 GB 25
    141 GB 14
    192 GB 11
    288 GB 7

    Round these counts up to a parallel configuration supported by the framework, such as 8, 16, or 32 GPUs.

    The reported 896 experts divide evenly across 8, 16, and 32 partitions. Whether the inference framework supports the corresponding expert-parallel layout remains to be verified.

    5.2 Estimating KV Cache Memory

    For a conventional KV cache, estimate memory as:

    1. KV cache bytes =
    2. 2 × layers × KV heads × head_dim × sequence length × batch size × bytes per element

    The factor of two accounts for separate key and value caches. Read the actual layer count, KV head count, and head_dim from the model configuration before calculating production requirements.

    Consider this illustrative configuration, which is not a verified description of the model:

    • 93 layers
    • 8 KV heads
    • head_dim of 128
    • BF16 cache storage at 2 bytes per element
    • Sequence length of 1,000,000 tokens
    • Batch size of 1

    The cache footprint per token would be:

    1. 2 × 93 × 8 × 128 × 2 = 380,928 bytes

    For one million tokens:

    1. 380,928 × 1,000,000 ≈ 381 GB

    A single million-token request can therefore require hundreds of gigabytes of KV cache in this example.

    This estimate assumes cache storage across all 93 layers. It would overestimate memory if the implementation retained KV only for the full-attention layers. FP8 cache storage could halve the footprint relative to BF16, but its accuracy impact requires separate evaluation.

    5.3 Activation and Framework Headroom

    Activation memory and framework overhead grow with batch size and concurrent requests. Begin with a 15%–25% reserve, then adjust it through load testing.

    Record peak GPU memory, not just average usage. A configuration that fits under light load may fail at peak concurrency.

    5.4 Download Verification and Storage Planning

    Use the following download procedure:

    1. Obtain the hash manifest from the release materials and record the expected SHA-256 value for each shard.
    2. Download shards with a tool that supports resumable transfers. Retry only failed shards.
    3. Verify every completed shard against its expected SHA-256 value. Download failed shards again individually rather than overwriting the entire model.
    4. Do not unpack or load files that fail verification.

    Estimate storage requirements as:

    1. Storage required ≈
    2. 1.56 TB × number of copies
    3. + temporary space for unpacking or format conversion
    4. + approximately 10% additional headroom

    For example, two copies require 3.12 TB. Adding 0.5 TB of temporary space brings the subtotal to about 3.6 TB, or roughly 4 TB with headroom.

    Transfer speed also affects preparation time:

    Transfer rate Approximate time for 1,560 GB
    1 GB/s 26 minutes
    100 MB/s 4.3 hours

    For multi-node deployments, first measure storage read speed on one machine before estimating overall model-loading time.

    Common risks include disks filling during download, partially written files being mistaken for complete shards, and files being checked only at download time. Verify them again before loading and retain the verification logs.

    5.5 Choosing Cloud Infrastructure

    Compare cloud configurations on:

    • Memory per GPU.
    • Inter-GPU interconnect bandwidth.
    • Native MXFP4 compute support.
    • Framework support for MoE expert parallelism.
    • Price.

    Expert parallelism introduces substantial communication between GPUs. Insufficient interconnect bandwidth can significantly reduce throughput.

    A simple monthly GPU cost estimate is:

    1. Monthly cost ≈
    2. hourly price per GPU × GPU count × operating hours per day × 30

    Before selecting an instance or cluster, check that:

    • Aggregate GPU memory can hold the weights, KV cache, and runtime reserve simultaneously.
    • The interconnect can handle expert-parallel communication.
    • The inference framework supports MXFP4 weight loading and the required MoE parallelism.
    • Suitable sparse-attention kernels are available. If the framework falls back to full attention, memory use and latency will differ.
    • The framework provides paging or offloading strategies for million-token contexts.

    Use each cloud provider’s official documentation for current GPU models, specifications, and prices.

    6. Who Should Evaluate It—and What to Verify Next

    Suitable Candidates

    Teams can reasonably move into evaluation if they:

    • Already have a high-memory cluster with 16 or more GPUs and workloads centered on code generation, tool use, or agent orchestration.
    • Have confirmed that the additional license conditions do not apply, or have already signed the required agreement with the publisher.

    Poor Fits or Premature Deployments

    Deployment is less suitable when:

    • The available hardware is eight 80 GB GPUs or less. This is insufficient for local deployment of the full model; consider a smaller model.
    • The main workload is scientific computation or multi-step mathematical reasoning. Do not rely on this model alone; retain human review.
    • A large internet company or cloud provider has not completed legal review. Resolve the licensing requirements before committing to the project.
    1. Verify the model card and configuration. Confirm the reported 896 experts, 93 layers, one-million-token context window, and other architectural details.
    2. Review LICENSE. Complete the five licensing checks above.
    3. Plan storage and verify downloads. Ensure sufficient capacity and complete SHA-256 validation.
    4. Test MXFP4 loading on the target hardware. Record peak GPU memory.
    5. Evaluate against relevant baselines. Use your own task set and record accuracy, throughput, and latency.

    The practical deployment question is not simply whether 104 billion parameters are active per token. The full weight footprint, long-context cache requirements, quantization support, parallel execution, and licensing terms all need to fit before the model is ready for production.