2.8T-Parameter Open-Source MoE Model: Architecture, Quantization, and Deployment Requirements

    This article examines an open-source MoE model with roughly 2.8T total parameters and about 104B activated per inference. It covers the model’s expert routing, sparse attention, MXFP4/MXFP8 quantization, capability profile, and the GPU memory and cloud deployment requirements implied by its approximately 1.56TB weight footprint.

    Large language models have reached a point where simply increasing parameter count makes it difficult to balance quality, inference speed, and cost at the same time. Mixture-of-Experts (MoE) architectures use sparse activation, invoking only a subset of parameters per request, and have become a common choice for very large models. An open-source model has recently released weights with roughly 2.8T parameters. It performs strongly on coding and agentic tasks, and it also imposes high requirements on deployment hardware.

    This article is based on information disclosed in the source article and explains the model’s architecture, quantization approach, and deployment threshold. Data that the source does not attribute are marked as source statements or estimates.

    1. MoE Architecture: Total Parameters vs. Active Parameters

    According to the source, the model has about 2.8T total parameters and activates about 104B parameters per inference, for an activation ratio of roughly 3.7%. Each token passes through only a small fraction of the parameters, so the compute volume is close to that of a tens-of-billions-parameter dense model, while knowledge capacity comes from all experts.

    The source also states that the network has 93 layers and a total of 896 routed experts. Each token is routed to 16 experts by the router network. In addition, 2 shared experts apply to all tokens and handle general features. The following simplified forward logic omits expert parallelism, load-balancing loss, and numerical precision handling. It is not directly usable in production:

    1. import torch
    2. import torch.nn.functional as F
    3. def moe_forward(x, experts, shared_experts, router_weight, top_k=16):
    4. # x: [tokens, hidden]; router_weight: [hidden, num_experts]
    5. assert top_k <= len(experts)
    6. logits = x @ router_weight
    7. top_vals, top_idx = torch.topk(logits, top_k, dim=-1)
    8. top_w = F.softmax(top_vals, dim=-1) # Normalize only over selected experts
    9. out = torch.zeros_like(x)
    10. for k in range(top_k):
    11. for e in top_idx[:, k].unique().tolist():
    12. mask = top_idx[:, k] == e
    13. out[mask] += top_w[mask, k:k+1] * experts[e](x[mask])
    14. for shared in shared_experts:
    15. out += shared(x)
    16. return out

    The source also describes the model as natively multimodal, with a context window of up to 1 million tokens and an integrated visual encoding module. The source does not describe the visual encoder architecture or provide the evaluation conditions for long context. These details should be confirmed against the official documentation.

    2. Attention Mechanism: Sparse Attention (DSA)

    According to the source, one major change from the previous generation is that full attention in 24 layers has been replaced with sparse attention, abbreviated as DSA. The source does not define the specific implementation of DSA, so the following only explains the general idea of sparse attention.

    The compute cost of standard self-attention grows quadratically with sequence length. At a context length of 1 million tokens, the attention score matrix becomes very large, and both compute and memory costs are difficult to sustain. Sparse attention lets each token attend to only some positions, such as a local window plus a small number of global positions, reducing compute to near-linear growth.

    The trade-off is that long-range dependencies may weaken. The claimed million-token usability should be validated with long-text retrieval and multi-turn question answering tests rather than judged only by the nominal value.

    3. Quantization: MXFP4 Weights and MXFP8 Activations

    The source states that the open weights are about 1.56TB. Weights use the MXFP4 format and activations use MXFP8. The official release provides only this quantized version, with no BF16 or FP8 full-precision version.

    The size can be roughly cross-checked. At 4 bits, 2.8T parameters correspond to about 1.4TB. MXFP4 uses groups of 32 elements that share an 8-bit scaling factor, which adds about 0.25 bits per parameter on average, for a total of about 1.49TB. The remaining difference may come from unquantized layers, embedding tables, or metadata. This part is an estimate; the source does not provide a breakdown.

    The source infers that the model used quantization-aware training (QAT) during training. This inference is plausible, but the source does not state whether it has been officially confirmed, so it should be treated as speculation. The accuracy loss from MXFP4 is task-dependent. It is advisable to compare results before and after quantization on your own evaluation set, with particular attention to mathematical reasoning and long-context tasks.

    4. Capability Profile: Strong on Coding and Agentic Tasks, Weaker on Scientific Reasoning

    According to the source, the model performs strongly on engineering- and automation-oriented benchmarks such as Terminal-Bench 4.0 and SWE Marathon. On academic reasoning tests such as Terminal-Bench-Science and HLE-Diamond, it lags noticeably behind closed-source reference models. The source does not provide specific scores, evaluation versions, or comparison models, so these conclusions can only serve as directional guidance.

    The source attributes this difference to the pretraining data mix, but that explanation has also not been verified. For developers, the practical decision criteria are straightforward. If the workload is mainly code generation, operations automation, or agentic workflows, the model can be included as a candidate. If the main use case is rigorous mathematical reasoning or scientific Q&A, it should first be tested on a small set of your own questions before a decision is made.

    5. Deployment Requirements: How Much GPU Memory Is Actually Needed?

    The source mentions that a single machine with 8 GPUs at 80GB each can support running the model. This claim lacks a calculation basis and needs correction.

    Start with the weights. Eight 80GB GPUs provide 640GB in total, while the weights are about 1.56TB. The weights alone exceed single-machine GPU memory by about 2.4 times. At 80GB per GPU, at least 20 GPUs are needed to hold all weights (1560 ÷ 80 ≈ 19.5, rounded up). With GPUs that have 141GB of memory, at least 12 are needed (1560 ÷ 141 ≈ 11.1, rounded up).

    Overall GPU memory requirements can be estimated with the following relationship:

    GPU memory requirement ≈ weight memory + KV cache + activations and temporary buffers + communication and framework overhead

    The KV cache can be estimated as:

    KV cache bytes ≈ 2 × layers × KV heads × head dimension × sequence length × bytes per element × concurrent requests

    The source does not provide the number of KV heads, and the cache structure of sparse attention layers may differ from standard layers, so the KV cache cannot be calculated precisely. In actual deployment, measure peak GPU memory in a small-scale environment first, then determine the number of GPUs.

    There are three common ways to handle the case where weights exceed single-machine GPU memory.

    1. Multi-node, multi-GPU: At least two nodes are required, using tensor parallelism or expert parallelism. Cross-node communication can easily become a bottleneck, so a high-speed interconnect such as InfiniBand or RoCE is needed.
    2. CPU or SSD offloading: This can make the model run, but throughput will drop noticeably. It is suitable for offline batch processing, not for low-latency online services.
    3. Cloud elastic instances: Rent multi-GPU instances on demand. This suits teams with fluctuating traffic that do not want to bear hardware procurement costs.

    Therefore, a single machine with 8 × 80GB GPUs cannot fully hold this model’s weights. Whether it can be deployed depends on whether offloading is acceptable, which quantized version is used, and the throughput and latency requirements.

    6. Cloud Deployment: Implementation Steps

    If you choose cloud deployment, the following steps are recommended.

    1. Weight management: Store the 1.56TB of weights in object storage, and record the version number and SHA256 checksum to avoid loading incorrect files during deployment.
    2. Instance selection: Prefer multi-GPU instances that meet both the total GPU memory and inter-GPU interconnect bandwidth requirements. Test first-token latency and throughput with small batches first.
    3. Cold start: Weight loading may take a long time, and the actual duration needs to be measured. Keep warm standby instances before autoscaling; otherwise, scaling may not keep up during traffic peaks.
    4. Monitoring metrics: Track GPU memory usage, tokens generated per second, queue time, and error rate. Logs and alerts should cover out-of-memory (OOM) events.
    5. Cost accounting: Compare hourly billing with long-term reserved instances, and include idle time in the total cost.