2.8T-Parameter Open-Weight MoE Model: Architecture, Quantization, and Deployment Requirements Explained

    A newly released 2.8-trillion-parameter open-weight multimodal model ships with roughly 1.56TB of weights and, according to its developers, is the largest downloadable open model to date. This article unpacks the three questions that actually matter for anyone considering an on-premises deployment: how a Mixture-of-Experts design produces 2.8T parameters while activating only about 104B per token, why 1.56TB of weights pushes the memory requirement to the 192GB-per-card class, and how the model's custom license differs from a genuine OSI-style open source license. It also covers sparse attention for the 1M-token context window, MXFP4 weights with MXFP8 activations, and a practical checklist of decisions to make before you commit hardware budget.

    A Trillion-Parameter Open-Weight Model Lands: A Signal of a New Era

    A model development team has released an open-weight multimodal model with 2.8 trillion (2.8T) total parameters and roughly 1.56TB of weight files on a mainstream model hosting platform, on the timeline it previously committed to. The developers describe it as the largest downloadable open model currently available. The scale goes well beyond any previously published open-weight model and pushes the practical ceiling of “open weights” to a new order of magnitude.

    But “downloadable” and “runnable” are two different things. This release raises at least three questions worth discussing seriously: what kind of architecture supports this many parameters, what 1.56TB of weights means for hardware, and whether “open weights” is the same thing the industry calls “open source.” This article follows those three threads.

    Inside the Architecture: How 2.8T Parameters Are Actually Assembled

    Mixture of Experts and Sparse Activation

    2.8T is the total parameter count, which does not mean every inference pass touches all of those weights. The model uses a Mixture of Experts (MoE) design: 896 experts internally, with each token routed to only 16 of them, plus 2 shared experts that always participate, across 93 layers. This “huge total, sparse activation” pattern is the mainstream route for very large models. Each forward pass activates only about 104B parameters, trading relatively controlled compute for model capacity and breadth of knowledge.

    This distinction matters: total parameter count determines memory footprint, while activated parameter count determines compute cost. It is also the basis for the later point that a large model may demand enormous memory without requiring extreme raw compute.

    Sparse Attention: Saving Compute on Long Contexts

    The model claims support for a 1 million token context. The dominant cost of very long context is that attention compute grows with the square of sequence length. To address this, the model replaces some of the originally full-attention layers with a sparse attention mechanism, so each token attends only to a subset of key positions. This significantly reduces attention overhead for long-document workloads and is a common approach to improving inference efficiency in long-context models.

    Low-Bit Quantization: MXFP4 Weights, MXFP8 Activations

    Weights are stored in MXFP4 (4-bit floating point with block-wise scaling) and activations are computed in MXFP8 (8-bit floating point). This is the product of quantization-aware training (QAT): quantization error is simulated during training so the model adapts to low precision in advance. Notably, this is the only version released. There are no accompanying BF16 or FP8 higher-precision weights.

    The consequence: what you get is a version already optimized for deployment. The benefit is a substantial reduction in memory footprint and transfer size. The cost is that precision is effectively fixed at a lower level, and you cannot roll back to higher precision to run your own quantization pass.

    Native Multimodality

    The model ships with an in-house visual encoder and accepts image input, making it a natively multimodal agent model. That means it handles not just text-only inference but mixed image-and-text tasks and tool-calling agent scenarios.

    The Local Deployment Math for 1.56TB of Weights

    Storage and Download

    The 1.56TB is spread across roughly 112 files and 96 weight shards. For individuals and teams, the first requirement is an appropriate storage and network budget. Major model hosting platforms and community mirrors generally offer resumable or synchronized downloads; for files this large, prefer a mirror close to your region to reduce the chance of failed pulls.

    GPU Memory Is the Real Gate

    The hard part is not disk space but memory. A rough estimate:

    1. Weights ÷ number of accelerators = weights per card
    2. 1.56 TB ÷ 8 ≈ 195 GB per card

    To fit the full set of weights on a single 8-card node, each card needs roughly 192GB of memory. That is high-memory accelerator territory. And this figure does not include the KV cache or activation overhead. A 1 million token context means the KV cache will consume a large amount of additional memory, and the pressure grows with every concurrent request.

    So “8 cards minimum” does not refer to the 80GB accelerators common in typical data centers. It points to hardware in the roughly 192GB-per-card class. If you only have 80GB-class cards, you will generally need to span multiple machines into a multi-node cluster, which raises inference latency and networking complexity considerably.

    Engineering Points on the Inference Side

    Low-bit weights reduce the size of the footprint, but low-precision compute places higher demands on the inference framework: kernel support for the quantized format, the quantization/dequantization flow, and the sharding strategy for tensor parallelism and pipeline parallelism. Before deploying, verify three things in particular:

    • Whether your chosen inference framework natively supports this quantization format.
    • Whether there is enough memory headroom once the model is sharded across cards.
    • Whether KV cache management stays efficient under long context.

    How to Read the Vendor Benchmarks: Self-Reported vs. Independent

    The developers published a set of comparison figures. By their own reporting: on coding-oriented tests (Terminal-Bench-style), the model scores about 65.7%, slightly below a leading closed model’s 66.4% but above another benchmark model’s 58.2%. On software engineering agent tasks (SWE-style), it leads at 51.7% versus a competitor’s 46.3%. The advantage is not across the board: on science benchmarks it scores only about 38.1%, clearly behind a competitor’s 63.3%, and on hard reasoning benchmarks (HLE-style) it trails 39.4% to 60.9%.

    The overall picture is mixed: stronger on coding and agentic capability, weaker on science and some forms of deep reasoning. A third-party independent evaluation’s intelligence index ranks it fourth globally.

    One caution for readers: vendor-reported benchmarks should be treated as reference points, not conclusions. Three reasons:

    1. The evaluation protocol, prompts, and sample selection may favor the model.
    2. A single leaderboard cannot cover the distribution of real workloads.
    3. Models iterate quickly, so the data has a short shelf life.

    For engineering decisions, always run a head-to-head comparison on real samples from your own workloads, and give independent third-party evaluations priority.

    “Open Weights” Does Not Mean “Open Source”: Read the License

    This is the most easily misread and most important point about the release.

    The model is published under a custom license, labeled “other” on the hosting platform rather than as a standard open source license like MIT or Apache. The core grant is MIT-style — free use, copy, modification, distribution, fine-tuning, and commercial use — but two restrictions are layered on top for larger-scale commercial operations:

    • Companies offering the model as a service (MaaS) with cumulative revenue above USD 20 million over any 12-month period must sign a separate agreement before commercial use.
    • Commercial products whose monthly active users exceed a certain scale (on the order of hundreds of millions) likewise require separate licensing.

    For the large majority of individual developers, research institutions, and small to mid-sized teams, these restrictions have essentially no impact on normal use. For major enterprises, especially those pursuing large-scale commercialization, these two lines are directly relevant to both compliance and the business model, and each one needs to be checked before integration. The broader lesson: when you see “weights available for download,” open the license text itself and distinguish between “open weights” and “open source” in the OSI sense. The former means you were given the weights; the latter additionally requires that the licensing terms not impose discriminatory restrictions on particular uses.

    Practical Decisions Before You Deploy

    Before committing to an on-premises deployment, answer these questions first:

    1. Does the workload actually need a trillion-parameter model? If tasks are concentrated in a vertical domain, a mid-sized model fine-tuned on your domain may offer better value.
    2. Does the hardware budget match? By the earlier estimate, a single 8-card node requires roughly 192GB of memory per card, and a multi-node design adds networking and parallelization overhead.
    3. Is the precision sufficient? Only a low-bit version is available. If your workload is highly sensitive to numerical precision, evaluate the accuracy loss introduced by quantization.
    4. Does the license allow commercial use? For teams with high revenue or large user counts, check the restrictions in the custom license first.
    5. Is a cloud-based alternative available? For workloads with variable or elastic demand, a managed inference service built on general-purpose cloud capabilities — container platforms, object storage, monitoring and alerting — is often less work than running your own cluster.

    Bottom Line

    The arrival of a trillion-parameter open-weight model confirms that the trend of the strongest models becoming downloadable is accelerating. For engineering teams, though, three things are non-negotiable: understanding capability boundaries, accounting for the hardware threshold, and getting the license right. Only when you understand MoE’s sparse activation, work out the memory math behind 1.56TB, and recognize the licensing difference between “open weights” and “open source” can “downloadable” turn into “affordable, correct, and compliant to use.” A sober assessment and workload-driven selection are the most reliable posture in the era of very large models.