2.8T MoE Open Weights Released: 1.56TB Package and the 8-GPU Barrier

    This article examines the open-weight release of a 2.8T-parameter mixture-of-experts model and what it means for developers and enterprises. It covers the 1.56TB weight package, the model’s 93-layer architecture, 1M-token context support, 4-bit weights with 8-bit activations, benchmark strengths in coding and agentic tasks, weaker scientific reasoning performance, the custom license terms, and the practical eight-GPU deployment threshold.

    On November 4, a research team released the full weights of an ultra-large open-weight model as promised. The release raised two questions: whether the model’s size and architecture truly support the claim that it is the largest open-weight model to date, and whether ordinary developers or enterprises can actually use it. The conclusion is that the weights have clear technical appeal, but the engineering requirements, license boundaries, and deployment costs make it far from a simple “download and run” proposition.

    1. Release timeline: from hosted access to open weights

    In late October, the model first became available through an online interface and web interaction, with a commitment to release the full weights by November 4. On November 4, the weight files appeared on a public model hosting platform. The repository remained closed that morning and opened officially in the evening; users could access it without applying for permission.

    The weight package contains 112 files, including 96 tensor shards, with a total size of about 1.56TB. Some download mirrors may also be available, reducing the difficulty of obtaining such a large file. For ordinary users, the significance of this event is not just that another model can be downloaded, but that an open-weight model has entered public view at this scale for the first time. In the past, models around 1.56TB were more likely to be found inside a limited number of organizations. This public release inevitably pushes the discussion of “how open are open weights, and who can afford them?” to the forefront.

    2. Model architecture: how 2.8T parameters are composed

    According to the public model description, the model is a native multimodal agent model and was further trained from the previous generation. Its total parameter count is 2.8T, but each inference only activates about 104B parameters. This is a typical mixture-of-experts design: the model contains 896 experts, each token selects 16 experts during computation, and 2 shared experts handle more general capabilities.

    The key point of this structure is the decoupling of capacity and compute. The 2.8T parameters provide substantial knowledge capacity and representational space, but only a small part is used in each forward pass. In an ideal case, inference compute does not scale proportionally with total parameters. However, sparse activation reduces compute, not memory usage. All expert weights must still be addressable in VRAM or system memory; otherwise, the model cannot dynamically select experts during inference.

    The model has 93 layers, supports a context length of up to 1 million tokens, and includes a vision encoder that can process image inputs. Compared with the previous generation, the architectural changes are not radical and focus mainly on long-context efficiency: 24 layers of full attention have been replaced with sparse attention. Full attention’s compute and memory costs rise quickly with sequence length, while sparse attention reduces the attention connection range to provide a more controllable resource curve for extremely long contexts.

    The weights are also released in a low-bit format produced after quantization-aware training: 4-bit floating-point weights and 8-bit floating-point activations. The official release provides only this precision version, with no higher-precision weights available. This means the release has already made a “precision for efficiency” tradeoff. Users receive a version optimized for inference efficiency, rather than a general base package that can freely switch between full precision, half precision, or low precision.

    3. Benchmark performance: strong in coding and agents, weaker in scientific reasoning

    Vendor-reported comparison data shows that the model performs strongly on coding and agentic tasks. For example, it scores 65.7% on a terminal-task benchmark, close to the 66.4% of a comparable top-tier closed-source model. On a software-engineering task benchmark, it scores 51.7%, above the latter’s 46.3%. This indicates strong competitiveness in code generation, command execution, tool calling, and long-workflow task decomposition.

    However, the advantage is not comprehensive. On scientific tasks, it scores only 38.1%, clearly lower than the 63.3% of a comparable top-tier model. On a high-difficulty reasoning benchmark, it scores 39.4%, also below the latter’s 60.9%. Overall, it leads on 16 of 37 benchmarks, showing a mixed win-loss pattern. More precisely, it is not an all-around model; it has relative advantages in programming, agent execution, and engineering tasks, while still lagging in scientific computation and complex reasoning.

    A third-party intelligence index score of 55 and a global ranking of fourth also indicate that it has entered the first tier, though it is not yet at the absolute top. For users, these results should not be interpreted simply as “comprehensive superiority” or “comprehensive inferiority.” They should be evaluated according to specific scenarios: if the goal is a coding assistant, automated operations, tool calling, or multimodal agents, it is worth evaluating; if the goal is mathematical proof, scientific question answering, or highly trusted reasoning, more careful validation is needed.

    4. Licensing: open weights are not open source

    The most important point to discuss separately is the license. The repository uses a custom license, marked as “Other license” on the hosting platform, rather than a standard open-source license. The terms are almost identical to those of the previous generation’s model license, with only the name updated.

    In substance, the license grants users relatively broad rights: the model can be used, copied, modified, distributed, fine-tuned, and used in commercial scenarios for free. This is also the basis on which “open weights” is established. However, open weights do not equal open source. Standard open-source licenses usually emphasize non-discrimination, no field-of-use restrictions, and free redistribution, while custom licenses often retain usage boundaries and liability clauses.

    For enterprises, this type of license requires review at at least three levels. First, whether commercial use and redistribution carry additional conditions, such as preserving notices, attribution, or restrictions on specific scenarios. Second, the ownership of responsibility for model outputs. Many open-weight models explicitly state that the accuracy, legality, and safety of generated content are the responsibility of the user. Third, whether modification and secondary distribution trigger new compliance obligations. If an enterprise fine-tunes, packages, or provides external services based on this model internally, legal and security teams should be involved in advance rather than assuming that open weights automatically mean unrestricted commercial use.

    Therefore, a more accurate statement is that this is an open-weight release, not a standard open-source release. It lowers the usage threshold, but does not completely eliminate the compliance threshold.

    5. Local deployment starts at eight GPUs: the hard constraint of 1.56TB

    “Can I run it locally?” is the most practical question after the release. The answer is clear: it is almost impossible on an ordinary personal computer, and enterprise on-premises deployment should start with at least 8 high-memory GPUs.

    The first reason is the size of the weights. The roughly 1.56TB weight files, even with low-bit quantization, mean that VRAM capacity must be sufficient to cover all parameters. Actual inference is not complete just by loading the weights; additional resources must also be reserved:

    • KV cache: the longer the context and the higher the concurrency, the more significant the cache usage;
    • Activations: temporary tensors during forward computation;
    • Communication buffers: used to synchronize intermediate results during multi-GPU parallelism;
    • Inference framework overhead: runtime, scheduler, memory pools, and monitoring components.

    Therefore, 8 GPUs are only a starting line, not a comfort line. If a single GPU’s VRAM level is insufficient, the total capacity of 8 GPUs may only barely meet the requirement; if long context, high concurrency, or multimodal input is enabled, VRAM pressure will quickly increase. A more realistic deployment often requires further expansion beyond 8 GPUs, or techniques such as KV cache quantization, paged caching, expert offloading, and pipeline parallelism to reduce resource usage.

    A simplified VRAM estimation logic can be understood as follows:

    1. Required VRAM ≈ Weights + KV cache + Activations + Communication and framework overhead
    2. KV cache ≈ 2 × layers × KV heads × head dimension × sequence length × concurrent batch size × bytes per element

    From this formula, weights are only a static lower bound; the real service capability is often determined by KV cache and concurrency strategy. For a context of 1 million tokens, without cache optimization, VRAM pressure will be very substantial.

    Parallel strategies also determine deployment efficiency. Common approaches include:

    1. Tensor parallelism: splitting a single layer’s matrices across multiple GPUs to reduce per-GPU computation pressure;
    2. Expert parallelism: distributing different experts across devices to reduce the load on a single GPU;
    3. Pipeline parallelism: partitioning by layer to improve cluster utilization;
    4. Quantization and cache optimization: reducing activation, KV cache, and communication overhead.

    It is worth emphasizing again: while the mixture-of-experts structure reduces the computation required for each token, it does not reduce the scale of weights that must be loaded. It makes “faster computation” more possible, but it does not make “fitting it in” easier.

    6. Who should try it: caution for individuals, cost accounting for enterprises

    For ordinary developers, direct local deployment is not realistic. The 1.56TB weights, the eight-GPU starting point, long-context cache optimization, and multi-GPU communication tuning each imply a relatively high threshold. A more reasonable approach is to validate through cloud-hosted services, inference APIs, or common industry solutions, rather than building a local cluster from the start.

    For enterprises, whether to adopt it should revolve around three questions:

    • Scenario value: is a trillion-parameter-scale model really needed? If the task is only ordinary Q&A, text summarization, or lightweight code completion, a smaller model may be more economical.
    • Total cost of ownership: in addition to GPUs, electricity, data center space, networking, operations, monitoring and alerting, logging services, and failure recovery costs must be calculated.
    • Compliance risk: does the custom license allow the target business model? Do model outputs involve industry regulatory requirements?

    If an enterprise decides to localize the model, it is recommended to start with small-scale validation: test loading success, first-token latency, throughput, long-context stability, and concurrency degradation on a small number of GPUs. Object storage can be used to store weight shards, container platforms can host inference services, logging services can record requests and exceptions, and monitoring and alerting can track VRAM usage, latency, and error rates. This turns “can it run?” into quantifiable engineering metrics.

    7. What this release actually changes

    The significance of this weight release is not that it simply provides a downloadable file, but that it pushes ultra-large open-weight models into a new stage. In the past, the main contradiction of open-weight models was “whether the capability is sufficient.” Now, the main contradiction has become “beyond capability, whether engineering, licensing, and cost are controllable.”

    From a technology trend perspective, at least three points deserve attention:

    1. Trillion-parameter models are entering the scope of open-weight discussion. Competition in model capability is no longer confined to closed-source APIs; it also affects the open ecosystem.
    2. Mixture-of-experts and sparse attention are becoming important directions for long-context models. The former controls computation, while the latter controls context overhead; together they determine whether large models can be practical.
    3. The boundary between open weights and open source will become increasingly important. Users cannot only look at “weights can be downloaded”; they also need to examine license conditions, responsibility allocation, and redistribution rules.

    Overall, this weight release demonstrates the capability ceiling of an ultra-large open model and also exposes its engineering threshold. For technical teams, it is a watershed worth studying: if you have multi-GPU inference, VRAM optimization, and compliance review capabilities, it can become an important resource for capability validation and scenario building; if you only want something that “works as soon as it is downloaded,” then the eight-GPU threshold and the 1.56TB size already show that this is not a lightweight choice.