Massive MoE Open-Weight Model Deep Dive: 2.8T Parameters, Custom License, and 8-GPU Deployment

    This article breaks down a newly released massive MoE open-weight model, covering its 2.8T total parameters, 104B active parameters, custom license terms, memory and hardware requirements, deployment options, benchmark interpretation, and evaluation recommendations.

    1. Release Background: Trillion-Parameter Models Enter the Open-Weight Era

    On November 4, a leading model vendor released the full weights of its next-generation flagship model on a major open-weight hosting platform, following an earlier commitment. The repository was created on November 2, remained closed during the day on November 4, and was opened that evening. Downloads did not require an application. It contains 112 files and 96 weight shards, totaling about 1.56 TB.

    This is the largest set of publicly released weights to date, an order of magnitude larger than previous open-weight models. However, being able to download the weights does not mean they are easy to use: the license is not a standard open-source license, and running the model locally requires at least a server with eight high-memory GPUs. For developers and enterprise technical decision-makers, the key questions are:

    • What is inside the model: How are trillion-scale parameters organized, and how much compute is used per inference?
    • How does the license work: Where is the boundary between “open weights” and “open source,” and are commercial use restrictions involved?
    • Can it be deployed, and is it worth deploying: What hardware is required, and how should self-hosting be compared with cloud services?

    2. Model Breakdown: 2.8T Total Parameters, Only 104B Activated per Token

    1. Overall Structure

    The model’s parameter scale and activation scale are reflected in its naming. Public materials indicate that it is a native multimodal Agent model continued from the previous generation. Core parameters:

    Dimension Parameters
    Total parameters / active parameters per token 2.8T / 104B
    Architecture Mixture of Experts (MoE)
    Expert configuration 896 routed experts + 2 shared experts; 16 routed experts selected per token
    Layers 93
    Context length 1 million tokens
    Multimodal capability Built-in vision encoder, supports image input

    Compared with the previous generation, only one architectural change was made: 24 full-attention layers were replaced with sparse attention specifically to reduce compute overhead for long-context scenarios. The weights use a 4-bit floating-point format and activations use an 8-bit floating-point format, produced directly by quantization-aware training (QAT). The vendor provides only this version; no higher-precision weights are available.

    2. Sparse Activation: Lower Compute, but Not Lower Memory

    The core idea of MoE is “large total parameter capacity, but only a small subset of parameters is invoked for each token.”

    • Activation ratio: Each token activates 104B parameters, about 3.7% of the total. In terms of per-token computation, it behaves closer to a hundred-billion-parameter dense model than to a 3-trillion-parameter model.
    • Expert division of labor: The 896 routed experts handle specialized knowledge, while the 2 shared experts are available to all tokens and handle general language capability. Routed experts are organized in groups of 128, and 16 are selected per layer for each token.
    • Persistent memory: Regardless of which experts a token is routed to, inference must be able to access all weights at any time. Therefore, memory demand is determined by the 2.8T total parameters, not the 104B active parameters. This is why activation is compute-efficient but memory is expensive.

    3. Sparse Attention: Saves Compute, Not Cache

    For million-token contexts, standard full attention causes both compute cost and KV cache size to grow quickly with sequence length. The model uses sparse attention in some layers so that each token attends only to filtered historical content, reducing attention compute cost for long sequences. Two points matter:

    • Sparse attention primarily reduces compute. The KV cache still needs to retain all historical tokens unless techniques such as paged management, quantized compression, or prefix sharing across requests are used.
    • Sparse attention changes the compute path. Its implementation may require adaptation to commonly used inference frameworks, and long-context correctness should be validated before deployment.

    4. Quantization Strategy: 4-Bit Weights Enable Deployability

    Rough estimates of weight size for 2.8T parameters at common precisions:

    • FP16/BF16: about 5.6 TB
    • FP8: about 2.8 TB
    • 4-bit: about 1.4 TB

    The released version is the 4-bit version, which is a prerequisite for deploying a trillion-parameter model at the “single machine with eight GPUs” scale. However, quantization-aware training cannot eliminate precision loss. For numerically sensitive tasks such as scientific computing and multi-step reasoning, validation with business data is still required. In long-context scenarios, KV-cache quantization often affects memory usage more than weight quantization and should be evaluated separately.

    3. License Interpretation: Open Weights Are Not Open Source

    The most notable aspect of this release is its license. The repository specifies the license type as other, meaning a custom license rather than a standard open-source license such as MIT or Apache.

    From the structure of the terms, the license is basically consistent with the license text of similar previous models. The main body is a permissive grant: free use, copying, modification, distribution, fine-tuning, and commercial use. However, two constraints are added for large-scale commercial scenarios:

    • Entities that provide the model as a service (MaaS) externally and whose related business has cumulative revenue exceeding $20 million over 12 months must enter into a separate agreement with the copyright holder before commercial use.
    • Commercial products whose monthly active users exceed a certain threshold also require separate authorization. The source report was truncated at this point; the exact threshold should be confirmed in the full license text.

    This leads to three conclusions:

    1. Most scenarios are unaffected: Internal enterprise use, product R&D, and small-to-medium commercial applications can generally be used directly under the license terms.
    2. The trigger is “scale,” not “use case”: The conditions are tied to revenue and user volume, not to a ban on a particular use case. Once a business grows to a certain scale, commercial planning should begin early.
    3. Do not define it as “open source”: Standard open-source licenses do not impose revenue or user-volume conditions. In compliance documentation and procurement materials, it should be accurately described as “open weights under a custom license.”

    Enterprises should establish a simple license self-check process, clarify the deployment mode and scale before launch, and review periodically:

    1. # Commercial use self-check for a custom open-weight license (illustrative; thresholds must follow the full license)
    2. LICENSE = {
    3. "maas_revenue_threshold_usd": 20_000_000, # Cumulative revenue over 12 months (USD)
    4. "commercial_mau_threshold": None, # Monthly active user threshold; confirm in the full license
    5. }
    6. def check_commercial(usage, trailing_12m_revenue_usd, commercial_mau):
    7. actions = []
    8. # Condition 1: external MaaS with revenue above the threshold
    9. if usage == "MaaS" and trailing_12m_revenue_usd > LICENSE["maas_revenue_threshold_usd"]:
    10. actions.append("sign a separate agreement with the copyright holder before commercial use")
    11. # Condition 2: commercial product MAU above the threshold (if threshold is unknown, require manual review)
    12. threshold = LICENSE["commercial_mau_threshold"]
    13. if usage == "PRODUCT" and (threshold is None or commercial_mau > threshold):
    14. actions.append("check the MAU clause in the full license")
    15. # Internal use and other standard scenarios do not require additional action
    16. return actions or ["use is permitted under the license terms"]

    4. Local Deployment: How the Eight-GPU Requirement Is Derived

    1. Memory Estimation Model

    The total memory requirement for a machine can be broken down into four parts:

    Total memory requirement = model weights + KV cache + runtime overhead + memory headroom

    • Model weights: Proportional to total parameter count and usually the largest portion.
    • KV cache: Proportional to the number of layers, KV heads, cache size per token, and context length.
    • Runtime overhead: CUDA contexts, communication buffers, temporary tensors, and similar items.
    • Memory headroom: Generally recommend keeping more than 10% to avoid out-of-memory errors.

    The following function can be used for a quick estimate; the estimated values can then be fed back into the function to check whether the configuration is self-consistent:

    1. import math
    2. def estimate_gib(total_params=2.8e12, weight_bits=4, overhead_ratio=0.01,
    3. layers=93, kv_bytes_per_token_per_layer=1024,
    4. ctx_len=131072, runtime_gib=32, per_gpu_gib=192):
    5. """
    6. Estimate memory requirements for local deployment of a trillion-scale MoE model (GiB).
    7. Parameter notes (illustrative assumptions; actual values depend on model configuration):
    8. overhead_ratio: additional overhead ratio for quantization scaling factors, embeddings, vision encoder, etc.
    9. kv_bytes_per_token_per_layer: KV cache bytes per token per layer (before sparse/compression optimizations)
    10. ctx_len: planned context length
    11. runtime_gib: runtime overhead for framework and communication buffers
    12. per_gpu_gib: usable memory per GPU
    13. """
    14. # 1. Weight memory: parameters × bits / 8
    15. weight = total_params * weight_bits / 8 / 1024**3
    16. # 2. KV cache: layers × bytes per token per layer × context length
    17. kv_cache = layers * kv_bytes_per_token_per_layer * ctx_len / 1024**3
    18. # 3. Total demand: weights (including additional parameters) + KV cache + runtime overhead + 10% headroom
    19. total = weight * (1 + overhead_ratio) + kv_cache + runtime_gib
    20. total *= 1.1
    21. # 4. Minimum GPU count: round up
    22. min_gpus = math.ceil(total / per_gpu_gib)
    23. return weight, kv_cache, total, min_gpus
    24. weight, kv, total, gpus = estimate_gib()
    25. print(f"weights {weight:.0f}GiB, KV cache {kv:.0f}GiB, "
    26. f"total {total:.0f}GiB, minimum {gpus} GPUs")
    27. # Example: 2.8T × 4-bit, 128K context, 192 GB per GPU
    28. # weights 1304GiB, KV cache 12GiB, total 1497GiB, minimum 8 GPUs

    2. Eight GPUs Are Only the Starting Point

    For 2.8T parameters and 4-bit weights:

    • The weights themselves are about 1.4 TB. The 1.56 TB repository size includes additional parameters such as the vision encoder, token embeddings, and quantization scaling factors.
    • Converted to binary units, the machine needs at least about 1.5 TiB of aggregate GPU memory to hold the weights.
    • An 8-GPU machine with 192 GB per GPU provides about 1.5 TiB total, almost just enough for the weights.

    The example above gives a total demand of around 1497 GiB at a 128K context, only about 3% below the 8-GPU ceiling. This means:

    • A single machine with 8 × 192 GB GPUs is the entry threshold, not a comfortable configuration. The KV cache for a 128K context is about 12 GiB, barely enough for a single request. Longer inputs or multiple concurrent requests can easily cause out-of-memory errors.
    • Once a 1.1 safety factor is considered, practical demand can be about 1.06 times the 8-GPU limit. Production environments need to reserve room for expansion.
    • P99 latency is hard to guarantee. When weights nearly fill GPU memory, very little cache remains available, making batch inference difficult.

    If per-GPU memory drops to 141 GB, eight GPUs provide only about 1.03 TiB, which is not enough even for the weights. At least 11 GPUs would be required. In that case, the options are to choose accelerators with more memory or use CPU offloading, but the latter can significantly reduce throughput.

    For million-token contexts, an 8-GPU machine cannot support full-length inference for a single request. Techniques such as context parallelism and chunked prefill are needed.

    3. Deployment Options: From Single Machine to Cluster

    Overall, deployment paths for this type of model can be summarized in three levels:

    • Capability validation: Call the model through an API or a cloud-hosted service to validate task effectiveness without owning hardware.
    • Small-scale private deployment: A single machine with 8 high-memory GPUs, suitable for fixed concurrency and scenarios where data cannot leave the premises.
    • Large-scale serving: A multi-machine, multi-GPU cluster with request scheduling, elastic scaling, and end-to-end monitoring.

    Large enterprises with sensitive data, deep customization needs, or long-term high-frequency usage may find self-hosted clusters more controllable. Small and medium-sized teams, or businesses with fluctuating traffic, may find cloud compute or cloud-hosted inference services more economical. For concrete resource planning, use the estimation model in Section 1 as a starting point and calibrate it with load tests based on actual concurrency.

    5. Benchmark Interpretation: How to Read Vendor-Reported Results

    At release, the vendor reported a set of self-selected results:

    • On a terminal-task benchmark, it scored 65.7%, close to a leading proprietary model’s 66.4%.
    • On a long-horizon software engineering benchmark, it scored 51.7%, clearly higher than the comparison model’s 46.3%.
    • In another comparison against a proprietary model, it scored 58.2%.
    • Weaknesses were also clear: it scored only 38.1% on scientific terminal tasks, compared with 63.3% for the comparison model, and 39.4% on a multidisciplinary reasoning benchmark, compared with 60.9%.
    • The vendor said that it outperformed the comparison model on 16 of 37 benchmarks, and a third-party evaluation platform gave it an intelligence index of about 55, ranking fourth globally.

    These results suggest that its strengths are concentrated in programming, terminal operation, and Agent-style tasks, while there remain clear gaps in scientific computing and some reasoning tasks. Three cautions apply when reading this kind of data:

    1. Vendor-reported results may be selectively presented: The chosen comparison models and benchmark combinations naturally favor strengths.
    2. The same model can score differently under different evaluation harnesses: Agent tasks are highly sensitive to prompts, tool-calling methods, and execution environments.
    3. Benchmark contamination cannot be fully ruled out: A model may have seen public test items, so leaderboard position should not be equated directly with business performance.

    A more reliable approach is to build an evaluation set based on your own business:

    1. # Minimal skeleton of a business evaluation set (illustrative)
    2. CASES = [
    3. {"id": "terminal-001", "input": "...", "check": "unit_test"},
    4. {"id": "science-001", "input": "...", "check": "exact_match"},
    5. {"id": "agent-001", "input": "...", "check": "human_review"},
    6. ]
    7. def evaluate(model_client, cases, max_ctx=131072):
    8. results = []
    9. for case in cases:
    10. if len(case["input"]) > max_ctx:
    11. results.append({"id": case["id"], "status": "skipped", "reason": "too long"})
    12. continue
    13. output = model_client(case["input"])
    14. results.append({"id": case["id"], "status": "done", "output": output})
    15. return results # Calculate pass rate, average latency, and cost per request by task type

    If the business is mainly code generation, automated operations, or tool calling, this type of model is worth testing in depth. If the business is mainly scientific computing or rigorous reasoning, comparison models should be included in the same evaluation set. Long-context and multimodal capabilities should also be tested with real-length documents and charts; parameter tables alone are not enough.

    6. Conclusions and Action Recommendations

    This release marks the entry of open-weight models into the trillion-parameter stage. The MoE sparse architecture makes “large parameters” more affordable in terms of compute, but the memory threshold remains high. Based on the full analysis, here are five recommendations:

    1. Distinguish “open weights” from “open source”: Treat it as a custom license in compliance reviews, and pay attention to revenue and monthly-active-user thresholds.
    2. Estimate memory before evaluating quality: Use total parameters, not active parameters, to estimate hardware needs; 8 × 192 GB is only the minimum.
    3. Build your own evaluation set: Use real business tasks instead of leaderboard rankings for model selection.
    4. Validate long-context and multimodal capabilities: Stress-test sparse attention effectiveness and stability on real data.
    5. Choose deployment based on scale: Start with API validation, then move to private deployment or clustered deployment as needed.