Trillion-Parameter Open-Weight Models: Architecture, Licensing, and Deployment Costs

    A flagship trillion-parameter model has become publicly downloadable, marking a shift where frontier-scale models are no longer accessible only through APIs. This article explains what such a model actually contains, how its Mixture-of-Experts (MoE) architecture and 4-bit quantization shape the split between compute and memory costs, how to read vendor-reported benchmark numbers, and why an open-weight license is not the same as open source. It then breaks down the real meaning of an "8-GPU minimum," the distributed-inference and memory-bandwidth bottlenecks that govern throughput, and the engineering work required for agent scenarios. It closes with three practical judgments on capability, compliance, and deployment.

    1. The Weight Release: What a Promised Delivery Adds to the Options

    On October 24, a research team launched its next-generation flagship model through an API and web interface, promising to publish the full weights by November 4. The weights arrived as promised on November 4:

    • The hosted repository was created two days early and stayed closed on the morning of release day. It opened that evening, and the weights were downloadable without any application or approval.
    • All weights came in 112 files, of which 96 were sharded tensor files totaling roughly 1.56 TB.

    By volume, this is the largest publicly downloadable open-weight model to date—an order of magnitude larger than any comparable release before it. One concept is worth clarifying up front: open weights means the weights can be downloaded, but that is not the same as a license that meets open-source standards. The difference is covered in detail in Section 5. What this release signals is a new industry option: trillion-parameter models are no longer accessible only through an interface—they can be downloaded into your own server room.

    This article focuses on just three questions that arise once the weights are on disk: what is inside, how the license works, and whether ordinary users can run it.

    2. What the 96 Shards Tell You: Read the Weight Manifest First

    The 96 tensor shards are not a technical necessity. They exist because the hosting platform imposes a per-file size limit, commonly in the range of a dozen-plus gigabytes. Splitting 1.56 TB by that limit yields close to a hundred shards. The remaining dozen-plus files typically include the model configuration, tokenizer, generation parameters, vision-module configuration, the model card, and the license file.

    This sharded format offers two benefits. First, no code needs to be deserialized to execute, which makes the loading path safer. Second, it supports zero-copy reads and memory mapping, so multiple processes can share the same read-only weights.

    Practical advice: read the model card and configuration before deciding whether to commit bandwidth. 1.56 TB means you need several terabytes of available storage, reliable large-file checksum verification, and a plan for resumable downloads and mirror synchronization. Mirror sources are worth preferring where available. Every time the scale doubles, both download and loading costs double as well, so selection should not rest on benchmark scores alone.

    3. How the Skeleton of a Trillion-Parameter Model Is Built

    Based on the official model card, the model’s key structural parameters are as follows:

    Item Parameter
    Total parameters 2.8 trillion
    Activated parameters per token 104 billion
    Expert configuration 896 routed experts + 2 shared experts, 16 selected per token
    Layers 93
    Attention Some layers changed from full attention to sparse attention
    Context length 1 million tokens
    Precision format 4-bit weights, 8-bit activations
    Vision Native multimodal, with a built-in vision encoder
    Release form Quantized only, no full-precision version

    1. MoE: Decoupling Compute from Memory

    The model uses a Mixture-of-Experts (MoE) architecture with two kinds of experts. Shared experts are traversed by every token and carry general knowledge. Routed experts are scored per token by a gating network, and only the 16 highest-scoring ones are invoked.

    1. # Illustrative code: MoE routing and activation ratio estimate
    2. def route(router_logits, top_k=16):
    3. k = min(top_k, router_logits.shape[-1])
    4. vals, ids = topk(router_logits, k=k, dim=-1)
    5. weights = softmax(vals, dim=-1) # Gating weights, sum to 1
    6. return weights, ids # Each token activates k experts
    7. active_ratio = 104e9 / 2.8e12
    8. print(f"Per-token activated parameter share: {active_ratio:.2%}") # ~3.71%

    This yields two conclusions that are often conflated. Compute cost is determined by the activated parameters: roughly 104 billion parameters of computation per token, in the same range as a dense model of about ten billion parameters. Memory cost is determined by the total parameters: even though each token uses under 4% of the parameters, all 2.8 trillion weights must still reside in memory. In effect, MoE decouples compute from memory—computation becomes cheaper, but the deployment barrier concentrates on memory.

    2. Attention: 24 Layers Switched to Sparse Attention

    Compared with the previous generation, the model changes the structure in only one place: 24 full-attention layers are replaced with sparse attention (DSA). This targets the two bottlenecks of long context—the compute and memory overhead of full attention grow quadratically with sequence length, while the KV Cache grows linearly with length. Sparse attention lets each token attend only to the portion of history most relevant to its current content; the difference from a fixed sliding window is that which tokens are retained is decided dynamically by content. The cost is a requirement on the inference framework: it needs a dedicated kernel for this sparse attention, otherwise the optimization is significantly reduced.

    3. Quantization: 4-Bit Weights + 8-Bit Activations

    • File-size comparison: The team provides only the quantized version and has not released BF16 or FP8 builds. Stored in BF16, the same parameters would require about 5.6 TB—roughly 3.6 times the quantized size. 4-bit brings the footprint down to 1.56 TB.
    • Quantization-aware training: The weights use 4-bit and the activations use 8-bit mixed precision, optimized with quantization-aware training (QAT). Compared with “train first, compress later,” this typically preserves more accuracy.
    • Block-level scaling: The weights use a Microscaling-class format in which every 32 elements share one 8-bit scaling factor, giving a per-parameter storage cost of about 4.25 bits. 2.8 trillion parameters therefore corresponds to roughly 1.49 TB of net weights, which closely matches the 1.56 TB total file size.

    Two caveats apply. This format requires hardware with native support for the corresponding low-precision data types; if the hardware can only emulate dequantization, inference speed drops noticeably. And with no full-precision release, fine-tuning at the original precision means dequantizing or requantizing on your own and accepting the precision risk that comes with it.

    4. How Capable Is It, and How to Read Vendor-Reported Data

    Below is the vendor’s comparison table, benchmarked against the current frontier closed-source model. All figures are vendor-reported and were not independently tested for this article:

    Capability (public leaderboard) This model Compared closed-source flagship
    Terminal/CLI coding tasks 65.7% 66.4%
    Real-world software engineering long-horizon tasks 51.7% 46.3%
    Combined coding and science tasks 38.1% 63.3%
    Hard multidisciplinary reasoning 39.4% 60.9%

    Three judgments can be drawn from this.

    1. The strength is in long-horizon software engineering. At 51.7% versus 46.3%, it leads clearly, and on terminal/CLI coding tasks at 65.7% versus 66.4% it is essentially on par with the closed-source flagship.
    2. The weakness is in combined coding-and-science and deep reasoning. On combined coding and science tasks the model reaches only 38.1% against 63.3%, about 25 points behind; on hard multidisciplinary reasoning it scores 39.4% against 60.9%, again more than 20 points behind.
    3. “Strongest” comes with conditions. The vendor claims the model exceeds the comparison on 16 of 37 benchmarks (again vendor-reported, not independently tested here) — fewer than half — so the reasonable framing is “one of the strongest in the open-weight camp,” not “the strongest overall.” A third-party evaluator has also assigned a composite intelligence score of 55, placing it fourth globally (likewise not independently verified, offered only as supporting evidence), which is consistent with the “close to, but not surpassing, the frontier” read.

    Be wary of four common evaluation biases: differences in reporting and comparison methodology (test scripts, prompts, and sampling counts may all differ); sampling count (pass@1 and pass@k are not directly comparable); context settings (hard benchmarks are sensitive to input length and retrieval strategy); and data contamination (code benchmarks are easily covered by training corpora, which can inflate scores). The practical approach is to build an internal evaluation set from 20–50 real tasks from your own organization, run a small-scale pilot on your business data, and rely on the measured results.

    5. Licensing: Open Weights Are Not Open Source

    This is the point most worth discussing on its own. The repository ships a custom license, labeled “Other license” on the platform, not a standard open-source license such as MIT or Apache 2.0. Compared with the previous generation, the terms are word-for-word identical, with only the name changed.

    The core grant is indeed close to an MIT-style permissive license: free use, copying, modification, distribution, fine-tuning, and even commercial use. But the differences from a standard open-source license are that it typically also includes attribution requirements, a separate authorization threshold for very-large-scale commercial use, and additional terms such as an acceptable use policy and liability limitations; and it has not been recognized by standard open-source license certification bodies, so enterprise compliance cannot simply “approve it as if it were MIT.”

    Because the new and old terms are identical and only the name changed, the team is clearly reusing the same licensing framework: the restrictions on use have not changed, and compliance conclusions can be reused, but you still need to compare the terms against your own scenario line by line—especially attribution obligations, user-scale thresholds, and usage-scope restrictions. Recommendations for engineering and legal teams:

    1. Compare line by line: Map the custom license against MIT and Apache 2.0 and produce a list of differences, especially commercial thresholds and redistribution obligations.
    2. Preserve attribution: When redistributing, keep the attribution information in the license file and model card, and add it to your product documentation or an about page:
    1. This product includes a model developed by <the research team>,
    2. released under its custom open-weight license. See MODEL_LICENSE for details.
    1. Monitor user scale: If your service’s user volume may approach the commercial authorization threshold, start the authorization conversation early.
    2. Re-review before fine-tuning: Derivative weights such as LoRA are bound by the same constraints, so re-check the license before any secondary release.

    In one sentence: it can be used, modified, and commercialized, but it is not “open source” in the standard sense.

    6. Deployment Feasibility: What Does an “8-GPU Minimum” Mean?

    1. The Memory Ledger

    Memory usage comes mainly in three parts: weights (4-bit weights plus block scaling factors, about 1.49 TB, the fixed bulk), the KV Cache (growing linearly with context length and concurrency, reaching hundreds of GB or more under long context), and framework and runtime overhead (per-card device context and communication buffers each take several GB, plus fragmentation and reserved margin). Added together, the total is at least 1.5 TB or more—which means a single machine needs at least 8 accelerator cards with no less than 200 GB of memory each. This is where “8-GPU minimum” comes from.

    A rough estimate across common configurations (order-of-magnitude only):

    Configuration Total memory Can it be deployed locally?
    2× 192 GB-class accelerators ~384 GB No
    Single machine, 8× 192 GB-class accelerators ~1.54 TB Barely (almost no room left for the KV Cache)
    Single machine, 8× 200 GB-class accelerators ~1.6 TB Feasible (requires compressing context or limiting concurrency)
    Consumer GPUs (24 GB × N) Needs 65+ Not feasible on a single machine

    The conclusion is direct: this is a model aimed at datacenter-grade hardware, and neither a desktop workstation nor a personal computer can host it.

    2. Three Practical Alternatives

    Path Suitable scenario Trade-off to weigh
    Call a cloud model service Individuals and small-to-medium teams; fast integration Ongoing cost, data compliance, network dependency
    Hybrid: a small model as fallback, the large model for hard cases Teams with some engineering capability Routing logic and call-chain complexity
    Choose a smaller open-weight model for local deployment Data-sensitive cases needing on-premises hosting Capability ceiling and fine-tuning investment

    When choosing the cloud route, you can evaluate the GPU compute and model-hosting services offered by cloud providers such as Baidu AI Cloud; for data-sensitive scenarios, on-premises deployment is preferable. It is worth stressing that the value of a trillion-scale MoE is often greater in agent workflows than in plain conversation: it can hand multi-step tasks to the model to complete automatically, thereby reducing the overall call volume.

    7. How to Actually Run a Trillion-Parameter Model Smoothly

    1. Distributed Inference: 8 GPUs Is Not Simple Stacking

    A single machine with 8 GPUs typically splits the weights via tensor parallelism or expert parallelism. Tensor parallelism slices each layer’s computation across cards by column or row, requiring collective communication at every layer, and is well suited to high-speed intra-machine interconnect. Expert parallelism distributes the 896 experts across different cards for on-demand routing; with few cards, each still holds over a hundred experts, and it adds token-distribution (all-to-all) overhead. The two are often combined, and an 8-GPU topology takes high-speed full interconnect as a basic prerequisite. When scaling across machines, the network becomes the bottleneck, so long-context, high-concurrency scenarios are better served by a fully loaded single machine.

    2. The Throughput Bottleneck: Memory Bandwidth, Not Compute

    MoE reads only the activated weights per token, which aligns neatly with the memory-bandwidth bottleneck of inference. A rough calculation: each token reads about 104 billion × 0.53 bytes ≈ 55 GB of weights; if the machine’s aggregate bandwidth is on the order of several TB/s (the exact figure depends on the intra-machine interconnect scheme, such as NVLink or a private full-interconnect fabric), a single request’s theoretical ceiling is around tens of tokens per second. Increasing the batch size or context further will not break the bandwidth constraint and will instead quickly exhaust the KV Cache. So the realistic levers for raising throughput are quantization, sparse attention, and sensible concurrency control—not blindly adding more cards.

    1. # Illustrative code: KV cache estimation (ignores implementation differences such as low-rank compression)
    2. def kv_cache_bytes(layers, kv_heads, head_dim, tokens, bytes_per_elem=1):
    3. return 2 * layers * kv_heads * head_dim * tokens * bytes_per_elem # Two copies: K and V
    4. est = kv_cache_bytes(layers=93, kv_heads=8, head_dim=128,
    5. tokens=1024 * 32768, bytes_per_elem=1)
    6. print(f"KV cache ~ {est / 1024**4:.2f} TiB")

    These figures are only an order-of-magnitude illustration based on common parameters; actual usage depends on implementation details and should follow the official configuration. What the numbers show is this: if the KV Cache still had to be stored token by token, the small amount of remaining memory across 8 GPUs could never hold it—whether the long-context capability actually materializes depends on the vendor’s cache-compression and sparsification implementation.

    3. Agent Scenarios: Four Gates After Reasoning

    Native multimodality, million-token context, and agent-oriented training mean the model is naturally suited to “long document + tool calling” tasks. But taking it into production still requires clearing four engineering gates: tool-calling reliability (structured output needs schema validation and retry fallbacks to prevent parameter drift from causing business errors), sandbox isolation (high-risk actions such as code execution and file operations must run in an isolated environment), least privilege (explicitly whitelist accessible resources and set rate limits), and observability (build monitoring and alerting for token usage, first-token latency, and tool-call success rate, and feed usage data into logs for unified analysis).

    8. Conclusion: Three Judgments

    1. Capability: Trillion-parameter MoE open-weight models mark a new stage in which “frontier capabilities open up.” Coding and agent capabilities—especially long-horizon software engineering—now approach the top closed-source models, though there remains a clear gap in combined coding-and-science tasks, scientific reasoning, and long-chain hard problems.
    2. Compliance: Open weights are not open source. The custom license is mainly a permissive grant, but it carries commercial thresholds and attribution obligations, so enterprises should complete a legal review before adopting it.
    3. Deployment: The essence of the 8-GPU minimum is a “memory wall,” not a lack of compute. For the vast majority of teams, the more practical path is to validate value with cloud capabilities first, then privatize gradually as the business scales.