Local Deployment and Licensing Guide for Trillion-Parameter MoE Open-Weight Models

    A trillion-parameter mixture-of-experts (MoE) model has arrived as downloadable open weights, but downloading it is the easy part. Using a 2.8-trillion-parameter sparse MoE model as a worked example, this article explains what actually lives inside the weight repository, the real hardware footprint of self-hosted inference, and the commercial red lines in the open-weight license. It covers the split between total and activated parameters, memory budgeting with a reusable GPU-memory function, the role of KV Cache in capping concurrency, the implications of shipping a single 4-bit micro-scaling quantization, a parallel-slicing strategy for a 64-GPU cluster, how to read vendor-reported benchmarks, and a compliance checklist for the custom license. Readers come away with an evaluation method they can apply directly to their own workloads.

    Introduction: The Real Work Starts After the Weights Are Public

    On November 4, 2025, an AI company released the complete weights of a trillion-parameter-scale model on a public model hosting platform, as promised. According to the model card, it is a natively multimodal agent model, further trained from the previous generation. It has roughly 2.8 trillion (2.8T) total parameters and activates only about 104 billion (104B) per token, using a mixture-of-experts (MoE) architecture. The weights come as 112 files containing 96 tensor shards, totaling about 1.56 TB — making it the largest open-weight model publicly available to date.

    The common claim is that inference can run on 8 GPUs each with about 141 GB of memory. But for most engineering teams, there is a long way between “it can load” and “it can serve reliably in production” — across four gates: memory budget, quantization precision, parallel communication, and license compliance. This article does not compare one model against another. Instead, using a trillion-parameter sparse MoE model as an example, it explains three things:

    • What is actually inside the weights once they land in your repository;
    • What resources self-hosted deployment really requires;
    • Where the commercial red lines lie in an open-weight license.

    By the end, you will have an evaluation method you can apply directly.

    1. Model Architecture: How 2.8 Trillion Parameters Are Composed

    Total vs. Activated Parameters: MoE Separates Capacity from Compute

    The model has 896 routed experts. For each token, a router selects 16 of them, plus 2 shared experts that always participate. Total parameters reach 2.8T, but each token actually uses only 104B — an activation ratio of about 3.7%. Parameters are stored at 4-bit, and during computation the weights are multiplied against 8-bit activations. So the rough per-token forward compute is:

    1. FLOPs ≈ 2 × 104×10^9 ≈ 2.08×10^11 (about 208 GFLOPs/token)

    This puts the model in the same range as a “hundred-billion-scale dense model.” In other words, MoE trades the compute cost of a near-hundred-billion dense model for the knowledge capacity carried by trillions of parameters. That is the fundamental reason MoE has been adopted at scale.

    But the advantage comes at a cost. Memory is calculated from total parameters, regardless of how much is activated. MoE shifts pressure from compute to memory capacity and memory bandwidth, and it introduces three new engineering challenges:

    • Routing imbalance. Hot experts get hit by a large concentration of tokens, causing load skew that slows down the entire batch.
    • Communication overhead. Experts are spread across different GPUs, so tokens must be sent to the GPU hosting the relevant expert and gathered back afterward. This relies on collective communication (All-to-All), and its cost grows with the number of nodes.
    • Scheduling uncertainty. Which experts activate depends on the input, so the actual compute cost for different requests is not identical. Capacity planning needs to leave headroom.

    Layers and Attention: Designing Down for Long Context

    The model has 93 layers. Of these, 24 switch from full attention to sparse attention (DSA) to reduce attention overhead under long context. Standard attention compute grows quadratically with sequence length: multiply the sequence by 10 and the attention portion grows roughly 100x. Sparse attention lets each token attend only to filtered historical information, bringing long-sequence processing closer to linear growth — one of the technical premises that lets the model advertise a 1-million-token context window.

    For users, two points matter:

    • The advertised context length is a capability ceiling, not a quality guarantee at that length. Effective context length (whether the model can reliably use key information in long inputs) must be validated against your own tasks.
    • Sparse attention typically depends on specific kernel implementations. If the deployment environment lacks support, it may fall back to a generic implementation, and the long-text performance advantage will shrink noticeably.

    Vision Encoder: Extra Memory from Multimodality

    The model has a built-in vision encoder and can accept image input directly. The vision component adds little in parameters but brings new memory overhead: high-resolution images are split into many visual tokens that enter attention computation alongside text tokens, so both sequence length and the KV Cache expand together. In multimodal workloads, text length is not the only variable when evaluating throughput.

    2. Memory Budget: 8 GPUs Are Only the Threshold, Not the Finish Line

    First, Verify Where the 1.56 TB Comes From

    In theory, 2.8T parameters at 4-bit storage give a main-weight size of:

    1. 2.8×10^12 × 4 bit ÷ 8 = 1.4×10^12 B = 1.4 TB

    On top of that, the embedding layers, the output layer (lm head), the vision encoder, and the config and tokenizer files bundled with each shard also take space. The actual weight size lands around 1.5 TB, matching the total repository size. This rough check tells you the repository holds low-bit quantized weights, not a high-precision original.

    It also helps to separate three easily confused units: weight size is usually given in TB/GB, memory capacity is counted in GiB, and network bandwidth is measured in Gb/s — and 1 Byte equals 8 bits. 1.56 TB (decimal) is about 1.42 TiB (binary). If a download script’s progress bar sits at 100% but verification fails, the cause is often right here. The safe approach is to verify each shard’s hash individually in the download script:

    1. # Read the weight manifest and verify each shard's integrity (pass the URL via a variable, don't hardcode it)
    2. while read -r name sha; do
    3. echo "${sha} ${name}" | sha256sum -c -
    4. done < manifest.txt

    KV Cache: The Hidden Variable That Caps Concurrency

    Weights are only static usage. What really determines “how many users you can serve at once” is the KV Cache, which grows linearly with “concurrency × context length”:

    1. KV_Cache = 2 × L × H_kv × D_head × S × B × bytes_per_element
    • L: number of layers;
    • H_kv: number of KV heads;
    • D_head: dimension per head;
    • S: sequence length;
    • B: batch size (concurrency);
    • the factor of 2: one copy each for K and V.

    For example, with 8 KV heads per layer, head dimension 128, 16-bit storage, and a 10% reserve for framework overhead and memory fragmentation, you can write a memory-budget function like this:

    1. def gpu_memory_budget(
    2. weight_tb: float, # actual weight usage in TB
    3. layers: int, # number of layers
    4. kv_heads: int, # KV heads per layer
    5. head_dim: int, # dimension per head
    6. seq_len: int, # sequence length
    7. batch_size: int, # concurrent batch size
    8. bytes_per_element: int = 2, # bytes per element (2 for 16-bit)
    9. reserve: float = 0.1, # framework reserve and fragmentation
    10. ) -> dict:
    11. """Estimate per-instance memory needs; kv_heads / head_dim per the model card"""
    12. kv_gb = (
    13. 2 * layers * kv_heads * head_dim * seq_len * batch_size
    14. * bytes_per_element / 1e9
    15. )
    16. weight_gb = weight_tb * 1000
    17. peak_gb = (weight_gb + kv_gb) / (1 - reserve)
    18. return {"weight_gb": weight_gb, "kv_cache_gb": kv_gb, "peak_gb": peak_gb}
    19. print(gpu_memory_budget(weight_tb=1.5, layers=93, kv_heads=8,
    20. head_dim=128, seq_len=32768, batch_size=16))
    21. # {'weight_gb': 1500.0, 'kv_cache_gb': 99.6, 'peak_gb': 1777.3}

    These parameters are illustrative; use the actual values from the model card. The result shows that just 16 concurrent requests with 32K context each already push total memory needs close to 1.8 TB.

    8 GPUs at 141 GB each add up to about 1128 GB — only enough for roughly 75% of the weights, still at least 4 same-spec GPUs short of runnable. So “start with 8 GPUs” actually corresponds to a configuration like 12× 141 GB GPUs, 16× 96 GB GPUs, or 8× 192 GB GPUs. Roughly estimating from the 1.5 TB weight size:

    Per-GPU Memory GPUs to Load Weights Only GPUs for 16-way × 32K Concurrency
    96GB 16 ≥19
    141GB 11 (about 12) ≥13
    192GB 8 ≥10

    Note that the table covers only weights and KV Cache. Inference-engine reservations, vision-encoding activations, communication buffers, and the framework’s own device memory all consume memory too, so bump the configuration up one more tier in practice.

    From a utilization standpoint, tensor parallelism splits each layer’s weights evenly across GPUs, so the weight share drops as the GPU count rises — no different in essence from a dense model. In expert parallelism, each EP group holds all 896 experts, so per-group memory depends only on the number of experts. If you increase expert parallelism to make room for the KV Cache, per-group memory does not drop, and the model can’t be sliced across more devices. So relying on expert parallelism to spread the memory burden is not realistic; expanding memory capacity depends mainly on tensor parallelism or pipeline parallelism.

    Decode Is Memory-Bandwidth Bound; Sparse Activation Isn’t Cheap

    During generation (decode), every token produced requires reading all activated expert weights fully from memory. Even estimating at just 104B activated parameters and 1 byte per parameter, the theoretical minimum weight read per token exceeds 100 GB. This means:

    • Throughput is bounded by memory bandwidth. Estimate the theoretical ceiling with “per-GPU memory bandwidth ÷ per-token weight read,” then multiply by the GPU count for a per-replica throughput reference.
    • 4-bit quantization mainly helps capacity and bandwidth. The model shrinks to about 1/4 the size, so the same memory holds it and reads cost less bandwidth; but compute demand is set by activated parameters and doesn’t drop.
    • A higher TP degree isn’t necessarily faster. Under expert parallelism, tokens must be routed across GPUs; if that all travels over inter-node network, All-to-All communication can become the bottleneck, so tensor-parallel degree isn’t a “the higher the better” dial.

    3. Quantization and Precision: What It Means That the Vendor Shipped Only One Version

    4-bit Weights + 8-bit Activations: The “Micro-Scaling” Format

    Weights use a 4-bit micro-scaling floating-point format. The typical approach groups every 32 parameters into a micro-block that shares one 8-bit exponent scaling factor; activations use 8-bit micro-scaling floating point. Compared to traditional per-tensor quantization that takes a max value across a tensor, the micro-scaling format applies finer-grained scaling, fitting parameter blocks with differing numerical distributions separately, and generally controls precision loss better.

    This format imposes constraints on the deployment side:

    • It depends on hardware and inference-framework kernel support. If the framework or hardware lacks the corresponding compute kernel, weights must first be dequantized to a high-precision format before computation. The weights are compressed in memory but still expanded during compute, so the compute cost doesn’t go down.
    • The vendor released only this one weight version. No BF16/FP8 version is provided, so don’t expect a different weight file to “improve precision.” For higher precision you can only convert or continue training on the existing 4-bit weights; and a model trained with quantization-aware training can’t recover its pre-training precision by converting back.
    • Quantization-aware training (QAT) cuts both ways. Because the model learns “with” the low-bit error during training, it’s typically more stable than post-training quantization (PTQ) at the same bit width. But the precision figures the vendor reports reflect this specific quantized version and shouldn’t be compared head-to-head with versions at other precisions or from other sources.

    Precision Validation Must Be Task-Specific

    4-bit quantization affects different tasks unevenly. The recommendation is to build a validation set for your business scenario, run the same batch of samples through both the official cloud service and the local deployment, and compare key metrics. The common experience is that summarization, rewriting, structured extraction, and multi-turn dialogue are relatively insensitive to low-bit quantization, while numerical computation, complex reasoning, and long-chain agent tasks are more sensitive. To be clear, this is general industry experience; your own testing should still be the deciding factor.

    4. Parallel Strategy: A Splitting Approach You Can Apply Directly

    To get a model running, there are usually two routes: fully self-hosted, or a hybrid of “private data + hosted inference.” The former requires you to plan the parallel strategy yourself. A typical 64-GPU cluster can use an “8 nodes × 8 GPUs” layout:

    • Set tensor-parallel degree to 8, aligning group boundaries with node boundaries. Tensor parallelism does an All-Reduce every layer — the most frequent communication — so placing it on the node-internal high-bandwidth interconnect is the most cost-effective.
    • Set the number of expert-parallel groups to 8. The 896 experts split evenly into 8 groups of 112, with the 8 GPUs in a tensor-parallel group collaborating within each group to complete the intra-expert matrix computation.
    • One replica fills exactly one 8-node cluster. Production can run multiple replicas, with a front gateway distributing requests by load.
    • Prioritize controlling context length rather than blindly scaling replicas. Doubling replicas roughly doubles memory needs. Rather than adding replicas, first use KV-Cache quantization, paging, and request scheduling to shrink per-request usage.

    By comparison, single-node deployment typically can only combine “tensor parallelism + pipeline parallelism,” using pipeline parallelism to add memory capacity. The cost is pipeline bubbles: when stages are unevenly loaded, devices wait on each other and overall throughput drops noticeably.

    1. # Illustrative parallel split for 8 nodes x 8 GPUs (parameter names are for explanation only, not a specific framework's config keys)
    2. world_size: 64 # total GPUs = nodes x GPUs per node
    3. tensor_parallel: 8 # tensor parallelism: split within a node, All-Reduce each layer
    4. expert_parallel: 8 # expert parallelism: 896 experts split into 8 groups of 112
    5. pipeline_parallel: 1 # prefer expert parallelism across nodes to avoid pipeline bubbles
    6. max_model_len: 32768 # set to the context length the business actually needs; don't max it out blindly

    Note that parameter names differ across inference frameworks — defer to your framework’s documentation. Once the split is set, also take care of the following:

    • Network. Use a low-latency network such as RDMA between nodes.
    • Inside a node. Ensure each node’s 8 GPUs sit within the same high-bandwidth interconnect domain.
    • Kernel support for quantization and sparse attention. Before launch, confirm the framework’s support for micro-scaling 4-bit weights and sparse attention, and run a connectivity test on a small shard set if needed.

    5. Reading the Benchmarks: How to Interpret Vendor-Reported Scores

    The model’s official results show a clear “uneven” profile. The vendor’s comparison table puts it at 65.7% on the coding benchmark Terminal-Bench 4.0 — slightly below a competitor’s 66.4% but above another competitor’s 58.2%. On the agentic long-task benchmark SWE Marathon it scores 51.7%, above a competitor’s 46.3%. On the other hand, on Terminal-Bench-Science it reaches only 38.1% against a competitor’s 63.3%, and on the high-difficulty knowledge-and-reasoning benchmark HLE-Diamond it gets 39.4% versus a competitor’s 60.9%. The vendor says it wins 16 of 37 benchmarks against competitors — a mixed record: stronger on coding and agentic tasks, weaker on science and some reasoning. A third-party evaluation platform gives it an intelligence index of 55, ranking fourth globally.

    You don’t need to cite these numbers one by one, but the way to read them is worth summarizing:

    1. Check the source and comparability. Vendor-reported scores usually use carefully chosen prompt templates, sampling parameters, and scaffolds. Switch to your own business prompts and calling patterns and the results can be completely different.
    2. Look at the task distribution, not the total score. Strong on coding but weak on science means it fits code-assistant, automated-ops, and agentic-workflow scenarios, while science computation and complex mathematical derivation call for cautious evaluation.
    3. Benchmarks saturate and get contaminated. Public benchmarks have diminishing discriminative power, and training data may overlap with them. The most valuable reference is an evaluation set built from your own business data.
    4. The intelligence index is a composite score. Compressing multiple capabilities into one number makes ranking easy but hides structural weaknesses. Selection should return to your own needs.

    The practical recommendation is to build a real business evaluation set of 200 to 500 items, fix the prompt templates and inference parameters, run scores continuously, and compare across versions, so selection decisions rest on your own data.

    6. Licensing: Open Weights Are Not Open Source

    This is the single point most worth explaining separately from this weight release. The repository carries a custom license, the Kimi K3.1 Pro License, and the platform labels it license:other — not a standard open-source license like MIT or Apache-2.0.

    The license is word-for-word identical to the previous generation’s, with only the name changed. The core is indeed an MIT-style grant: free to use, copy, modify, distribute, fine-tune, and use commercially. But two restrictions are added on top, and they target only large-scale external business:

    1. Model-as-a-Service clause. Companies that offer the model as a service (MaaS) externally and whose cumulative revenue over 12 months exceeds $20 million must sign a separate agreement with the relevant rights holder before commercial use.
    2. Commercial product scale clause. Commercial products whose monthly active users exceed a certain scale (the original text is cut off here; refer to the license text for the exact number) likewise require separate authorization.

    To understand this license, grasp three things:

    • “Open weights” and “open source” are not the same thing. The industry-accepted definition of open source forbids discriminating against specific fields of use or products and prohibits added conditions like revenue or user counts. Custom licenses with such restrictions are usually classified as “source-available” rather than open source.
    • Small and mid-sized teams are basically unaffected. The vast majority of SMEs, startups, and individual developers don’t cross either threshold and can use it normally under the MIT-style license.
    • Teams near the thresholds should assess early. Teams with revenue or MAU close to the thresholds should factor license compliance into review at the product-planning stage, and watch for license updates and the rights holder’s interpretation to avoid being forced into changes after launch.

    For engineering, you can maintain a license-compliance ledger with at least the following:

    Check Item Key Question
    License type Standard open-source license, or a conditional custom license?
    Commercial use Is commercial use allowed? Are there revenue or MAU thresholds?
    Distribution & sublicensing Must copyright notices be retained on redistribution? Can it be kept closed-source?
    Derivative models Can fine-tuned models be published? Any naming or notice restrictions?
    Output ownership How is ownership of and restrictions on model-generated output defined?
    Compliance trigger If a threshold is crossed, how far in advance must authorization talks start?

    7. Deployment and Compliance Recommendations

    • Budget memory first, then talk GPU count. Plug your business’s context length and concurrency into the budget function above, multiply by a safety factor, and you get the minimum GPU configuration.
    • Validate in the cloud first. No need to buy an entire cluster up front. On cloud GPU instances you can use “object storage for weights + a container platform for the inference service + monitoring and alerting to track memory levels” to quickly validate feasibility, then decide on resource scale once it’s confirmed stable.
    • Model routing and tiering. A trillion-parameter MoE isn’t right for every request. Use a small model for simple tasks and route only complex tasks to this model to lower overall cost.
    • Continuous monitoring. Track memory usage, KV-Cache hit rate, expert load distribution, and P99 latency — these metrics directly determine service-quality stability.
    • Compliance up front. Licensing, data residency and cross-border data transfer, and generated-content safety should be part of review at the architecture-design stage, not patched before launch.

    Conclusion

    The release of a trillion-parameter open-weight model pushes the capability frontier of open-weight models higher again. For developers, the real barrier isn’t “can you download it” but “can you use it well”:

    • Structurally, it’s a sparse MoE that separates capacity from compute: memory is counted by total parameters, compute by activated parameters;
    • In deployment, 8 GPUs are just the start, and KV Cache and concurrency scale are the main cost variables;
    • In precision, the single 4-bit micro-scaling format depends on hardware and framework support;
    • In compliance, the custom license adds revenue and MAU thresholds, which must be checked line by line before commercial use.

    Once these are clear, a trillion-parameter model is no longer a distant “file in a repository” but an engineering capability you can plan for, measure, and use compliantly.