Trillion-Parameter MoE Weights Open: Architecture, Licensing, and Deployment Thresholds
This article uses a newly open-weighted 2.8T-parameter Mixture-of-Experts (MoE) model as a case study to explain the three things developers care about most: how a trillion-parameter model is architected, what a custom open-weight license actually permits and restricts, and what hardware and engineering setup it takes to run locally. It walks through the sparse MoE design, native multimodal and sparse-attention components, how to read vendor self-reports against third-party benchmarks, a four-step self-testing method, a license-compliance checklist, VRAM budgeting for an 8-GPU starting point, parallelism and inference-optimization strategies, and engineering practices for downloading and distributing large sharded weights. It closes with a trend outlook and practical guidance on whether, and in what scenario, local deployment makes sense.
Introduction: open weights enter the “trillion-parameter era”
Recently, a Chinese AI team released the weights of a Mixture-of-Experts (MoE) model totaling 2.8T parameters, as it had previously promised. Continued from its prior-generation native multimodal Agent model, the release weighs about 1.56TB in total, making it the largest model with publicly available weights to date. Two caveats are worth noting up front: the release uses a custom license rather than a standard open-source license, and local inference requires at least 8 high-VRAM accelerator cards.
Rather than judge a single product, this article uses it as a sample to answer the three questions developers care about most:
- How is a trillion-parameter model structured?
- What does an open-weight license actually allow and restrict?
- What hardware and engineering configuration is needed to actually run it?
1. Release timeline: hosted service first, then open weights
The team’s timeline is clear and reflects a rhythm common among top model teams over the past two years:
- Launch API and web service first. This quickly collects real traffic and feedback, and makes it easy to stress-test inference cost.
- Commit to a weight-release date to keep the developer community engaged.
- The weight repository was created two days before the release date, stayed closed that morning, and officially opened that evening — no application or approval required.
- The full weights were then synced to major model-community mirrors to relieve download pressure on a single hosting platform.
Once the weights landed, the community immediately gained three capabilities: independently reproducing third-party evaluations; parameter-efficient fine-tuning (such as LoRA) or private deployment based on the weights; and layer-by-layer analysis of attention and expert distributions to study the architecture.
Opening the weights also brings two knock-on consequences:
- Evaluation power shifts from the vendor to the user, so self-tested results may differ from official claims;
- The responsibility boundary moves down accordingly. Before going live, you must complete your own license review and safety evaluation rather than outsource compliance judgment to the model’s authors.
2. Anatomy: how the trillion parameters are constructed
According to the model card, the model is a native multimodal Agent model. Key structural parameters:
| Dimension | Configuration | Engineering implication |
|---|---|---|
| Total parameters | 2.8T | Determines weight storage and VRAM footprint |
| Activated parameters per token | 104B | Determines per-token compute and decode speed |
| Expert configuration | 896 routed experts, 16 selected per token, plus 2 shared experts | High sparsity; capacity decoupled from compute |
| Layers | 93 | Affects the granularity of parallel splitting |
| Context length | 1M tokens | Places extreme demands on KV cache and attention compute |
| Vision encoder | Self-developed, supports image-text input | Built-in component for multimodal capability |
| Quantization format | Weights in 4-bit microscaling FP4, activations in FP8 | Product of quantization-aware training (QAT) |
Compared with the previous generation, there is only one structural change: 24 layers switch from full attention to sparse attention, aiming to save compute under long context. The team provides only this single quantized version — no BF16 or FP8 variant. Both points look like details, but they directly determine the feasibility and cost of downstream deployment.
2.1 Sparse MoE: decoupling capacity from compute
In a standard Transformer, every parameter participates in the computation for every token, so parameter scale and inference cost scale roughly proportionally. MoE instead splits the feed-forward layer into multiple parallel “experts,” and a router (gating network) selects a few experts to activate for each token.
Take this model as an example: the activation ratio is only about 104B / 2.8T ≈ 3.7%, so roughly 96% of parameters stay dormant in a single forward pass. The benefits are threefold:
- Much larger knowledge capacity: different experts can specialize in different domains or languages, while per-inference cost stays roughly unchanged.
- Better training efficiency: more parameters can be trained under the same compute budget.
- Tunable sparsity: the number of experts, experts activated per token, and number of shared experts are all tuning knobs, trading off load balancing against model quality.
The costs are equally clear:
- VRAM cannot be saved: all expert weights must stay resident in VRAM (or be swapped in and out at very low cost), otherwise every routing step would load from storage.
- Load balancing is hard: popular experts form compute hotspots, so a capacity factor caps the number of tokens each expert receives, with fallback handling for overflow tokens.
- High communication overhead: when experts are distributed across devices, each layer performs all-to-all dispatch and combine, which is very sensitive to inter-card interconnect bandwidth.
The “2 shared experts” is a common design. Shared experts are always active for all tokens and carry general knowledge, while routed experts handle differentiated capabilities. This “shared + routed” combination reduces redundant parameters across experts and improves parameter utilization.
2.2 Native multimodal: the vision encoder is a standalone component
The model continues from the previous native multimodal model, so visual encoding is part of the base rather than an add-on module. At inference, an image is first turned into visual tokens by the vision encoder, then fed into the backbone network together with text tokens. This has two direct implications for deployment:
- The VRAM budget must additionally account for the vision encoder’s weights and the image-feature cache;
- Extra tokens from images eat into the context window.
For Agent scenarios, the value of this design is direct: the model can process UI screenshots, document charts, and code simultaneously over long-horizon tasks and call tools on top of that, without stitching together separate “text” and “vision” models.
2.3 Sparse attention: the long-context compute bill
Full attention’s compute and VRAM cost grows quadratically with sequence length. Let the sequence length be L; the attention matrix is L × L. When L grows from 32K to 1M (about 31×), the attention compute theoretically increases by roughly 1000×. So the key to long-context support is not parameters but the attention implementation.
Of the 93 layers, 24 switch to sparse attention so that these layers compute only limited token interactions (common strategies include sliding window, global tokens, and block-level selection). The remaining layers stay full attention to preserve long-range information integration. Note that:
- Sparse modes usually trade off first-token latency against long-range recall accuracy;
- 1M tokens is a support ceiling, not a guarantee that quality and cost are both ideal at that length;
- The specific sparse-attention implementation (for example, algorithms designed for long context) significantly affects long-context task performance, so selection should rely on hands-on testing.
3. Capability evaluation: reading vendor self-reports vs. third-party benchmarks
The biggest benefit of open weights is that they are “verifiable.” The team’s self-test data:
| Test dimension | This model (vendor self-reported) | Closed-source model A (vendor self-reported) | Closed-source model B (vendor self-reported) |
|---|---|---|---|
| Terminal / command-line task benchmark | 65.7% | 66.4% | 58.2% |
| Ultra-long-task software engineering benchmark | 51.7% | 46.3% | — |
| Scientific terminal benchmark | 38.1% | 63.3% | — |
| Multi-disciplinary hard knowledge-reasoning benchmark | 39.4% | 60.9% | — |
The vendor says the model beats the comparison model A on 16 of 37 benchmarks. Separately, a third-party composite intelligence index reportedly rated it 55 points, ranking fourth globally. Three takeaways:
- Capability is strongly scenario-dependent: the model performs on par with or ahead of closed-source flagships on coding and Agent tasks, but clearly trails on scientific reasoning and some hard reasoning.
- Self-reported scores cannot replace your own testing: prompt templates, inference settings (such as thinking budget), and sampling temperature all significantly affect scores, and vendors typically choose the most favorable, compute-feasible configuration. Details the vendor didn’t publish need to be reproduced and verified yourself.
- A composite ranking ≠ business performance: a composite index weights all capabilities equally, while a business only cares about specific scenarios.
Recommended four-step self-test method
- Lock the scenario: sample tasks from real business, e.g., code review, ticket classification, document Q&A, data extraction.
- Build a small eval set: 100–500 labeled samples are enough to surface obvious gaps. Also cover three “traps”:
- Long context: plant key information inside very long documents to test recall;
- Multimodal: samples containing charts and scans;
- Agent-style: multi-step tool-calling tasks.
- Fix evaluation conditions: standardize prompts, temperature, and max output length, then compare against existing models under the same conditions.
- Establish a regression baseline: rerun the same eval after every model update to avoid “an upgrade that breaks things.”
Sample code: a minimal evaluation script
import jsonimport redef normalize(text: str) -> str:"""Normalize whitespace and case to reduce the impact of irrelevant differences on scoring"""return re.sub(r"\s+", " ", text.strip()).lower()def run_eval(model_client, dataset_path: str):"""model_client: a unified model-call wrapper that must support switching between model backends"""correct = total = 0with open(dataset_path, encoding="utf-8") as f:for line in f:case = json.loads(line)pred = model_client.chat(case["prompt"], temperature=0)if normalize(pred) == normalize(case["answer"]):correct += 1total += 1print(f'{case["id"]}\tpred={pred[:40]!r}\texp={case["answer"]!r}')print(f"accuracy: {correct}/{total} = {correct / total:.2%}")
4. License review: open weights ≠ open source
This is the point most worth discussing on its own. The repository ships a custom “Kimi K3.1 Pro License,” labeled license:other on the platform rather than a standard open-source license such as MIT or Apache 2.0. The terms are word-for-word identical to the previous generation, with only the name changed.
Granting party: the terms are essentially MIT-style, allowing free use, copying, modification, distribution, fine-tuning, and commercial use.
Additional restrictions: aimed only at large-scale external business, mainly two:
- Entities that provide the model as a service (MaaS) externally and exceed $20M in cumulative revenue over 12 months must sign a separate agreement before commercial use;
- Commercial products whose monthly active users reach an agreed scale likewise require a separate agreement (the public terms are truncated here; refer to the official original text).
4.1 Why “open weights” ≠ “open source”
The industry typically uses the Open Source Initiative (OSI) definition as the prevailing standard for “open source.” That standard emphasizes free use, study, modification, and redistribution, and prohibits field-of-use restrictions. Measured against it, open weights differ in at least three ways:
- The license is custom: a license with added conditions is fundamentally different from a standard open-source license with unrestricted scope of use.
- Training data and code are not public: you can reproduce inference, but it is hard to independently reproduce training.
- Additional terms form a commercial boundary: most teams are unaffected, but entities that cross the thresholds must renegotiate.
4.2 Compliance review checklist
- Identify the using entity: whether revenue and MAU are counted per legal entity or per product, and whether affiliated companies are included;
- Define “providing an external service”: whether internal tools, private deployments, and model features embedded in a SaaS count as MaaS;
- Redistribution assessment: when packaging weights into your own product or a cloud-marketplace image, confirm the license allows it;
- Derivative models: whether a fine-tuned model remains bound by the original license (terms are usually inherited);
- Traceable archiving: keep the license text and hash of the downloaded version. Licenses may change across versions, and archiving is evidence of compliance;
- Involve legal early: when planning overseas expansion, fundraising, or M&A, bring legal in at the selection stage.
5. Local deployment: the arithmetic behind the 8-card starting point
5.1 VRAM budget estimation
With 4-bit weights (~0.5 bytes per parameter), a MoE model’s VRAM need can be estimated as:
VRAM need ≈ total parameters × bytes per parameter+ unquantized parts such as the vision encoder and word embeddings+ KV cache+ activations and temporary buffers
Plugging in this model:
- Backbone weights ≈ 2.8 × 10^12 × 0.5 B ≈ 1.4 TB (decimal, about 1.27 TiB);
- Actual weight files are about 1.56 TB; the ~0.1–0.2 TB gap roughly corresponds to the unquantized vision encoder, word embeddings, and quantization scale factors;
- 1.56 TB ÷ 8 ≈ 195 GB/card, so an “8-card start” means each card needs on the order of 200 GB of VRAM; ordinary ~80 GB training cards need about 20;
- Even with exactly 8 cards, very little room is left for KV cache: 1.6 TB − 1.56 TB = 0.04 TB, only about 40 GB.
Long context is a VRAM killer. The general KV-cache formula:
KV cache = 2 × layers × KV heads × head dim × context length × bytes per element
If the KV cache reaches hundreds of GB under a 1M context, an 8-card configuration simply cannot hold it; you need to measure it in practice, or combine with KV-cache quantization and offloading strategies. So 8 cards is the minimum line to “get it running,” not a “comfortable” configuration.
5.2 Communication and parallelism strategy
- Tensor parallelism (TP): splits a single layer’s operators across cards; suits high-speed intra-node interconnects such as NVLink.
- Expert parallelism (EP): distributes experts across cards; heavy all-to-all traffic strongly depends on high-speed interconnects, so cross-machine deployment needs special evaluation.
- Pipeline parallelism (PP): splits by layer, passing only activations between devices; suits cross-node scaling.
- Prefill / Decode disaggregation: prefill is compute-intensive while decode is memory-access-intensive; separating them lets each use different parallelism and batching strategies.
Also, 4-bit microscaling FP4 weights require matching operator support in the inference framework. If the hardware or framework does not support it, you can only dequantize to higher precision for compute, which cancels out the VRAM and bandwidth advantage.
5.3 Inference optimization techniques
- Quantized KV cache: reduce KV cache to FP8 or lower to trade for longer context;
- Paged cache management: reduce VRAM fragmentation and improve concurrency;
- Prefix cache reuse: in Agent scenarios, system prompts, tool definitions, and long documents often repeat, and reusing them greatly cuts repeated prefill cost;
- Speculative decoding: accelerate decoding with a small model or the model’s own draft mechanism;
- Expert load balancing and continuous batching: reduce latency jitter from hotspot experts;
- On-disk weight cache: keep weight files on fast local disks to avoid repeatedly reading remote storage during cold starts.
5.4 Sample code: a VRAM budget estimator
def estimate_vram_kb(total_params, quant_bytes, layers,kv_heads, head_dim, ctx_len, kv_bytes,buffer_gb=2.0):"""Rough VRAM estimate; MoE models must additionally add the unquantized parts and expert metadata"""weights_tb = total_params * quant_bytes / 1e12kv_cache_tb = (2 * layers * kv_heads * head_dim * ctx_len * kv_bytes) / 1e12return weights_tb + kv_cache_tb + buffer_gb / 1e3if __name__ == "__main__":# Values are illustrative; refer to the actual model configurationneed = estimate_vram_kb(total_params=2.8e12, quant_bytes=0.5, layers=93,kv_heads=8, head_dim=128, ctx_len=32768, kv_bytes=1)print(f"32K context VRAM budget ≈ {need:.2f} TB; at 0.2 TB per card, at least {need / 0.2:.0f} cards are needed")
5.5 Phased deployment recommendations
- Phase 1: validate via hosted API. First wire up the business flow with the API, quantify call volume, latency, and cost, and confirm whether the model truly meets the need.
- Phase 2: choose the deployment form as needed. You can get a preconfigured instance via one-click deployment from a model marketplace, or build your own with a container platform + high-performance parallel file storage + a high-VRAM GPU instance group. For inference serving, put a gateway and load balancer in front, and set up logging, monitoring/alerting, and per-token usage metering.
- Phase 3: cost balancing. If request volume is stable and data-compliance requirements are high, consider private deployment; when request volume fluctuates a lot, pay-as-you-go hosted service is usually more economical.
- Phase 4: lightweighting. If you only need part of the model’s capabilities, distill to a smaller model or use LoRA to adapt to a specific task, greatly lowering the hardware threshold.
6. Weight download and distribution: engineering practice for large-scale sharding
1.56TB across 96 weight shards is no small test for the download and distribution pipeline. The repository has 112 files in total; besides the weight shards, it also contains model config, index files, a tokenizer, the vision encoder, and generation config.
6.1 Mirror and bandwidth strategy
- Prefer a nearby mirror source to avoid cross-border downloads;
- Host the weights on object storage with a content delivery network (CDN) in front for acceleration;
- Use multi-threaded segmented downloads with resume support;
- Verify each file’s hash after download to ensure shard integrity.
6.2 Sample code: resumable download + hash verification
import hashlibimport osimport urllib.requestfrom concurrent.futures import ThreadPoolExecutorBASE_URL = "https://example.com/models/moe-weight-repo" # replace with the actual mirror addressdef sha256_file(path: str, chunk_size: int = 8 << 20) -> str:h = hashlib.sha256()with open(path, "rb") as f:for chunk in iter(lambda: f.read(chunk_size), b""):h.update(chunk)return h.hexdigest()def download(url: str, dst: str, expect_sha256: str | None = None) -> str:"""Single-file download with resume support, plus an integrity check on completion"""tmp = dst + ".part"# The number of already-downloaded bytes determines the start of the Range requestoffset = os.path.getsize(tmp) if os.path.exists(tmp) else 0req = urllib.request.Request(url)if offset:req.add_header("Range", f"bytes={offset}-")with urllib.request.urlopen(req, timeout=120) as resp, open(tmp, "ab") as out:for chunk in iter(lambda: resp.read(1 << 20), b""):out.write(chunk)if expect_sha256 and sha256_file(tmp) != expect_sha256:raise ValueError(f"verification failed; delete {tmp} and re-download")os.replace(tmp, dst)return dstdef fetch_all(file_list: list[dict], workers: int = 4):"""Each item in file_list looks like {"name": "xxx", "url": "...", "sha256": "..."}"""with ThreadPoolExecutor(max_workers=workers) as pool:futures = [pool.submit(download, f["url"], f["name"], f.get("sha256"))for f in file_list]for fut in futures:print("ready:", fut.result())
After the download completes, you can mount read-only and list the files with:
# Read-only mount to avoid corrupting the weights with accidental writessudo mount --bind -o ro /mnt/nvme/weights /mnt/weights-rols -lh /mnt/weights-ro | head
6.3 Storage planning recommendations
- Capacity: estimate “one primary copy + 1–2 hot copies”; 3 TB of local NVMe is just the starting point;
- Cold backup: weights can be re-downloaded, but to shorten recovery time, archive the verified version to object storage with versioning enabled;
- Bandwidth: for offline incremental sync to a remote data center, prefer a dedicated line or migration service to avoid saturating the public egress;
- Permissions: set least privilege on the weight directory and run the inference service as a non-root user.
7. Trend outlook and action recommendations
Three trends emerge from this weight release:
- “Sparsification + quantization” is becoming the standard combo for trillion-parameter models: sparse MoE handles capacity while quantization-aware training handles deployment cost. Only when both land does “open weights” become genuinely usable.
- Evaluation power is moving down: after weights open, evaluation is no longer a vendor monologue but is done by the community and enterprises in real scenarios. Whoever builds their own eval system faster can make selection decisions faster.
- License compliance is becoming a standalone engineering topic: the combo of a custom license plus commercial thresholds requires enterprises to bring legal into the process at the model-selection stage, rather than patching the review before going live.
Action recommendations for developers:
- Evaluate: compare on real business data under equal conditions; don’t blindly trust leaderboards;
- Deploy: do the VRAM math first, then choose card count and parallelism strategy — 8 cards is a floor, not a goal;
- Compliance: archive the license text and hash on download; start the authorization conversation before crossing revenue or MAU thresholds;
- Reduce cost: prioritize hosted service or marketplace instances to validate value, then consider private deployment or distillation/lightweighting once stable.
Opening the weights of a trillion-parameter model marks a shift in the LLM competition from a “parameter race” to a “comprehensive race of engineering and compliance.” For most teams, more important than “can it run locally” is “should it run locally, and in what scenario.” Only by thinking these two questions through before deploying can you truly turn open weights into business value.
