Deploying a 2.8T-Parameter MoE Open-Weight Model Locally: A Practical Guide

    After a Chinese AI research team released the full weights of its 2.8-trillion-parameter flagship model, K3.1 Pro, this guide examines what the release actually contains and what it means for deployment. It breaks down the MoE architecture and MXFP4/MXFP8 quantization, explains how to read the vendor's self-reported benchmarks, clarifies that the open weights ship under a custom license rather than a standard open-source one, and works through the real GPU memory math—showing why an eight-GPU, data-center-class setup is the practical starting point. It also covers distributed inference strategies, weight download and loading, cloud deployment practices, and guidance on who should adopt the model now versus wait.

    1. Background: The Largest Open-Weight Model to Date

    On November 4, a Chinese AI research team released the complete weights of its next-generation flagship model, K3.1 Pro, as it had previously promised. Announced at the team’s launch event, the news quickly spread through the community. The model has 2.8T (roughly 2.8 trillion) parameters and a download size of about 1.56TB, and it is being described as the most capable openly available model to date. This “open” release, however, comes with two caveats: the license sets thresholds for very large-scale businesses, and running it locally requires at least eight high-end accelerator cards.

    Release timeline

    • October 24: The model first became available through online services (API and web), with the team committing to releasing the full weights by November 4.
    • November 2: A public weight repository was created on a major model-hosting community, initially kept closed.
    • November 4: The repository was still closed in the morning, then officially opened for download that evening with no application or approval required. The weights comprise 112 files, 96 of which are tensor shards totaling about 1.56TB (roughly 1.42TiB in binary terms), and the same weights are mirrored on other model-hosting communities.

    For developers who care about the technology’s evolution and need to deploy models locally or in private environments, three questions are worth answering:

    1. What exactly is inside the weights;
    2. What kind of license this is;
    3. Whether an ordinary team can actually run something this large (1.56TB).

    The sections below address each in turn.

    2. The Model Itself: Where the 2.8T Parameters Come From

    2.1 MoE is the key to the whole deployment question

    According to the model card, this is a natively multimodal agent model, continued-trained from the previous generation. It has 2.8T total parameters but activates only about 104B per token—an activation ratio of roughly 3.7%.

    There is a concept here that is easy to confuse: total parameters determine memory footprint, while activated parameters determine compute.

    • Loading stage: All expert weights must be placed in GPU memory or host memory, so the memory threshold is dictated by the 2.8T total.
    • Inference stage: Each token interacts computationally with only a few experts, so the total compute is comparable to a dense 104B model.

    This is exactly why the model is “enormous in parameter count, hard to deploy locally, yet not necessarily slow to run inference.”

    2.2 Structural parameters at a glance

    Dimension Value Engineering implication
    Total parameters ~2.8T Determines weight size and the memory floor
    Activated per token ~104B Determines compute per inference
    Routed experts 896, 16 selected per token Requires large memory; split via expert parallelism
    Shared experts 2, involved for every token Carry general capability; touched by every request
    Layers 93 ~30B compute per layer; memory demand grows linearly with depth
    Attention structure First 24 layers changed from full attention to sparse attention (DSA) Eases the compute and bandwidth pressure of attention under long context
    Context length 1,000,000 tokens Implies a massive KV cache overhead
    Vision encoding Built-in vision encoder MoonViT-V2 Can understand images; this is a multimodal model
    Quantization format Weights MXFP4, activations MXFP8 Produced via quantization-aware training (QAT); halves memory and bandwidth overhead or more

    Compared with the previous generation, only one structural change was made: 24 full-attention layers were replaced with sparse attention. Under very long context, standard attention’s compute and memory traffic grow quadratically with sequence length, and the sparsification targets precisely that bottleneck.

    One more point worth stressing: the team released only the MXFP4/MXFP8 version and did not provide a BF16 or FP8 version. This means:

    • You cannot choose a higher inference precision on demand;
    • Users who need BF16 weights for evaluation or research must either wait for a later official release or dequantize themselves—though dequantization cannot recover the precision discarded during training.

    2.3 What MXFP4 and MXFP8 are

    MXFP4 and MXFP8 are low-bit data types belonging to the Microscaling (MX) family, backed by public specifications:

    • MXFP4: 4-bit mantissa, with every 32 elements sharing an 8-bit scale factor, averaging about 4.25 bits per parameter;
    • MXFP8: 8-bit mantissa, likewise sharing a scale factor across each group of 32 elements.

    Compared with ordinary INT4 quantization, sharing a scale factor per group adapts more finely to the dynamic range of different value intervals, representing the long-tailed distribution of LLM weights more accurately. Combined with the fact that the weights come from quantization-aware training (which simulates quantization error during training so the model learns to live with it), the accuracy loss under MXFP4 is typically smaller than with post-training quantization.

    2.4 The details behind the 1.56TB figure

    Estimating at 2.8T parameters and about 4.25 bits per parameter, the parameters alone come to roughly 1.49TB. Adding scale factors and a small number of higher-precision parameters, the officially reported 1.56TB is reasonable, with the vast majority being weight data. Two implications for deployment follow:

    • The official size is an upper-bound reference for capacity planning: the 99 non-shard files (configs, indices, the vision encoder, and so on) account for only a small fraction, so memory and bandwidth evaluation revolve primarily around the weight data;
    • The sharded format makes streaming loads and per-card parallelism easy, but the loader must ensure every shard is consistent with the index—a single missing or corrupted shard will cause weight assembly to fail.

    3. How to Read the Benchmarks: Interpreting Vendor-Reported Numbers

    Before the weights were released, the team published a set of benchmark results. Vendor-reported data can indicate a model’s capability profile, but it should not be treated directly as a selection criterion.

    Start with code and agent capabilities: The official comparison table uses two mainstream overseas closed-source flagship models as references, with representative examples below (all vendor-reported):

    Benchmark This model Reference A Reference B
    Terminal task benchmark 65.7% 66.4% 58.2%
    Long-horizon software engineering benchmark 51.7% 46.3% —
    Scientific terminal task benchmark 38.1% 63.3% —
    Hard composite reasoning benchmark 39.4% 60.9% —

    The team states that it outperformed Reference A on 16 of 37 benchmarks. The table reveals a clear pattern: the model performs strongly on programming and agent tasks such as terminal operation and software engineering—trading wins and losses with Reference A—while lagging noticeably on scientific reasoning and composite reasoning tasks. In addition, a third-party evaluation organization’s intelligence index ranks it fourth globally.

    Three reminders for interpreting the data:

    1. All numbers are vendor-reported. Deployers should re-test on their own business datasets, focusing on output-format stability, tool-call success rate, and long-context performance.
    2. Watch for benchmark saturation and contamination. For widely circulated benchmarks, training data can rarely rule out overlap entirely, so scores may be inflated. Leaderboard rankings are just a reference coordinate, not a technical conclusion.
    3. Capability gaps amplify deployment cost. On task types where the model is clearly weaker, you may need multiple retries, larger sampling budgets, or external tools to compensate—which directly drives up compute costs. Cost estimation should be based on real per-task token consumption, not the ideal case of a single call.

    4. The License: Open Weights Are Not the Same as Open Source

    This is the most important point to call out separately. The weight repository ships with a custom license (the Kimi K3.1 Pro License), and the repository metadata marks the license type as other—not a widely recognized standard open-source license like MIT or Apache 2.0. The license is word-for-word identical to the previous generation’s, with only the name changed.

    Licensing body: a permissive, near-MIT grant. Free use, copying, modification, distribution, fine-tuning, and commercial use are all permitted, with essentially no additional constraints on individual developers and small-to-medium businesses.

    Additional terms: aimed only at very large-scale businesses. Commercial use requires a separate agreement with the model provider when either of the following conditions is met:

    • Condition 1: Offering the model commercially as a “model-as-a-service,” with cumulative revenue exceeding $20 million within 12 months;
    • Condition 2: The commercial product exceeding a certain monthly active user (MAU) scale. The exact threshold values are governed by the license text.

    Understanding its nature precisely matters:

    • Under mainstream open-source conventions, a mature open-source license may not impose discriminatory restrictions based on use case, user scale, or revenue (such as “no commercial use” or “a separate license is required above a certain scale”).
    • This license is a hybrid of “open weights + additional conditions”: the core grant is permissive enough, but the additional terms mean it does not meet the strict definition of open source.
    • For the vast majority of users, the effective rights are no different from a permissive license; only once scale reaches the agreed thresholds does commercial negotiation need to begin.

    Usage recommendations:

    1. Track usage metrics continuously: Service revenue and product MAU are the core trigger indicators, so it is best to build measurement and alerting into the product design phase.
    2. Distinguish internal from external scenarios: Internal R&D and offline evaluation generally do not trigger the additional terms, but public-facing paid services should be assessed in advance.
    3. Keep audit records: Retain call-volume, revenue, and active-user data to make compliance evidence easy to produce when needed.
    4. Evaluate fine-tuned derivatives separately: Whether a derivative model obtained by fine-tuning this model is likewise bound by the additional terms should be governed by the license text and confirmed with legal counsel in advance.

    5. Can It Run Locally? Doing the GPU Memory Math

    The conclusion up front: for individual developers and most enterprises, “running it locally” does not mean installing it on your own machine, but rather deploying it on owned or rented GPU servers with large memory. The so-called “8 cards” refers to high-end data-center-class accelerators, not eight consumer-grade graphics cards.

    The reason lies in memory scale:

    • Eight high-end cards commonly have 80–96GB each, totaling only 640–768GB—less than half the total weight size;
    • Consumer cards typically have 24–32GB each; eight of them total under 256GB;
    • An eight-card setup that can actually hold 1.56TB of weights requires high-end accelerators with nearly 192GB per card. Eight of them total about 1.54TiB (roughly 1.65TB), leaving extremely little headroom after the weights;
    • A safer approach is a hybrid of “GPU memory + host memory.” The inference engine offloads part of the weights or the KV cache to host memory, trading performance for feasibility.

    5.1 GPU memory usage has four components

    1. Total memory demand = weights + KV cache + activations & intermediates + framework & communication overhead
    • Weights: a fixed ~1.56TB, an unavoidable floor for every approach;
    • KV cache: grows linearly with “concurrency × context length,” and is the easiest part to lose control of;
    • Activations & intermediates: related to batch size and sequence length, generally requiring a few GB to tens of GB reserved;
    • Framework & communication overhead: including the runtime, CUDA graphs, NCCL communication buffers, and so on, usually reserving 5%–10%.

    After an eight-card single machine holds the weights, only a few tens of GB remain, which translates into a KV cache that supports only limited concurrency and context length. Long-context and high-concurrency capability must be bought with hybrid offloading or scale-out.

    5.2 How large is the KV cache

    For a standard autoregressive model, each token per layer requires caching two tensors, Key and Value:

    1. KV bytes = 2 × layers × KV heads × head dim × bytes per element × total tokens × concurrency

    KV size depends on the model’s hidden dimension and attention-head configuration, which are governed by the model config. Estimating from the 93 layers disclosed in the source material, even if the per-token KV footprint is small on average, at a 1,000,000-token context the KV cache reaches the GB to hundreds-of-GB range—and that is before multiplying by concurrency. This is also why inference engines broadly support KV quantization, paged management, and prefix caching.

    5.3 A quick estimate with a short script

    Plug the model config into the script below to get a preliminary estimate of memory and theoretical limits (the parameters below come from public information and are for demonstrating the method only):

    1. import json
    2. CFG = {
    3. "num_hidden_layers": 93, # Number of layers
    4. "num_kv_heads": 64, # Illustrative: KV head count; replace with the real config
    5. "head_dim": 128, # Illustrative: head dim; replace with the real config
    6. "max_position": 1_000_000, # Context length
    7. "total_params": 2.8e12, # Total parameters
    8. "active_params": 104e9, # Activated parameters per token
    9. }
    10. CARD_MEM_GB = 192 # Per-card memory
    11. CARD_BW_TBPS = 8 # Per-card memory bandwidth, illustrative value
    12. CARD_FLOPS = 2e15 # Per-card low-precision compute (2 PFLOPS, illustrative)
    13. CARD_COUNT = 8
    14. GB = 1e9
    15. weight_gb = 1560 # Officially reported total weight size (GB)
    16. total_mem_gb = CARD_MEM_GB * CARD_COUNT
    17. room_gb = total_mem_gb - weight_gb # Room left for KV, activations, and the framework
    18. def kv_gb(seq_len, concurrency, kv_bytes=1):
    19. """kv_bytes: KV bytes per token per layer, computable from the real config"""
    20. return (2 * CFG["num_hidden_layers"] * CFG["num_kv_heads"]
    21. * CFG["head_dim"] * kv_bytes * seq_len * concurrency) / GB
    22. # Weights read per token: activated params × bytes per param (MXFP4 ~ 4.25 bits)
    23. read_per_token_gb = CFG["active_params"] * 4.25 / 8 / GB
    24. # Theoretical decode ceiling: aggregate memory bandwidth ÷ per-token read volume
    25. bandwidth = CARD_BW_TBPS * CARD_COUNT * 1000 # GB/s
    26. decode_tps = bandwidth / read_per_token_gb
    27. # Theoretical prefill throughput: aggregate compute ÷ per-token compute (~2 × activated params)
    28. flops = CARD_FLOPS * CARD_COUNT
    29. prefill_tps = flops / (2 * CFG["active_params"])
    30. if __name__ == "__main__":
    31. print(f"Available memory headroom: {room_gb:.0f} GB")
    32. for n in (8192, 32768, 131072):
    33. used = kv_gb(n, 1)
    34. # Convert all headroom into a token budget at 2 bytes per token per layer (illustrative)
    35. budget = room_gb / kv_gb(1, 1, kv_bytes=2)
    36. print(f"Context {n:>7} × 1 concurrency: KV ~ {used:>6.0f} GB, "
    37. f"{'fits' if used < room_gb * 0.7 else 'exceeds safe headroom'}")
    38. print(f" All headroom supports ~ {budget/n:.0f} concurrent sequences of this length")
    39. print(f"Weights read per token: {read_per_token_gb:.0f} GB")
    40. print(f"Theoretical decode ceiling: {decode_tps:.0f} tokens/s (excludes communication and memory-access efficiency losses)")
    41. print(f"Theoretical prefill throughput: {prefill_tps:.0f} tokens/s")

    The script’s output is theoretical, meant to judge the feasibility boundary; real throughput must be assessed in combination with communication efficiency, expert load balancing, and framework implementation, and actual values are typically lower than theoretical ones.

    6. Distributed Inference: How the 8 Cards Are Organized

    Deploying a 2.8T-parameter MoE model on an eight-card single machine comes down to arranging three things well: who stores what, who computes what, and who communicates with whom. The three common parallelism strategies are:

    • Tensor parallelism (TP): Shards each layer’s weight matrix across multiple cards, requiring an all-reduce synchronization per forward pass of each layer. Suited to high-bandwidth intra-node interconnects (such as NVLink-class solutions).
    • Expert parallelism (EP): Distributes the 896 routed experts across different cards, with tokens sent via all-to-all communication to the card holding the relevant expert. Scales well but is sensitive to network bandwidth and expert load balancing.
    • Pipeline parallelism (PP): Splits by layer, passing activations between cards. Memory-friendly but introduces pipeline bubbles, so it is usually a supplementary measure.

    For an eight-card single machine, a combination of TP and EP is most common. Below is an illustrative deployment config; parameter names vary by framework, and the point is to convey the configuration approach:

    1. # Single-machine 8-card deployment example; parameter names vary by inference framework
    2. model:
    3. weight_format: mxfp4 # Weights are MXFP4, activations computed in MXFP8
    4. dtype: "bfloat16" # Non-quantized modules like attention and normalization stay at higher precision
    5. parallel:
    6. tensor_parallel_size: 8 # Tensor parallelism across 8 cards on one machine
    7. expert_parallel_size: 8 # Expert parallelism, routed experts split by card count
    8. memory:
    9. gpu_memory_utilization: 0.90 # Reserve space for activations and fragmentation
    10. kv_cache_dtype: "fp8" # Reduce KV cache footprint
    11. max_model_len: 32768 # Set as needed; a 1M context needs separate evaluation
    12. scheduling:
    13. max_num_batched_tokens: 32768 # Cap tokens per batch to balance throughput and memory

    Three things to note:

    1. KV quantization is not always lossless. When length and concurrency requirements are low, FP16/BF16 KV caches often yield better output quality.
    2. Set context length as needed. If real business requests never exceed 30,000 tokens, there is no need to enable the 1M context by default. The KV budget should be allocated to concurrency first, not extreme length.
    3. MoE configs beyond the TP size need extra validation. Some inference frameworks require the expert-parallel size to be no smaller than the tensor-parallel size. On an eight-card machine you can configure TP=2, EP=8 (experts split across cards); cross-node scaling is better suited to an “intra-node TP + cross-node PP/EP” combination, but configuration and validation costs rise significantly.

    7. Downloading and Loading the Weights: Easily Overlooked Engineering Details

    1. Download and verification. A 1.56TB download puts pressure on both network and storage:

    • Prefer download tools that support shard-level concurrency and resumable downloads; the same weights are available through other model-hosting communities;
    • After all files are downloaded, verify each file’s hash (the repository usually provides a checksum manifest) and confirm the index file is consistent with the shard manifest.

    2. Storage and loading strategy. The loading approach depends on the combination of GPU memory and host memory:

    • Direct loading: Each rank reads shards straight into GPU memory; fast startup, but the host side does not need to hold all the weights;
    • Host-memory staging: Weights are first loaded into host memory, then distributed to each card; host memory needs to be at least 1.5TB or more;
    • Disk offloading: When host memory is insufficient, the inference engine can read weights from disk on demand; startup is slower, and runtime may suffer jitter from paging in and out.
    1. import os, json, shutil
    2. def check_weight_dir(path: str, index_name: str = "model.safetensors.index.json"):
    3. """Pre-flight check before inference: are shards complete, is the index consistent, is local temp space sufficient"""
    4. index_path = os.path.join(path, index_name)
    5. if not os.path.isfile(index_path):
    6. raise FileNotFoundError(f"Missing index file: {index_path}")
    7. with open(index_path, "r", encoding="utf-8") as f:
    8. shards = sorted(set(json.load(f)["weight_map"].values()))
    9. missing = [s for s in shards if not os.path.isfile(os.path.join(path, s))]
    10. if missing:
    11. raise RuntimeError(f"Missing {len(missing)} shards: {missing[:3]} ...")
    12. total = sum(os.path.getsize(os.path.join(path, s)) for s in shards)
    13. print(f"Shard check passed: {len(shards)} shards, weights total {total/1e12:.2f} TB")
    14. # In disk-offload mode, reserve ~30% temp space; in direct-load mode, used to stage converted weights
    15. target = os.environ.get("HF_HUB_CACHE", os.path.join(os.path.expanduser("~"), ".cache"))
    16. free = shutil.disk_usage(target).free
    17. need = total * 0.3
    18. if free < need:
    19. raise RuntimeError(f"Available disk {free/1e12:.2f} TB is below the recommended {need/1e12:.2f} TB")
    20. if __name__ == "__main__":
    21. check_weight_dir("./K3.1-Pro") # Local weight directory

    3. Precision and sampling parameters. As a QAT model, MXFP4 weights participate directly in computation at low precision. Before going live, fix sampling parameters such as temperature and top_p and run regression tests to avoid output-format drift that causes tool-call failures. For JSON-output scenarios, pair it with structured-output constraints.

    4. Fine-tuning and secondary development. Full-parameter fine-tuning demands far more memory and optimizer-state overhead than inference. If you only need domain adaptation, consider parameter-efficient fine-tuning first (such as LoRA-style methods). Also confirm whether the inference framework supports mounting adapters on an MXFP4 base.

    8. Cloud Deployment Practices

    For teams without the conditions to build their own eight-card GPU cluster, on-demand cloud deployment is a more realistic option. It avoids a one-time hardware investment and allows elastic scaling on demand. General recommendations:

    1. Weight distribution and transfer: Store the 1.56TB of weights in object storage and distribute them to each node via parallel shard downloads or data-transfer services, avoiding having every machine pull them repeatedly from the public internet;
    2. Heterogeneous compute scheduling: Run the inference service on a container platform or bare-metal GPU servers, preferring instance types with high-speed intra-node interconnects and stable topology to reduce cross-node communication overhead;
    3. Serving and elasticity: Expose an API through load balancing and auto-scale based on request queue length and GPU utilization; because the memory threshold is high, plan scaling granularity in whole-machine units;
    4. Observability: Set up monitoring and alerting for GPU memory usage, KV cache hit rate, queue latency, time-to-first-token (TTFT), and time-per-output-token (TPOT);
    5. Cost control: Design separate resource pools for offline batch processing and online interaction. The former tolerates queuing and suits low-cost or interruptible instances; the latter needs stable latency and suits always-on high-memory instances.

    Baidu AI Cloud’s general capabilities in high-performance computing, containerized deployment, and observability can serve as a reference option when implementing such a solution.

    9. Selection Guidance: Who Should Use It, and Who Should Wait

    Use case Recommendation
    Individual developers, research and learning Weights can be downloaded freely, but the local hardware threshold is high. Try it via on-demand cloud instances or the official API; there is no rush to build your own
    Small-to-medium teams (below the license threshold) You can rent a single eight-card high-end instance in the cloud for small-scale production validation; programming and agent scenarios show clear gains, while scientific reasoning scenarios need supplementary solutions
    Enterprises with high data-security requirements Private deployment is feasible, but you must plan ahead for data-center power, cooling, high-speed interconnect, and operations capability; hardware investment and operating costs are high
    Very large-scale businesses When revenue or MAU approaches the license threshold, start commercial discussions early and retain usage data

    For most teams, the pragmatic path is: first use the API to validate the fit between the model and your business scenario, then evaluate token consumption and unit cost on real data, and only then decide whether to invest in your own compute.

    10. Conclusion

    The significance of these weights lies not only in resetting the parameter scale of open models, but in putting an engineering problem squarely on the table: 2.8T parameters, 1.56TB of weights, only MXFP4 precision, and a custom license with tiered restrictions—these four things together determine what it can do, how to deploy it, and whether it can be used commercially.

    For developers, the conclusions are clear:

    • The memory math determines the deployment form (an eight-card setup is a hard threshold);
    • The license terms determine the applicable boundaries (fine for the vast majority of scenarios; very large-scale businesses must communicate in advance);
    • The benchmark results determine the suitable scenarios (strong in programming and agents, still weak in scientific and composite reasoning).

    Only by weighing all these factors together in your selection can you make a rational technical decision.