Deploying Ultra-Scale Open Weights: 2.8T Parameters, 1.56TB, and the 8-GPU Threshold
This article examines the real-world deployment of Kimi K3.1 Pro, the largest open-weight model publicly available to date at roughly 1.56TB and 2.8T total parameters. Instead of focusing on launch-time marketing, it analyzes three practical layers once the weights are on the ground: model architecture (2.8T total with 104B active per token, 896 routed experts, 24 sparse-attention layers, a 1M-token context, and 4-bit weights), deployment requirements (why 8 GPUs is only a starting point, and how memory, KV cache, throughput, and latency shape real capacity), and licensing (why open weights are not open source, including the MaaS revenue and MAU thresholds). It also explains how to read the vendor's self-reported benchmarks. Engineers will learn the three bills to account for—memory, throughput, and license—when deciding whether such a model is actually usable for their workload.
1. Introduction: When “Downloadable” Meets “Deployable”
On the evening of November 4, a vendor made good on its promise and publicly released the full weights of Kimi K3.1 Pro on a model hosting platform. The repository is accessible without any application process. The weights comprise 112 files across 96 shards, totaling about 1.56TB.
This is the largest open-weight model that has ever been publicly downloadable. Once the news broke, developers focused on three questions: what is actually inside, whether the license permits use, and whether an ordinary person can run it.
This article does not discuss the launch-time performance claims. Instead, it focuses on the facts that follow once the weights are on the ground, analyzing three layers: structure, license, and compute.
- Hardware: an 8-GPU starting point effectively rules out individual consumer devices and pushes the bar toward enterprise-grade infrastructure.
- License: open weights are not the same as open source, and large-scale commercial use carries additional constraints.
- Benchmarks: vendor self-reported data has reference value, but it cannot replace independent verification.
2. Structure Analysis: How 2.8T Parameters Are Composed
According to the model card, Kimi K3.1 Pro is a natively multimodal agent model, continued training from the previous generation Kimi K3. Its key architecture is as follows:
| Dimension | Configuration |
|---|---|
| Total params / active params per token | 2.8T / 104B |
| Expert structure | 896 routed experts, 16 selected per token, plus 2 shared experts |
| Layers | 93 |
| Attention | 24 layers use sparse attention (DSA); the rest are full attention |
| Context length | 1M tokens |
| Visual encoder | MoonViT-V2, handles image inputs |
| Weight / activation precision | 4-bit floating point / 8-bit floating point, products of quantization-aware training |
Compared to K3, the structure changed in only one place: 24 full-attention layers were replaced with sparse attention. What is more noteworthy are several design trade-offs that did not change.
2.1 Total Parameters vs. Active Parameters: Why the 3.7% Ratio
104B is about 3.7% of the 2.8T total parameters. Estimating with the parameter-count convention commonly used in MoE inference:
# Illustrative comparison of publicly listed parameterstotal_params = 2.8e12 # total parametersactive_params = 104e9 # active parameters per tokenexperts, top_k, shared = 896, 16, 2active_ratio = active_params / total_params # ~3.7%expert_hit_rate = top_k / experts # ~1.8%shared_share = shared / (top_k + shared) # ~11.1%print(f"Active parameter ratio: {active_ratio:.1%}") # Active parameter ratio: 3.7%print(f"Routed expert hit rate: {expert_hit_rate:.1%}") # Routed expert hit rate: 1.8%
These two sets of numbers explain two basic characteristics of this class of models:
- Low compute: the forward compute per token is roughly equivalent to a 104B dense model. The 18th shared expert is also reused by every token, which is one key to stable quality.
- Large memory: MoE routing is data-dependent. In a single batch, different samples and different positions may hit different experts, so it is impossible to know in advance which weights will go unused. Only by holding all 2.8T parameters in GPU memory can inference avoid stalls waiting for an expert to load mid-run.
This is also why high-throughput MoE serving commonly adopts expert parallelism. Splitting different experts across different devices lets each device read only the portion of weights it is responsible for. The cost is inter-device communication, which the deployment section below expands on.
2.2 Sparse Attention: Saving Compute for Million-Level Contexts
The compute of full attention grows quadratically with sequence length. When the context is pushed to a million tokens, attention itself can become a bottleneck even if the model backbone only activates 104B.
After 24 layers are switched to sparse attention, each layer only attends to a carefully selected subset of historical tokens, significantly reducing the per-layer attention cost. The remaining layers keep full attention to preserve the ability to integrate global information across many distant positions.
Two points need to be clear:
- This is only an architectural inference. The specific sparse pattern and how much historical information is retained per token are not fully disclosed in the model card and must be verified empirically before deployment.
- Sparse attention reduces compute, not the storage of the KV cache. The cache usually still retains keys and values for all historical tokens, so the memory pressure from long contexts does not go away because of this.
2.3 4-bit Weights: The Chain Effects of Precision
The weights use a 4-bit floating-point format and the activations use 8-bit floating point, both products of quantization-aware training, with the goal of keeping actual inference precision close to the quantization target. This also explains an apparently contradictory phenomenon: 2.8T parameters, yet a download of only about 1.56TB.
A quick check: 2.8×10¹² parameters at 0.5 bytes each (4-bit) is about 1.4TB. Add the overhead of quantization group factors, embedding layers, vocabulary, and so on, and the total lands around 1.56TB, which matches the order of magnitude.
Quantization compression brings three chain effects:
- Lower download and load cost: 1.56TB is far smaller than a same-scale BF16 version (theoretically about 5.6TB), so the initial download and load are much faster.
- Only one precision is provided: the vendor did not release BF16 or FP8 versions. Researchers who want to run precision ablation studies or secondary training must handle format and precision conversion themselves.
- Deployment depends more on hardware and framework support: if the inference framework cannot efficiently compute 4-bit weights, it must dequantize to higher precision at load time, which cancels out the size and bandwidth advantages.
3. Deployment Analysis: What 8 GPUs Minimum Means
3.1 The Memory Ledger: 8 GPUs Is Only the Starting Point
Splitting 1.56TB of weights across 8 GPUs means each card must hold about 195GB of weights on average. On top of that come the KV cache, activations, communication buffers, and framework overhead. A rough estimate:
# Memory budget example (illustrative; excludes communication buffers and other overhead)weights_bytes = 1.56e12 # total weightsgpus = 8gpu_mem_gb = 288 # assumed per-GPU memorygpu_mem_bytes = gpu_mem_gb * 1024**3weights_per_gpu = weights_bytes / gpus # ~195GB (decimal)headroom = gpu_mem_bytes - weights_per_gpu # ~113GB (after binary conversion)util_ratio = weights_per_gpu / gpu_mem_bytesprint(f"Weights per GPU: {weights_per_gpu / 1024**3:.0f}GB")print(f"Headroom per GPU: {headroom / 1024**3:.0f}GB")print(f"Weight memory share: {util_ratio:.1%}")
The conclusion is direct: 8 GPUs means each card needs to be in the ~200GB memory class, and once fully used for weights there is almost no headroom. When headroom is insufficient, the system swaps in and out frequently, or triggers out-of-memory errors under long contexts.
In practice, an asymmetric “weight cards + cache cards” deployment is common. In a 12-GPU configuration, for example, 8 cards hold the weights and 4 cards are dedicated to the KV cache. Using a 288GB card as the example:
# KV cache pool under asymmetric deployment (illustrative)weight_gpus, cache_gpus = 8, 4cache_pool_gb = cache_gpus * 288tokens_per_1m_ctx = 200 # assumption: 200GB cache per 1M-token contextprint(f"Cache pool: {cache_pool_gb}GB")print(f"1M-context concurrency: {cache_pool_gb // tokens_per_1m_ctx}")
This means 8 GPUs is only the lower bound for “it fits,” still short of “it runs well.” In addition, whether the hardware natively supports 4-bit and 8-bit floating-point formats, and how the inter-card interconnect bandwidth performs, both directly affect real throughput.
3.2 Single-Request Latency and Throughput: Two Accounts of MoE Inference
We can estimate roughly from two perspectives: single-request latency and batch throughput.
Single-request latency looks at the “active parameters” that must be fully read once for every token: 104B parameters at 0.5 bytes each is about 52GB. These 52GB must be read in full each time a token is generated, so latency depends mainly on memory bandwidth:
# Single-request theoretical lower bound (illustrative)active_bytes = 104e9 * 0.5 # ~52GBhbm_bw_tbs = 8.0 # assumed memory bandwidth of 8TB/stpot_ms = active_bytes / (hbm_bw_tbs * 1e12) * 1000print(f"Per-token latency lower bound: {tpot_ms:.1f}ms") # ~6.5ms
A per-token latency of about 6.5 milliseconds is only a theoretical lower bound. Whether 4-bit weights can be computed directly and efficiently, whether sparse-attention memory access is regular, and whether cross-card communication can overlap with compute will all push the real value well above the lower bound.
Throughput looks at compute. Each token in the batch activates about 104B parameters on average, and a single forward pass is roughly 0.2×10¹² floating-point operations:
# Batch upper bound under single-GPU theoretical compute (illustrative)flops_per_token = 0.2 * 104e9 # ~2.1e11gpu_tflops = 1000 # assumed single-GPU 1000 TFLOPSmax_batch = gpu_tflops * 1e12 / flops_per_tokenprint(f"Single-GPU theoretical max batch: {max_batch:.0f}") # ~4800
But the larger the batch, the more experts each token hits, and the more weights are actually read. When the batch is large enough that nearly all experts are hit, each token is effectively reading the entire weight set; increasing the batch further barely adds to the read volume. MoE throughput typically peaks near this critical point. More importantly, the larger the batch, the less compute each request gets, so throughput and latency trade off against each other and must be balanced according to business SLAs.
For capacity planning, it is advisable to replace theoretical estimates with load testing on your own workload:
# Capacity planning sketch: rely on measured metricstarget_qps = 20 # target request rateavg_in_tokens = 8000 # average input lengthavg_out_tokens = 500 # average output lengthmeasured_tps_per_gpu = 1500 # measured: single-GPU output token throughput, from your own workloadbilled_ratio = (avg_in_tokens + avg_out_tokens) / avg_out_tokensgpu_count = math.ceil(target_qps * avg_out_tokens * billed_ratio / measured_tps_per_gpu)print(f"Prefill amplification: {billed_ratio:.1f}x, estimated GPUs: {gpu_count}")
Many teams only realize after launch that a request with 8,000 input tokens and just 500 output tokens consumes roughly 17 times the compute of the output tokens. If the business is mainly long-document question answering, the bottleneck is almost certainly in prefill rather than decode, and the hardware scale must be estimated according to this amplification factor.
3.3 The Long-Context KV Cache: The Easiest to Lose Control Of
The 1M-token context is a selling point, but it is also the biggest variable in memory management. The KV cache size can be estimated as follows:
# KV cache magnitude estimate (illustrative; layer count from the model card, rest assumed)layers = 93kv_heads, head_dim, bytes_per_elem = 8, 128, 2 # assumed valuesbytes_per_token = 2 * layers * kv_heads * head_dim * bytes_per_elemprint(f"Cache per token: {bytes_per_token / 1024:.0f}KB") # ~360KBfor ctx in (32_000, 128_000, 1_000_000):print(f"{ctx:>9,} tokens -> {bytes_per_token * ctx / 1024**3:.1f}GB")
That is about 11GB for 32K, about 44GB for 128K, and about 344GB for 1M, growing linearly. This means a single million-token session’s cache alone could fill the entire cache pool of a “weight cards + cache cards” design.
Engineering practice usually controls this from three directions:
- Cache quantization: reduce the KV from 16-bit to 8-bit or lower, directly halving it or more.
- Cache reuse: shared prefixes such as system prompts, tool definitions, and long documents are computed only once and shared across multiple requests.
- On-demand eviction: set eviction policies for sessions approaching the context limit to avoid a long session dragging down the whole service.
3.4 Engineering Essentials: Don’t Spend Bandwidth on the Wrong Things
- Parallelism strategy: expert parallelism lowers the per-card weight read volume but introduces cross-card communication; tensor parallelism communicates frequently and scales inefficiently. Choose based on interconnect bandwidth.
- Chunked prefill: split long inputs into small chunks interleaved with decoding to avoid long requests blocking others.
- Speculative decoding: the gains from a small model on MoE are less stable than on dense models, because the draft model and target model have quite different expert-hit distributions, so measure the acceptance rate before launch.
- Quantization-aware inference: if the framework does not support efficient computation of 4-bit weights, it will dequantize at load time, and the memory advantage disappears.
- Observability: bring memory watermarks, per-card weight read volume, KV cache hit rate, and expert load distribution into monitoring and alerting; many problems only surface here.
3.5 Who Can Run It: From Individual to Enterprise
| Role | Feasibility | Recommended path |
|---|---|---|
| Individual developer | Basically infeasible locally | Use pay-as-you-go high-memory instances on the cloud for validation; use a hosted API for day-to-day development |
| Small and medium teams | Conditionally feasible | Compare self-hosted fixed clusters vs. API call costs; prioritize well-defined coding/agent tasks |
| Medium and large enterprises | Feasible | Start with a small pilot, then plan asymmetric deployment and a cache pool |
On the cloud platform side, general capabilities such as high-memory GPU instances, containerized inference services, object storage, logging services, and monitoring/alerting can be combined. Taking Baidu AI Cloud as an example: model weights can first be placed in object storage, then inference instances can be launched through a container service, combined with monitoring/alerting and pay-as-you-go billing to control costs. If you call a hosted API, then confirm whether the platform has already made the model available, what its SLA is, and how data is isolated.
4. Reading the Numbers: How to Approach Vendor Self-Reported Data
First, the scores the vendor published (all vendor self-reported, citing different evaluation organizations’ naming and conventions):
- Terminal-Bench 4.0: 65.7%, slightly below Claude Opus 5.5’s 66.4%, above GPT-6 Astra’s 58.2%.
- SWE Marathon: 51.7%, above Claude Opus 5.5’s 46.3%.
- Terminal-Bench-Science: 38.1%, far below Claude Opus 5.5’s 63.3%.
- HLE-Diamond: 39.4%, below Claude Opus 5.5’s 60.9%.
So it is not a clean sweep. The vendor’s own claim is that 16 of 37 benchmarks exceed Claude Opus 5.5. Reading across the table, the conclusion is fairly clear: it is competitive on coding and agent tasks but clearly behind on scientific reasoning.
It is reported that the Artificial Analysis Intelligence Index gives it 55, ranking fourth globally.
For self-reported data, three points of caution are advisable:
- Consistent methodology: evaluations from different organizations differ greatly in question selection, prompt templates, and tool-calling methods, so scores across leaderboards cannot be compared directly.
- Agent tasks depend more on scaffolding: the same model paired with different tool sets and prompt strategies can produce very different results. The evaluation environment is not your business environment.
- Plan for weaknesses early: if the business involves scientific computation or rigorous reasoning, prepare mechanisms for human review or routing to other models.
5. License Reading: Open Weights Is Not Open Source
The most important point worth calling out separately in this release is the license. The repository carries a custom Kimi K3.1 Pro License, marked as license:other on the model hosting platform, not a standard open source license like MIT or Apache 2.0. It is identical word-for-word to K3’s license, with only the name changed.
The main body is indeed MIT-style permission: free use, copying, modification, distribution, fine-tuning, and commercial use. But two restrictions are added on top, aimed only at large-scale external business:
- MaaS revenue threshold: companies that offer model-as-a-service externally and whose cumulative revenue over 12 consecutive months exceeds $20 million need to sign a separate agreement with Moonshot AI before commercial use.
- Commercial product MAU threshold: commercial products with over 100 million monthly active users are subject to similar additional constraints.
Read against the original terms, these two constraints target “large-scale external business,” not “whether you use it.” For individual developers, research institutions, and most small and medium enterprises, normal use, modification, distribution, and commercial use are not affected by these two restrictions. But for companies that have already embedded the model in their own product and whose user count or revenue is approaching the threshold, this line must be considered clearly at the architecture design stage, not negotiated after the product is already running.
Two concepts need to be distinguished here:
| Concept | Meaning | This case |
|---|---|---|
| Open source | License meets the open source definition: free use, modification, and redistribution, with no discriminatory restrictions | Does not qualify: revenue and user-count thresholds apply |
| Open weights | Model weights are public, downloadable, runnable, and fine-tunable under the license terms | Qualifies: weights are public, but with additional terms |
The license also typically contains the standard MIT disclaimer and limitation of liability. When it comes to engineering adoption, three things are advisable:
- Bring the license text into the compliance review process, managed alongside the dependency license list.
- Clarify license-attachment obligations when redistributing (such as plugins, all-in-one appliances, and private deployments).
- Watch for version changes; the identical license between generations shows the terms carry forward, so do not assume the next generation will relax them.
6. Who Should Use It, Who Should Wait
Back to the original question: can ordinary people run it? The conclusion is “you can download it, but you can’t run it locally.”
- Worth using: medium and large teams that need coding and agent capabilities, want private deployment, and whose business scale will not hit the two license lines.
- Proceed with caution: businesses involving scientific computation or rigorous reasoning; the self-reported data is clearly weak on these tasks, so validate on a small sample first.
- Can wait: individual developers and small and medium teams; 8 GPUs locally is not a realistic option, and waiting for quantized community versions or a smaller same-architecture model may be more cost-effective.
What is really worth remembering about this release is not “fourth globally” or “16 benchmarks beat a certain model,” but that competition among open-weight models has entered a phase where “scale × license × engineering” pull at the same time. For developers, evaluating whether a model is usable means, beyond looking at scores, accounting for three bills: the memory bill, the throughput bill, and the license bill.
For cloud service providers, whoever can combine general capabilities such as high-memory GPU instances, containerized inference, object storage, and monitoring/alerting into a smooth onboarding path is better positioned to handle this class of “it fits but is hard to run” model demand.
