2.8T Open-Weights Model Local Deployment: Is 8-GPU VRAM Enough?
This article analyzes local deployment of a 2.8T-parameter open-weights model, including its architecture, VRAM requirements, MoE routing, sparse attention, quantization, license constraints, and deployment options. It explains why an 8-GPU server with 192GB per card is a minimum starting point rather than a comfortable operating configuration, and compares self-hosting, cloud GPUs, and hosted APIs.
Introduction: The “landing” questions in the open-weights era
In late October, a model with 2.8 trillion total parameters and an approximately 1.56 TB download size first appeared through an API and web chat. The publisher said it would release the complete weights by November 4 and described it as one of the largest open-weights models at the time. Actual capability should still be judged by independent evaluations and hands-on testing.
On November 4, the weights appeared on major model hosting communities. The repository was publicly downloadable and did not require an application. It is one of the largest open-weights models publicly released to date.
For developers, “being able to call it” and “being able to take it” are different problems. APIs hide model size and compute requirements, while benchmark scores only show an upper bound of capability. What determines deployment feasibility is what is inside the weights, what the license allows, and how much the hardware costs. This article breaks down three questions:
- How are the 2.8T parameters structured, and how does sparse activation change inference cost?
- Is open weights the same as open source, and which license terms must enterprises verify before commercial use?
- What does an 8-GPU starting point mean, and how should teams choose between self-hosting, cloud GPUs, and API calls?
1. Event recap: From “callable” to “obtainable”
- Late October: The model first appeared through an API and web interface. The publisher promised to release full weights before November 4.
- November 2: A weight repository was created on a major model hosting community, but it remained closed at that time.
- November 4: The repository was still closed in the morning and opened that evening. It contained 112 files, 96 shards, and approximately 1.56 TB in total size. No application was required for download.
This article does not repeat the leaderboard results announced at launch. It focuses on engineering issues after the weights become available: what is inside them, how the license applies, and whether they can run.
2. Unpacking 2.8T: How sparse activation changes the inference shape
| Item | Parameter or detail | Engineering implication |
|---|---|---|
| Model type | Native multimodal agent model, further trained on the previous-generation base | Supports image understanding and tool calling; a multimodal inference runtime is required |
| Total parameters | 2.8T | Determines VRAM usage |
| Activated per inference | 104B, about 3.7% of total parameters | Determines compute consumption |
| Expert structure | 896 routed experts, 16 selected per token, plus 2 shared experts | Routing layers and all-to-all communication are performance hotspots |
| Layers | 93 | KV cache accumulates layer by layer |
| Context length | 1,000,000 tokens | Suitable for long documents and long-horizon agent tasks |
| Vision encoder | MoonViT-V2 | Images participate in inference as additional tokens |
| Structural difference from previous generation | 24 layers of full attention replaced with sparse attention (DSA) | Reduces compute and memory-bandwidth pressure for long contexts |
| Weight precision | 4-bit-level floating point, MXFP4-like format | Size is about one quarter of the equivalent BF16 weights, roughly 25% |
| Activation precision | 8-bit-level floating point, MXFP8-like format | Matrix multiplications execute in low precision |
| Other versions | Not released | There are no BF16 or FP8 versions to choose from |
2.1 Total parameters and activated parameters: Two numbers that must be viewed separately
Mixture-of-experts models often publish both total parameter count and activated parameter count. The former determines how much VRAM the model occupies, while the latter determines how much compute is needed to generate each token.
For this model, only about 3.7% of the parameters are computed for each token, which substantially reduces compute demand. However, inactive expert weights still remain resident in VRAM. The storage advantage does not shrink proportionally.
Total parameters determine whether the model can fit. Activated parameters determine how fast it can run. Both are required when estimating deployment cost.
2.2 Expert routing: The key mechanism behind the shift from dense to sparse
When tokens pass through the routing layer, the model assigns scores to all experts. It then selects the 16 highest-scoring experts to perform computation, in addition to 2 shared experts that always participate.
Routed experts provide specialization, carrying capabilities associated with different domains and patterns. Shared experts handle common computation, similar to the feed-forward layers in a traditional dense model, allowing routed experts to focus on differentiated representations.
For engineering implementation, this mechanism has two direct consequences:
- No single GPU can hold all 896 experts. Expert parallelism must distribute experts across GPUs.
- Every time a batch of tokens is processed, GPUs must exchange data. Communication volume is directly related to how many GPUs each token is sent to. A high-bandwidth all-to-all interconnect is therefore a basic requirement.
2.3 Low-bit quantization-aware training: Why there is no BF16 version
The model uses 4-bit-level floating-point weights, such as an MXFP4-like format, and 8-bit-level floating-point activations, such as an MXFP8-like format. These are products of quantization-aware training.
QAT incorporates quantization error into the training objective, which usually produces better results than compressing a model after training. Compared with post-training quantization models, QAT models can be used more confidently with low-bit weights directly. This is one reason the publisher provides only one version.
However, this also introduces constraints:
- The inference framework must natively support the low-bit format and its associated block-scaling metadata.
- If the framework only supports conventional low-bit formats, the weights must first be dequantized back to 16-bit floating point. VRAM usage becomes roughly four times larger and should be estimated at four times the original size. For example, 1.56 TB becomes roughly 6.2 TB after dequantization to 16-bit floating point. Estimating with 192GB per GPU, more than thirty GPUs would be required, before adding sharding, runtime, and communication overhead.
- Fine-tuning generally requires dequantizing the weights and then adding low-rank adapters such as LoRA.
2.4 Sparse attention: Reducing the burden of million-token contexts
The model has 93 layers. In 24 of them, full attention is replaced by sparse attention.
Full-attention computation and KV reads grow quadratically with sequence length. With a 1,000,000-token context, this structure would be extremely expensive. Sparse attention first performs coarse filtering over historical tokens and allows each query to attend only to the most relevant few tokens, significantly reducing long-context computation.
Two engineering points are important:
- Sparse attention directly saves attention compute and memory reads. It does not automatically reduce the storage size of the KV cache. How much the KV cache can be reduced depends on whether the implementation also uses grouped-query attention to compress the number of heads or further compresses keys and values in a low-rank manner.
- Sparse attention and grouped-query attention solve different problems and cannot replace each other. Sparse attention reduces how much is read at each step. Grouped-query attention reduces how much is stored overall. They are often used together.
3. From 1.56 TB and 8 GPUs: A VRAM budget walkthrough
3.1 Back-calculating size: About 4.46 bits per parameter
Dividing 1.56 TB by 2.8 trillion parameters gives an average of about 4.46 bits per parameter, consistent with a pattern of “mostly 4-bit weights plus some modules kept at higher precision”:
- At 0.5 bytes, or 4 bits, the main body of the 2.8T parameters is roughly 1.4 TB.
- The remaining roughly 0.16 TB comes from parts that are usually not compressed to 4 bits, such as the vision encoder, routing layers, and normalization layers, plus metadata such as scaling factors for each quantization block.
- Based on experience, this kind of overhead typically makes the actual size 5% to 15% larger than the pure 4-bit theoretical value. If the proportion clearly exceeds 15%, it is reasonable to suspect that some weights remain at higher precision.
| Item | Estimated value | Notes |
|---|---|---|
| Total parameters | 2.8T | Published by the vendor |
| Activated parameters | 104B | Published by the vendor |
| Theoretical size of 4-bit weights | About 1.4 TB | 2.8T × 0.5 bytes |
| Actual download size | About 1.56 TB | Includes scaling factors and non-quantized modules |
| Total HBM capacity of 8 GPUs at 192GB each | 1.536 TB | 8 × 192GB |
| Weight load per GPU | About 195GB | 1.56 TB ÷ 8 |
| Physical headroom | About 0 | Sharding granularity, metadata, and alignment overhead further reduce available space |
3.2 The implicit premise of “8 GPUs”
The calculation above points to one conclusion: “8 GPUs” refers to 8 accelerator cards with about 192GB of VRAM per card and high-bandwidth interconnect between cards, which typically means a single 8-GPU server.
If cards with 96GB per GPU are used instead, 16 GPUs would barely be enough to hold the weights. If they cannot fit in one machine, tensor parallelism or expert parallelism must span machines. Cross-machine network bandwidth is often only a fraction, sometimes one tenth, of intra-machine interconnect bandwidth, causing latency to rise significantly.
Therefore, “8 GPUs to start” should not be understood as “any 8 GPUs.” It is effectively the threshold of a high-end server.
3.3 VRAM budget: Fitting the model is only the starting point
Within per-GPU VRAM, weights are only one component:
# Rough VRAM budget calculation: numbers are assumptions for demonstrationWEIGHTS_TB = 1.56 # Model weights, based on actual download size# KV cache = 2 (K and V) x layers x KV heads x head dim x bytes per element x tokenskv_cache_gb = 2 * 93 * 8 * 128 * 1 * 1_000_000 / 1e9 # Assumes 8 GQA groups and FP8 storagekv_cache_tb = kv_cache_gb / 1000runtime_gb = 24 # Inference framework, communication buffers, temporary tensors, etc.total_tb = WEIGHTS_TB + kv_cache_tb + runtime_gb / 1000print(f"Weights {WEIGHTS_TB} TB + KV cache {kv_cache_tb:.2f} TB + runtime {runtime_gb} GB")print(f"Total approx {total_tb:.2f} TB; 8 x 192GB = 1.536 TB")print("Conclusion: with long contexts, KV quantization or tiered offload and lower concurrency are required")
Output:
Weights 1.56 TB + KV cache 0.19 TB + runtime 24 GBTotal approx 1.77 TB; 8 x 192GB = 1.536 TBConclusion: with long contexts, KV quantization or tiered offload and lower concurrency are required
As shown, the weights alone already consume essentially all available VRAM. Even with relatively optimistic KV-cache compression assumptions, a 1,000,000-token KV cache far exceeds the physical headroom.
In real deployments, teams must make tradeoffs among the following options:
- Reduce concurrency, which usually means supporting only a very small number of simultaneous requests.
- Quantize and compress the KV cache, for example from 16-bit to 8-bit or 4-bit.
- Enable tiered caching by evicting inactive KV blocks to host memory or local storage.
- Strictly limit the context length of each request.
This also explains why “the model is open” and “everyone can run it” are still far apart. The inference framework can keep only the weights themselves resident in VRAM. Long contexts and high concurrency are a separate accounting problem.
3.4 Throughput estimation: MoE saves compute, not bandwidth
Single-request token-by-token generation is a typical memory-bandwidth-bound scenario. For each generated token, a MoE model must read all activated 104B parameters. At 4 bits, that is about 52GB.
# Upper-bound estimate for single-request decoding speedactive_params = 104e9 # Activated parameters per tokenbytes_per_param = 0.5 # 4-bitcards = 8card_bandwidth = 3.35e12 # Assumed VRAM bandwidth per card: 3.35 TB/sideal_tps = cards * card_bandwidth / (active_params * bytes_per_param)realistic_tps = ideal_tps * 0.4 # 40% effective bandwidth after communication and routing lossesprint(f"Ideal upper bound approx {ideal_tps:.0f} tokens/s")print(f"Realistic estimate approx {realistic_tps:.0f} tokens/s")
Output:
Ideal upper bound approx 515 tokens/sRealistic estimate approx 206 tokens/s
This result is based on assumptions of 3.35 TB/s per-card bandwidth, 4-bit activated weights, and 8 GPUs in one machine. It is not measured performance and should only be used as an order-of-magnitude reference.
The number of weights that must be read per token is related to concurrency. The set of experts touched by a single batch depends on token count, routing distribution, and concurrency, and can reach at most 896 experts. At low concurrency, insufficient expert reuse reduces bandwidth utilization. As concurrency increases, the weight-read cost amortized per token decreases and per-GPU throughput rises.
The advantage of MoE is precisely lower serving cost under high concurrency. However, an “8-GPU starting point” also means that idle capacity can be very expensive.
4. How to read evaluations: Vendor-reported data and independent rankings
The publisher released several vendor-reported comparison results. Strengths and weaknesses are both clear:
| Capability | This model, self-reported | Reference closed-source flagship, self-reported |
|---|---|---|
| Terminal task execution | 65.7% | 66.4% |
| Repository-level software engineering tasks | 51.7% | 46.3% |
| Scientific tasks | 38.1% | 63.3% |
| Difficult comprehensive reasoning | 39.4% | 60.9% |
Before using these numbers, record the evaluation date, sample size, pass@1 or task-completion metric, prompt templates, sampling parameters, and scoring scripts. If this information is missing, treat the numbers only as directional references rather than direct procurement evidence.
Three observations:
- Coding and agent capabilities are strengths. The model leads by about 5 percentage points on repository-level software engineering tasks and is nearly tied on terminal tasks. This is the most relevant signal for CI/CD pipelines, automated operations, and coding assistants.
- Scientific tasks and difficult reasoning are weaknesses. Both are more than 20 percentage points below the comparison model. Scientific assistance and complex reasoning applications should be evaluated cautiously.
- The overall score is not uniformly leading. According to the official statement, the model leads on 16 of 37 benchmarks. Some independent evaluations place it in the top tier, but it is not the strongest on every item.
Vendor-reported data can be heavily affected by prompts, sampling strategies, and scoring methods. At minimum, model selection should include the following:
- Re-test with your own business samples, prioritizing first-pass success rate and full-task completion rate rather than only per-question accuracy.
- For agent scenarios, separately measure tool-call success rate, multi-step task completion rate, and average number of steps.
- For long-context scenarios, add needle-in-a-haystack retrieval tests, such as hiding key information in the middle of a document containing hundreds of thousands of characters.
- Cross-check vendor-reported data with independent rankings. Do not directly compare numbers from different sources as if they were measured under the same conditions.
# Example: weighted scoring by business dimensiondimensions = {"Code generation": {"this_model": 0.66, "closed_flagship": 0.64, "weight": 0.30},"Software engineering": {"this_model": 0.52, "closed_flagship": 0.46, "weight": 0.25},"Tool calling": {"this_model": 0.80, "closed_flagship": 0.82, "weight": 0.20}, # illustrative"Scientific reasoning": {"this_model": 0.38, "closed_flagship": 0.63, "weight": 0.15},"Long-document retrieval":{"this_model": 0.90, "closed_flagship": 0.93, "weight": 0.10}, # illustrative}for name in ("this_model", "closed_flagship"):score = sum(d[name] * d["weight"] for d in dimensions.values())print(f"{name} weighted score: {score:.3f}")
5. License: Open weights does not equal open source
5.1 What a custom license means
The repository is attached with a custom license named “Kimi K3.1 Pro License.” On the community page, it is labeled as an “other” license rather than a standard open-source license such as MIT or Apache-2.0.
Standard licenses that have been reviewed by legal teams and can be automatically recognized by tools are easier to procure and redistribute. Custom licenses require manual review clause by clause.
This is not an isolated case. In recent years, major model publishers releasing open weights have commonly used custom licenses of the form “standard grant plus additional terms.” Such texts often include some of the following clauses:
- Use-purpose restrictions, such as prohibiting military use or prohibiting the generation of unlawful content.
- Attribution requirements, such as stating in the product that it is “based on a certain model.”
- Scale thresholds, such as prominently displaying the model name in the interface when monthly active users exceed a certain level.
- Redistribution and derivative-work terms, such as whether reselling the model is allowed and whether outputs can be used to train other models.
The publisher’s definition of “open” is also not identical to the strict OSI definition. Whether something is “open” cannot be judged only by whether weights can be downloaded. It also depends on whether the model can be freely used, modified, and redistributed.
For teams subject to local compliance requirements, additional attention should be paid to mirror distribution, data export, content safety, and jurisdiction-specific regulations.
5.2 Enterprise compliance checklist
Before review, use the LICENSE file at the repository root, the model card, and the publisher’s announcements as the source of truth, and preserve version snapshots.
# License review checklist example (simulated checks; does not parse a real license)LICENSE = {"commercial_use": "allowed", # Commercial use permitted"modification": "allowed", # Modification/fine-tuning permitted"redistribution": "allowed", # Redistribution of weights permitted"attribution": "required", # Attribution required"use_restrictions": ["prohibited_unlawful_use"], # Use restrictions exist}def review(license_text: dict, scenario: str) -> list[str]:risks = []if license_text["commercial_use"] != "allowed":risks.append(f"[{scenario}] Commercial use is not allowed; separate permission is required")if scenario in ("saas_service", "on_premise_delivery"):if license_text["redistribution"] != "allowed":risks.append(f"[{scenario}] Redistribution is restricted; confirm whether the delivery model complies")if license_text["attribution"] == "required":risks.append(f"[{scenario}] Model source must be indicated in the product or documentation")if license_text["use_restrictions"]:risks.append(f"[{scenario}] Business use must be reviewed against the restriction clauses")return risksfor scene in ("internal_tool", "saas_service"):for item in review(LICENSE, scene):print(item)
Output:
[internal_tool] Model source must be indicated in the product or documentation[internal_tool] Business use must be reviewed against the restriction clauses[saas_service] Model source must be indicated in the product or documentation[saas_service] Business use must be reviewed against the restriction clauses
In practice, the following items should be confirmed one by one:
- Is commercial use free of charge, and does it include attribution or scale thresholds?
- Is redistribution restricted, and does private deployment count as “distribution”?
- Can a new model fine-tuned from the model’s outputs be released again?
- Does the prohibited-use list conflict with the business scenario, such as public-facing medical or financial advice?
- Is the license version tied to a specific weight version, and does upgrading the model require a new review?
- Does the license text change with weight versions, and is a re-review mechanism required?
6. Deployment choice: Self-hosted, cloud, or API
| Dimension | Self-hosted, 8-GPU server | Cloud GPU rental | Hosted API |
|---|---|---|---|
| Minimum hardware | One server with 8 192GB-class accelerators | Rent the same specification on demand | No hardware required |
| Data residency | Data stays within the private network | Depends on dedicated or shared deployment | Data must be sent to the service provider |
| Time to launch | Weeks or months for procurement and deployment | Hour-level provisioning | Minute-level integration |
| Operational burden | Drivers, frameworks, and VRAM tuning are all handled in-house | Inference operations are still required | Handled by the service provider |
| Cost profile | High upfront cost, lowest unit cost at full load | Pay-as-you-go, suitable for peaks and troughs | Per-token pricing, best elasticity |
When choosing, ask the following questions in order:
- Can the data leave the internal network? If not, prioritize self-hosting or a dedicated cloud deployment.
- Does the team have distributed inference operations capability? If not, prioritize cloud GPUs or hosted APIs.
- Is request volume stable over the long term? If yes, self-hosting has an advantage. If not, pay-as-you-go is more economical.
- Is fine-tuning required? If yes, reserve additional VRAM and storage resources for LoRA training and weight storage.
If the cloud route is selected, a typical engineering combination includes:
- Object storage: store model weights and datasets, and maintain version control.
- Container services: run inference images and enable elastic scaling.
- Logging and monitoring services: track first-token latency, per-token latency, VRAM usage, and other metrics.
- Message queues: smooth peaks and valleys, and balance bursty inference request traffic.
A hybrid architecture is also possible: offline batch jobs can run on a self-managed cluster, while elastic peaks are handled by cloud GPUs or hosted APIs. When planning such deployments on mainstream cloud platforms or self-managed environments, the VRAM budget and license review checklist above still apply.
7. Summary: Three core conclusions
- How much VRAM does a 2.8T model require? The 1.56 TB of weights alone is already close to the theoretical capacity of 8 × 192GB. Long contexts and high concurrency require additional VRAM. Eight GPUs are only the starting line, not a comfortable operating line.
- Does open weights equal open source? No. Although the custom license preserves freedom for commercial use and fine-tuning, attribution, use restrictions, and redistribution terms must be reviewed clause by clause by legal teams, and license version snapshots must be preserved.
- Are 8 GPUs enough, and should you self-host? For low concurrency, short contexts, and controllable tasks, self-hosting or a dedicated cloud deployment can be tried. For high concurrency, long contexts, and multi-tenant services, a new VRAM and bandwidth budget is required, or a hosted API should first be used to validate business value.
