2.8T Open-Weights Model Local Deployment: Is 8-GPU VRAM Enough?

    This article analyzes local deployment of a 2.8T-parameter open-weights model, including its architecture, VRAM requirements, MoE routing, sparse attention, quantization, license constraints, and deployment options. It explains why an 8-GPU server with 192GB per card is a minimum starting point rather than a comfortable operating configuration, and compares self-hosting, cloud GPUs, and hosted APIs.

    Introduction: The “landing” questions in the open-weights era

    In late October, a model with 2.8 trillion total parameters and an approximately 1.56 TB download size first appeared through an API and web chat. The publisher said it would release the complete weights by November 4 and described it as one of the largest open-weights models at the time. Actual capability should still be judged by independent evaluations and hands-on testing.

    On November 4, the weights appeared on major model hosting communities. The repository was publicly downloadable and did not require an application. It is one of the largest open-weights models publicly released to date.

    For developers, “being able to call it” and “being able to take it” are different problems. APIs hide model size and compute requirements, while benchmark scores only show an upper bound of capability. What determines deployment feasibility is what is inside the weights, what the license allows, and how much the hardware costs. This article breaks down three questions:

    • How are the 2.8T parameters structured, and how does sparse activation change inference cost?
    • Is open weights the same as open source, and which license terms must enterprises verify before commercial use?
    • What does an 8-GPU starting point mean, and how should teams choose between self-hosting, cloud GPUs, and API calls?

    1. Event recap: From “callable” to “obtainable”

    • Late October: The model first appeared through an API and web interface. The publisher promised to release full weights before November 4.
    • November 2: A weight repository was created on a major model hosting community, but it remained closed at that time.
    • November 4: The repository was still closed in the morning and opened that evening. It contained 112 files, 96 shards, and approximately 1.56 TB in total size. No application was required for download.

    This article does not repeat the leaderboard results announced at launch. It focuses on engineering issues after the weights become available: what is inside them, how the license applies, and whether they can run.

    2. Unpacking 2.8T: How sparse activation changes the inference shape

    Item Parameter or detail Engineering implication
    Model type Native multimodal agent model, further trained on the previous-generation base Supports image understanding and tool calling; a multimodal inference runtime is required
    Total parameters 2.8T Determines VRAM usage
    Activated per inference 104B, about 3.7% of total parameters Determines compute consumption
    Expert structure 896 routed experts, 16 selected per token, plus 2 shared experts Routing layers and all-to-all communication are performance hotspots
    Layers 93 KV cache accumulates layer by layer
    Context length 1,000,000 tokens Suitable for long documents and long-horizon agent tasks
    Vision encoder MoonViT-V2 Images participate in inference as additional tokens
    Structural difference from previous generation 24 layers of full attention replaced with sparse attention (DSA) Reduces compute and memory-bandwidth pressure for long contexts
    Weight precision 4-bit-level floating point, MXFP4-like format Size is about one quarter of the equivalent BF16 weights, roughly 25%
    Activation precision 8-bit-level floating point, MXFP8-like format Matrix multiplications execute in low precision
    Other versions Not released There are no BF16 or FP8 versions to choose from

    2.1 Total parameters and activated parameters: Two numbers that must be viewed separately

    Mixture-of-experts models often publish both total parameter count and activated parameter count. The former determines how much VRAM the model occupies, while the latter determines how much compute is needed to generate each token.

    For this model, only about 3.7% of the parameters are computed for each token, which substantially reduces compute demand. However, inactive expert weights still remain resident in VRAM. The storage advantage does not shrink proportionally.

    Total parameters determine whether the model can fit. Activated parameters determine how fast it can run. Both are required when estimating deployment cost.

    2.2 Expert routing: The key mechanism behind the shift from dense to sparse

    When tokens pass through the routing layer, the model assigns scores to all experts. It then selects the 16 highest-scoring experts to perform computation, in addition to 2 shared experts that always participate.

    Routed experts provide specialization, carrying capabilities associated with different domains and patterns. Shared experts handle common computation, similar to the feed-forward layers in a traditional dense model, allowing routed experts to focus on differentiated representations.

    For engineering implementation, this mechanism has two direct consequences:

    • No single GPU can hold all 896 experts. Expert parallelism must distribute experts across GPUs.
    • Every time a batch of tokens is processed, GPUs must exchange data. Communication volume is directly related to how many GPUs each token is sent to. A high-bandwidth all-to-all interconnect is therefore a basic requirement.

    2.3 Low-bit quantization-aware training: Why there is no BF16 version

    The model uses 4-bit-level floating-point weights, such as an MXFP4-like format, and 8-bit-level floating-point activations, such as an MXFP8-like format. These are products of quantization-aware training.

    QAT incorporates quantization error into the training objective, which usually produces better results than compressing a model after training. Compared with post-training quantization models, QAT models can be used more confidently with low-bit weights directly. This is one reason the publisher provides only one version.

    However, this also introduces constraints:

    • The inference framework must natively support the low-bit format and its associated block-scaling metadata.
    • If the framework only supports conventional low-bit formats, the weights must first be dequantized back to 16-bit floating point. VRAM usage becomes roughly four times larger and should be estimated at four times the original size. For example, 1.56 TB becomes roughly 6.2 TB after dequantization to 16-bit floating point. Estimating with 192GB per GPU, more than thirty GPUs would be required, before adding sharding, runtime, and communication overhead.
    • Fine-tuning generally requires dequantizing the weights and then adding low-rank adapters such as LoRA.

    2.4 Sparse attention: Reducing the burden of million-token contexts

    The model has 93 layers. In 24 of them, full attention is replaced by sparse attention.

    Full-attention computation and KV reads grow quadratically with sequence length. With a 1,000,000-token context, this structure would be extremely expensive. Sparse attention first performs coarse filtering over historical tokens and allows each query to attend only to the most relevant few tokens, significantly reducing long-context computation.

    Two engineering points are important:

    • Sparse attention directly saves attention compute and memory reads. It does not automatically reduce the storage size of the KV cache. How much the KV cache can be reduced depends on whether the implementation also uses grouped-query attention to compress the number of heads or further compresses keys and values in a low-rank manner.
    • Sparse attention and grouped-query attention solve different problems and cannot replace each other. Sparse attention reduces how much is read at each step. Grouped-query attention reduces how much is stored overall. They are often used together.

    3. From 1.56 TB and 8 GPUs: A VRAM budget walkthrough

    3.1 Back-calculating size: About 4.46 bits per parameter

    Dividing 1.56 TB by 2.8 trillion parameters gives an average of about 4.46 bits per parameter, consistent with a pattern of “mostly 4-bit weights plus some modules kept at higher precision”:

    • At 0.5 bytes, or 4 bits, the main body of the 2.8T parameters is roughly 1.4 TB.
    • The remaining roughly 0.16 TB comes from parts that are usually not compressed to 4 bits, such as the vision encoder, routing layers, and normalization layers, plus metadata such as scaling factors for each quantization block.
    • Based on experience, this kind of overhead typically makes the actual size 5% to 15% larger than the pure 4-bit theoretical value. If the proportion clearly exceeds 15%, it is reasonable to suspect that some weights remain at higher precision.
    Item Estimated value Notes
    Total parameters 2.8T Published by the vendor
    Activated parameters 104B Published by the vendor
    Theoretical size of 4-bit weights About 1.4 TB 2.8T × 0.5 bytes
    Actual download size About 1.56 TB Includes scaling factors and non-quantized modules
    Total HBM capacity of 8 GPUs at 192GB each 1.536 TB 8 × 192GB
    Weight load per GPU About 195GB 1.56 TB ÷ 8
    Physical headroom About 0 Sharding granularity, metadata, and alignment overhead further reduce available space

    3.2 The implicit premise of “8 GPUs”

    The calculation above points to one conclusion: “8 GPUs” refers to 8 accelerator cards with about 192GB of VRAM per card and high-bandwidth interconnect between cards, which typically means a single 8-GPU server.

    If cards with 96GB per GPU are used instead, 16 GPUs would barely be enough to hold the weights. If they cannot fit in one machine, tensor parallelism or expert parallelism must span machines. Cross-machine network bandwidth is often only a fraction, sometimes one tenth, of intra-machine interconnect bandwidth, causing latency to rise significantly.

    Therefore, “8 GPUs to start” should not be understood as “any 8 GPUs.” It is effectively the threshold of a high-end server.

    3.3 VRAM budget: Fitting the model is only the starting point

    Within per-GPU VRAM, weights are only one component:

    1. # Rough VRAM budget calculation: numbers are assumptions for demonstration
    2. WEIGHTS_TB = 1.56 # Model weights, based on actual download size
    3. # KV cache = 2 (K and V) x layers x KV heads x head dim x bytes per element x tokens
    4. kv_cache_gb = 2 * 93 * 8 * 128 * 1 * 1_000_000 / 1e9 # Assumes 8 GQA groups and FP8 storage
    5. kv_cache_tb = kv_cache_gb / 1000
    6. runtime_gb = 24 # Inference framework, communication buffers, temporary tensors, etc.
    7. total_tb = WEIGHTS_TB + kv_cache_tb + runtime_gb / 1000
    8. print(f"Weights {WEIGHTS_TB} TB + KV cache {kv_cache_tb:.2f} TB + runtime {runtime_gb} GB")
    9. print(f"Total approx {total_tb:.2f} TB; 8 x 192GB = 1.536 TB")
    10. print("Conclusion: with long contexts, KV quantization or tiered offload and lower concurrency are required")

    Output:

    1. Weights 1.56 TB + KV cache 0.19 TB + runtime 24 GB
    2. Total approx 1.77 TB; 8 x 192GB = 1.536 TB
    3. Conclusion: with long contexts, KV quantization or tiered offload and lower concurrency are required

    As shown, the weights alone already consume essentially all available VRAM. Even with relatively optimistic KV-cache compression assumptions, a 1,000,000-token KV cache far exceeds the physical headroom.

    In real deployments, teams must make tradeoffs among the following options:

    • Reduce concurrency, which usually means supporting only a very small number of simultaneous requests.
    • Quantize and compress the KV cache, for example from 16-bit to 8-bit or 4-bit.
    • Enable tiered caching by evicting inactive KV blocks to host memory or local storage.
    • Strictly limit the context length of each request.

    This also explains why “the model is open” and “everyone can run it” are still far apart. The inference framework can keep only the weights themselves resident in VRAM. Long contexts and high concurrency are a separate accounting problem.

    3.4 Throughput estimation: MoE saves compute, not bandwidth

    Single-request token-by-token generation is a typical memory-bandwidth-bound scenario. For each generated token, a MoE model must read all activated 104B parameters. At 4 bits, that is about 52GB.

    1. # Upper-bound estimate for single-request decoding speed
    2. active_params = 104e9 # Activated parameters per token
    3. bytes_per_param = 0.5 # 4-bit
    4. cards = 8
    5. card_bandwidth = 3.35e12 # Assumed VRAM bandwidth per card: 3.35 TB/s
    6. ideal_tps = cards * card_bandwidth / (active_params * bytes_per_param)
    7. realistic_tps = ideal_tps * 0.4 # 40% effective bandwidth after communication and routing losses
    8. print(f"Ideal upper bound approx {ideal_tps:.0f} tokens/s")
    9. print(f"Realistic estimate approx {realistic_tps:.0f} tokens/s")

    Output:

    1. Ideal upper bound approx 515 tokens/s
    2. Realistic estimate approx 206 tokens/s

    This result is based on assumptions of 3.35 TB/s per-card bandwidth, 4-bit activated weights, and 8 GPUs in one machine. It is not measured performance and should only be used as an order-of-magnitude reference.

    The number of weights that must be read per token is related to concurrency. The set of experts touched by a single batch depends on token count, routing distribution, and concurrency, and can reach at most 896 experts. At low concurrency, insufficient expert reuse reduces bandwidth utilization. As concurrency increases, the weight-read cost amortized per token decreases and per-GPU throughput rises.

    The advantage of MoE is precisely lower serving cost under high concurrency. However, an “8-GPU starting point” also means that idle capacity can be very expensive.

    4. How to read evaluations: Vendor-reported data and independent rankings

    The publisher released several vendor-reported comparison results. Strengths and weaknesses are both clear:

    Capability This model, self-reported Reference closed-source flagship, self-reported
    Terminal task execution 65.7% 66.4%
    Repository-level software engineering tasks 51.7% 46.3%
    Scientific tasks 38.1% 63.3%
    Difficult comprehensive reasoning 39.4% 60.9%

    Before using these numbers, record the evaluation date, sample size, pass@1 or task-completion metric, prompt templates, sampling parameters, and scoring scripts. If this information is missing, treat the numbers only as directional references rather than direct procurement evidence.

    Three observations:

    • Coding and agent capabilities are strengths. The model leads by about 5 percentage points on repository-level software engineering tasks and is nearly tied on terminal tasks. This is the most relevant signal for CI/CD pipelines, automated operations, and coding assistants.
    • Scientific tasks and difficult reasoning are weaknesses. Both are more than 20 percentage points below the comparison model. Scientific assistance and complex reasoning applications should be evaluated cautiously.
    • The overall score is not uniformly leading. According to the official statement, the model leads on 16 of 37 benchmarks. Some independent evaluations place it in the top tier, but it is not the strongest on every item.

    Vendor-reported data can be heavily affected by prompts, sampling strategies, and scoring methods. At minimum, model selection should include the following:

    • Re-test with your own business samples, prioritizing first-pass success rate and full-task completion rate rather than only per-question accuracy.
    • For agent scenarios, separately measure tool-call success rate, multi-step task completion rate, and average number of steps.
    • For long-context scenarios, add needle-in-a-haystack retrieval tests, such as hiding key information in the middle of a document containing hundreds of thousands of characters.
    • Cross-check vendor-reported data with independent rankings. Do not directly compare numbers from different sources as if they were measured under the same conditions.
    1. # Example: weighted scoring by business dimension
    2. dimensions = {
    3. "Code generation": {"this_model": 0.66, "closed_flagship": 0.64, "weight": 0.30},
    4. "Software engineering": {"this_model": 0.52, "closed_flagship": 0.46, "weight": 0.25},
    5. "Tool calling": {"this_model": 0.80, "closed_flagship": 0.82, "weight": 0.20}, # illustrative
    6. "Scientific reasoning": {"this_model": 0.38, "closed_flagship": 0.63, "weight": 0.15},
    7. "Long-document retrieval":{"this_model": 0.90, "closed_flagship": 0.93, "weight": 0.10}, # illustrative
    8. }
    9. for name in ("this_model", "closed_flagship"):
    10. score = sum(d[name] * d["weight"] for d in dimensions.values())
    11. print(f"{name} weighted score: {score:.3f}")

    5. License: Open weights does not equal open source

    5.1 What a custom license means

    The repository is attached with a custom license named “Kimi K3.1 Pro License.” On the community page, it is labeled as an “other” license rather than a standard open-source license such as MIT or Apache-2.0.

    Standard licenses that have been reviewed by legal teams and can be automatically recognized by tools are easier to procure and redistribute. Custom licenses require manual review clause by clause.

    This is not an isolated case. In recent years, major model publishers releasing open weights have commonly used custom licenses of the form “standard grant plus additional terms.” Such texts often include some of the following clauses:

    • Use-purpose restrictions, such as prohibiting military use or prohibiting the generation of unlawful content.
    • Attribution requirements, such as stating in the product that it is “based on a certain model.”
    • Scale thresholds, such as prominently displaying the model name in the interface when monthly active users exceed a certain level.
    • Redistribution and derivative-work terms, such as whether reselling the model is allowed and whether outputs can be used to train other models.

    The publisher’s definition of “open” is also not identical to the strict OSI definition. Whether something is “open” cannot be judged only by whether weights can be downloaded. It also depends on whether the model can be freely used, modified, and redistributed.

    For teams subject to local compliance requirements, additional attention should be paid to mirror distribution, data export, content safety, and jurisdiction-specific regulations.

    5.2 Enterprise compliance checklist

    Before review, use the LICENSE file at the repository root, the model card, and the publisher’s announcements as the source of truth, and preserve version snapshots.

    1. # License review checklist example (simulated checks; does not parse a real license)
    2. LICENSE = {
    3. "commercial_use": "allowed", # Commercial use permitted
    4. "modification": "allowed", # Modification/fine-tuning permitted
    5. "redistribution": "allowed", # Redistribution of weights permitted
    6. "attribution": "required", # Attribution required
    7. "use_restrictions": ["prohibited_unlawful_use"], # Use restrictions exist
    8. }
    9. def review(license_text: dict, scenario: str) -> list[str]:
    10. risks = []
    11. if license_text["commercial_use"] != "allowed":
    12. risks.append(f"[{scenario}] Commercial use is not allowed; separate permission is required")
    13. if scenario in ("saas_service", "on_premise_delivery"):
    14. if license_text["redistribution"] != "allowed":
    15. risks.append(f"[{scenario}] Redistribution is restricted; confirm whether the delivery model complies")
    16. if license_text["attribution"] == "required":
    17. risks.append(f"[{scenario}] Model source must be indicated in the product or documentation")
    18. if license_text["use_restrictions"]:
    19. risks.append(f"[{scenario}] Business use must be reviewed against the restriction clauses")
    20. return risks
    21. for scene in ("internal_tool", "saas_service"):
    22. for item in review(LICENSE, scene):
    23. print(item)

    Output:

    1. [internal_tool] Model source must be indicated in the product or documentation
    2. [internal_tool] Business use must be reviewed against the restriction clauses
    3. [saas_service] Model source must be indicated in the product or documentation
    4. [saas_service] Business use must be reviewed against the restriction clauses

    In practice, the following items should be confirmed one by one:

    1. Is commercial use free of charge, and does it include attribution or scale thresholds?
    2. Is redistribution restricted, and does private deployment count as “distribution”?
    3. Can a new model fine-tuned from the model’s outputs be released again?
    4. Does the prohibited-use list conflict with the business scenario, such as public-facing medical or financial advice?
    5. Is the license version tied to a specific weight version, and does upgrading the model require a new review?
    6. Does the license text change with weight versions, and is a re-review mechanism required?

    6. Deployment choice: Self-hosted, cloud, or API

    Dimension Self-hosted, 8-GPU server Cloud GPU rental Hosted API
    Minimum hardware One server with 8 192GB-class accelerators Rent the same specification on demand No hardware required
    Data residency Data stays within the private network Depends on dedicated or shared deployment Data must be sent to the service provider
    Time to launch Weeks or months for procurement and deployment Hour-level provisioning Minute-level integration
    Operational burden Drivers, frameworks, and VRAM tuning are all handled in-house Inference operations are still required Handled by the service provider
    Cost profile High upfront cost, lowest unit cost at full load Pay-as-you-go, suitable for peaks and troughs Per-token pricing, best elasticity

    When choosing, ask the following questions in order:

    1. Can the data leave the internal network? If not, prioritize self-hosting or a dedicated cloud deployment.
    2. Does the team have distributed inference operations capability? If not, prioritize cloud GPUs or hosted APIs.
    3. Is request volume stable over the long term? If yes, self-hosting has an advantage. If not, pay-as-you-go is more economical.
    4. Is fine-tuning required? If yes, reserve additional VRAM and storage resources for LoRA training and weight storage.

    If the cloud route is selected, a typical engineering combination includes:

    • Object storage: store model weights and datasets, and maintain version control.
    • Container services: run inference images and enable elastic scaling.
    • Logging and monitoring services: track first-token latency, per-token latency, VRAM usage, and other metrics.
    • Message queues: smooth peaks and valleys, and balance bursty inference request traffic.

    A hybrid architecture is also possible: offline batch jobs can run on a self-managed cluster, while elastic peaks are handled by cloud GPUs or hosted APIs. When planning such deployments on mainstream cloud platforms or self-managed environments, the VRAM budget and license review checklist above still apply.

    7. Summary: Three core conclusions

    • How much VRAM does a 2.8T model require? The 1.56 TB of weights alone is already close to the theoretical capacity of 8 × 192GB. Long contexts and high concurrency require additional VRAM. Eight GPUs are only the starting line, not a comfortable operating line.
    • Does open weights equal open source? No. Although the custom license preserves freedom for commercial use and fine-tuning, attribution, use restrictions, and redistribution terms must be reviewed clause by clause by legal teams, and license version snapshots must be preserved.
    • Are 8 GPUs enough, and should you self-host? For low concurrency, short contexts, and controllable tasks, self-hosting or a dedicated cloud deployment can be tried. For high concurrency, long contexts, and multi-tenant services, a new VRAM and bandwidth budget is required, or a hosted API should first be used to validate business value.