# Deploying a Trillion-Parameter Open-Weight Model Locally: VRAM and GPU Count Guide
Before you benchmark or download weights for a 2.8T-parameter, 1.56TB open-weight model quantized with MXFP4, work out four things first: whether the weights fit in VRAM at all, how much VRAM you actually need, whether the KV cache becomes the bottleneck, and whether the license permits your use case. This guide gives you a practical framework for answering those questions. It includes a working capacity formula, per-card-size GPU count estimates, KV cache math for long context, parallelism trade-offs (TP, EP, PP), and a pre-launch checklist.
If you are evaluating whether a 2.8T-parameter, 1.56TB open-weight model quantized with MXFP4 can run locally, do not start with leaderboards or a download. Work out four things first: whether the weights fit in VRAM at all, how much VRAM you need, whether the KV cache becomes the bottleneck, and whether the license permits your business scenario. This guide offers a practical framework for those questions.
Start With the Answer: 8 x 192GB Is Not a Stable Deployment Baseline
The most common question is: “Can a trillion-parameter open-weight model run on 8 GPUs?” With 192GB per card, 8 cards give you 1536GB of total VRAM. The weights are roughly 1.56TB, or about 1560GB. On capacity alone, 8 x 192GB is already smaller than the full weight footprint, and that is before reserving room for the CUDA context, framework overhead, communication buffers, KV cache, and memory fragmentation.
So 8 x 192GB cannot directly hold the full 1.56TB MXFP4 weights. At most it serves as a non-production test rig, a setup using host memory or NVMe offloading, aggressive re-quantization, or an extreme low-concurrency, short-context experiment. It is not a stable production configuration.
Using a 0.85 usable-VRAM ratio, the card count required for 1.56TB of weights is roughly:
1.56TB / (192GB x 0.85) ~= 9.56, rounded up to about 10
In other words, with 192GB-class accelerators, about 10 cards is the safer starting point. If you want headroom for long context, KV cache, and high concurrency, add more.
With 256GB-class accelerators, 8 cards give 2048GB total. At a 0.85 usable ratio that is about 1740GB, which exceeds the 1.56TB weight footprint, so 8 x 256GB is relatively viable. But that only means the weights fit. It does not mean you can comfortably support a 1M-token context, high concurrency, or multimodal input. You still need to budget KV cache, activations, batch size, and framework reservations separately.
Verify the Weight Files Before Loading Them
Released weights rarely ship as a single file. For this 2.8T model, the release likely includes more than 100 safetensors shards plus config and tokenizer files. Sharded releases make resumable downloads, parallel downloads, and parallel reads at load time easier, but they also make download integrity checks more important.
After downloading, check at least three things:
- Whether the file count matches the official description.
- Whether shard indices are contiguous, with none missing.
- Whether the hash of each shard (or of the full set) matches the official checksum.
If the weights come from an unofficial mirror, a third-party copy, or a community sync, supply-chain risk rises sharply. For production, do not use weights from an unknown source or without an official hash.
Also, 1.56TB is larger than the ~1.4TB you would get from a rough 4-bit estimate, and that does not automatically mean something is wrong. Three common causes:
- Block-wise quantization formats like MXFP4 store scale factors, which add storage overhead.
- Components such as embedding layers, the vision encoder, and the output head may not all use the same precision.
- TB versus TiB counting conventions differ by nearly 10% in displayed size.
In the end, trust the official model card, the file metadata, and the actual load logs.
2.8T Total Parameters vs. 104B Active Parameters: Compute Can Be Low, VRAM Cannot
This model is a large-scale MoE. Total parameters are about 2.8T; active parameters per token are about 104B. Many deployment discussions conflate the two, which is a mistake.
The 2.8T total determines the storage and VRAM footprint of the weights. The 104B active parameters are only used to estimate per-token decode compute. You cannot assume VRAM only needs to cover the parameters corresponding to 104B.
For stable MoE inference you normally need to keep all routable expert weights resident, or shard the experts across GPUs with tensor parallelism (TP), expert parallelism (EP), or pipeline parallelism (PP). Dynamic offloading that temporarily moves some experts to host memory or NVMe can start up in theory, but you must also account for the performance hit, PCIe bottlenecks, expert swap-in/swap-out latency, and engineering complexity.
Put simply:
- Weight VRAM: estimate from 2.8T total parameters and the quantization precision.
- Compute cost: roughly estimable from the 104B active parameters.
- Stable inference: you cannot load only 104B of parameters unless you explicitly state that offloading, pruning, or a partial model is in use.
Long-context capability also needs separate evaluation. Sparse attention can cut attention compute at long context lengths, but it does not necessarily reduce KV cache. In many cases KV is still stored per layer per token. So in a 1M-token context scenario, KV cache may become a larger VRAM pressure source than the weights themselves.
VRAM Budget: Weights First, Then KV, Then Framework Headroom
At a rough MXFP4 estimate of 0.5 bytes per parameter, 2.8T parameters is theoretically ~1.4TB. The actual weights are ~1.56TB, so you also need to account for quantization scales, non-quantized modules, config files, and counting-convention differences.
Estimates by common card size:
| VRAM per GPU | Estimated count at 0.85 usable | Assessment |
|---|---|---|
| 192GB | ~10 | 8 x 192GB cannot directly hold the full 1.56TB of weights; only viable for offloading or non-production testing |
| 256GB | ~8 | 8 x 256GB is relatively viable, but leave headroom for KV cache, activations, and framework overhead |
| 140GB | ~14 | More GPUs, with higher cross-node communication and scheduling complexity |
If your inference framework cannot load MXFP4 weights directly and dequantizes them to BF16 or FP16 at startup, VRAM usage can multiply. A setup that barely fit may then fail to launch. So framework selection must confirm support for MXFP4 weights, MXFP8 activations, and the relevant TP, EP, and PP combinations.
A VRAM budget is never just weights. A typical deployment should include:
- Weight VRAM.
- KV cache.
- Activations and temporary tensors.
- CUDA context.
- Communication buffers.
- Memory fragmentation and framework reservations.
- Operational overhead: monitoring, logging, debug symbols.
Estimating the KV Cache: Why a 1M Context Amplifies VRAM So Much
KV cache estimates require explicit assumptions. If the official model card does not publish KV head count, head dimension, layer count, and storage precision, the numbers below are illustrative only.
Assume 93 layers, 32 KV heads, head dimension 128, and 1 byte per KV element. Per-token KV cost is then:
2 x 93 x 32 x 128 x 1 byte ~= 761856 bytes ~= 0.727 MiB/token
The factor of 2 is because each token normally stores both K and V.
Per-request KV cache estimates at different context lengths:
| Context length | Estimated KV cache per request | Notes |
|---|---|---|
| 32K | ~23.3 GiB | Manageable for a single request, but concurrency scales it fast |
| 128K | ~93.0 GiB | Noticeable pressure on 8- or 10-GPU setups |
| 1M | ~744 GiB | At long context, KV can become the dominant VRAM consumer |
If KV is stored in BF16 or FP16, roughly double these numbers. More aggressive KV quantization or sparse KV strategies will change them. So “can 8 cards handle a 1M-token context?” cannot be answered by comparing weight size alone. You need the KV estimate, concurrent request count, batching strategy, and framework implementation together.
Parallelism: TP, EP, and PP Are Not Either/Or Choices
Trillion-parameter MoE models rarely rely on a single parallelism strategy. Common combinations:
- Tensor parallelism (TP): shards individual expert weight matrices across GPUs. Suits high-bandwidth intra-node interconnect, but may require communication at every layer.
- Expert parallelism (EP): distributes many experts across GPUs. Routed tokens must be sent across cards, which demands good network bandwidth and expert load balancing.
- Pipeline parallelism (PP): splits the model by layer, reducing per-GPU VRAM pressure but introducing pipeline bubbles.
- Data parallelism or request-level replication: improves throughput, but multiplies total VRAM demand.
With high-bandwidth intra-node links such as NVLink or equivalent, TP tends to pay off. For cross-node deployment, EP and PP communication costs rise, so watch InfiniBand, RoCE, or high-speed Ethernet bandwidth. MoE models also need attention to expert load imbalance: popular experts can drive per-GPU VRAM and communication pressure up on individual cards.
Do Not Trust Self-Reported Benchmarks Alone: Compare Under Identical Conditions
Vendor-published evaluations are useful reference, but should not be the only input to a purchase decision. The safer approach is a like-for-like comparison table containing at least:
| Field | Requirement |
|---|---|
| Benchmark name and version | Must be traceable |
| Context length | Short and long context reported separately |
| Tool calling allowed | Explicitly on or off |
| Sampling protocol | For example pass@1, pass@5, or majority voting |
| Hardware and quantization | Weight precision, activation precision, inference framework |
| Test samples | Prefer your own business data |
For end-user coding, long-running agents, long-document understanding, and scientific Q&A, do not judge by a single aggregate score. Results for trillion-parameter models depend heavily on deployment configuration: context length, tool calling, batch size, and expert routing policy. Start with short-context, single-request accuracy smoke tests, then increase context length and concurrency step by step while recording VRAM usage, throughput, latency, and error rates.
Licensing: Open Weights Are Not Unconditional Commercial Permission
Open-weight models usually permit downloading, use, and fine-tuning, but the license still needs careful review. Some custom licenses allow free use, copying, modification, distribution, fine-tuning, and commercial use, while setting thresholds for large-scale commercial scenarios, such as:
- Revenue thresholds for model-as-a-service offerings.
- Monthly active user or monthly revenue thresholds for end products.
- Requirements to prominently credit the model name in certain scenarios.
- A separate agreement with the copyright holder once thresholds are exceeded.
The thresholds below are examples only and are not a legal conclusion. Actual revenue definitions, business attribution, edge cases, and final applicability should be based on the official license text and professional legal advice.
Example checks:1. Is the model offered to others as a service (MaaS)?2. Does trailing 12-month revenue for that service exceed the example threshold?3. Do monthly active users of the end product exceed the example threshold?4. Does monthly revenue of the end product exceed the example threshold?5. Must copyright notices, license notices, or model name attribution be retained?
If your business involves a public API, SaaS, embedding the model in an end product, redistribution, or sublicensing, add license review to your compliance checklist early. Teams with large annual revenue or user scale should confirm with the copyright holder in advance whether a separate agreement is required.
Security and Operations: Account for Risk, Not Just VRAM
Local deployment of a trillion-parameter model is not just downloading weights. It involves an entire serving stack. At minimum:
- Enable authentication on the inference service; do not expose it directly to the public internet.
- Put API keys, object storage credentials, and model registry credentials into a secrets manager.
- Redact logs so sensitive user input is not recorded.
- Apply least privilege to object storage, separating public read-only, private download, and write permissions.
- Hash-verify weight files to guard against tampered mirrors.
- Network-isolate the inference gateway and restrict access to management ports.
- Monitor load failures, CUDA OOM, abnormal expert routing, and long-context VRAM overflow.
Before going live, run a pre-deployment check:
- Short-context, single-request smoke test.
- Verify VRAM usage and that weights loaded correctly.
- Increase context length gradually.
- Increase concurrency gradually.
- Monitor KV cache growth and communication latency.
- Record accuracy acceptance results and framework logs.
FAQ
Q: How many 192GB GPUs does a 2.8T-parameter model need?
A: Based only on 1.56TB of weights and a 0.85 usable ratio, about 10 x 192GB. Eight 192GB cards total less VRAM than 1.56TB and cannot load the full weights directly, unless you use offloading, compression, or non-production testing.
Q: How much VRAM does MXFP4 save?
A: At 4 bits, about 0.5 bytes per parameter, so 2.8T parameters is theoretically ~1.4TB. The actual weights may reach 1.56TB due to block-wise scales, non-quantized components, and counting conventions. KV cache and activations must be calculated separately.
Q: How do I estimate the KV cache?
A: Per-token cost is roughly “layers x 2 x KV heads x head dimension x bytes per element.” If KV head count or layer count is not published, use explicit example assumptions and label them as such.
Q: Can 8 x 256GB work?
A: Eight 256GB cards give about 2048GB total, or ~1740GB at a 0.85 usable ratio, so loading the weights is relatively feasible. But you still have to evaluate KV cache, concurrent requests, context length, and framework overhead. Do not assume stable operation at a 1M-token context.
Q: Can I use it commercially?
A: Check the specific license. Many small and mid-sized scenarios may fall under permissive terms, but reaching revenue, user-scale, or monthly-revenue thresholds may require additional attribution or a separate agreement. The official license text and legal advice are authoritative.
