# 2.8T MoE Model On-Premises Deployment Guide: Licensing, VRAM Math, 8 vs 16 GPUs, and Expert Offloading
Open weights do not mean a model is ready to go into production. This guide walks through what it actually takes to deploy a ~2.8T-parameter MoE model (about 104B activated parameters, ~1.56TB of weight files) locally: how to review the license first, how to estimate VRAM from weights and KV Cache, how the 8-GPU and 16-GPU options compare at 128K context, how to configure parallelism in vLLM, how to handle expert offloading, and which errors to expect during bring-up. It includes KV Cache calculations for fp16/bf16 and fp8, a configuration comparison table, troubleshooting checklists, TCO formulas, and a phased launch checklist.
Quick takeaway: open weights are not the same as production-ready
If you are evaluating an open-weight MoE model with roughly 2.8T total parameters, about 104B activated parameters, and around 1.56TB of weight files, there is one thing to settle first: downloadable weights do not mean a low barrier to local deployment. For a model at this scale, the deployment decision cannot rest on “can we get it to run.” You also need to close the loop on licensing, VRAM, KV Cache, parallelism strategy, expert offloading, content safety, and long-term operational cost.
On the specific question of whether 8 GPUs with large per-card memory are enough for a 2.8T MoE model, the reasonable framing is:
- An 8-GPU setup is not a general-purpose production starting point. It only becomes viable when each card offers around 192GB of memory, and you accept expert offloading, low concurrency, short contexts, and higher latency, in which case it suits small-scale validation or controlled production.
- With 80GB-class accelerators, you typically need 16 or more cards, combined with expert offloading, KV Cache quantization, context compression, and request tiering.
- The 1.56TB of weights sets the floor for VRAM, but KV Cache, activations, runtime overhead, and communication buffers all push the real requirement higher.
- The hard part of MoE is not just that the model is large. It also includes expert routing, hot experts, cold expert loading, expert-parallel communication, and inference framework support.
- Licensing is a precondition, not an afterthought. Open weights do not equal standard open source, and they do not automatically permit commercial use, redistribution, or publishing derivative models.
The more rational path: review the license first, then calculate VRAM, then validate on 16 GPUs or a controlled 8-GPU setup, and only then decide whether to scale.
1. Start with the license: open weights are not standard open source
The most commonly overlooked risk with models like this is not insufficient VRAM but license terms that do not match the business scenario. Open weights usually mean only that “the model files can be obtained.” Training data, training code, data recipes, and evaluation details may not be released. Enterprises should verify: whether commercial use is allowed, whether private internal deployment is allowed, whether external API serving is allowed, whether redistribution is allowed, whether fine-tuning, distillation, or publishing derivative models is allowed, how liability for generated output is allocated, and whether a license upgrade requires re-review.
It is good practice to retain the model download hash, license version, reviewer, review date, and re-review triggers. In particular, after secondary fine-tuning, do not assume the original license conclusion still applies. Reassess whether the derivative model can be published externally. If the official terms contain no explicit commercial clause, do not assume it is usable. The safest sequence is legal review first, engineering testing second, which avoids rework after launch.
2. Model structure: 2.8T parameters cannot be reasoned about in one step
A defining trait of this class of model is a very large total parameter count, while no single inference pass involves all parameters at once. The MoE structure routes each token to a subset of experts, which reduces the compute required per inference. Using the figures in this article as the reference point: roughly 2.8T total parameters, about 104B activated parameters, roughly 1.56TB of weight files, about 896 experts, 16 experts selected per token, plus 2 shared experts.
Note that the activated parameter count is not simply “the sum of 16 experts’ parameters.” It also includes shared expert parameters, attention layers, normalization layers, embedding layers, and other non-expert parameters. If the model supports multimodality, the vision encoding module adds further activation overhead. So you cannot derive 104B activated parameters from “16 out of 896” alone. You still need to check the model configuration file and the official parameter documentation.
The advantage of MoE is that total capacity grows while each inference activates only part of it, which suits multi-task, multi-domain, long-document, coding assistant, and agent scenarios. The engineering challenges are equally clear: expert routing can be unstable and load can be imbalanced; some experts stay cold for long periods, occupying VRAM while rarely being hit; hot experts increase communication and cache pressure; expert parallelism is more complex than conventional tensor parallelism; and the inference framework must support expert routing, parallel partitioning, and scheduling. If the framework does not support expert parallelism or expert offloading, many of these plans stay theoretical.
3. How to estimate VRAM: weights first, then KV Cache
The VRAM requirement can be simplified as:
Required VRAM ≈ weight footprint + activation footprint + KV Cache + runtime overhead + communication buffers
For a 2.8T-class MoE model, the weights are only the first gate. Unless stated otherwise in this article, weights and GPU capacity are expressed in decimal GB/TB, while KV Cache is given in both GiB and engineering approximations. The 1.56TB of weights can be approximated as 1560GB.
1. Weight footprint: ideal averaging is not actual allocation
If the total weight size is about 1.56TB, you can estimate the ideal per-card weight by dividing total weight by the number of cards:
Ideal per-card weight ≈ 1.56TB ÷ number of GPUs
This is only an ideal figure. Actual per-card weight varies with tensor parallelism, pipeline parallelism, expert parallelism, and expert offloading. Under TP, a single layer’s weights are split across multiple cards. Under PP, a card may hold one stage of layers, including all experts belonging to that stage. Under EP, a card may hold a subset of experts across all layers. Any real deployment must therefore account for the parallelism strategy, layer placement, and expert partitioning.
2. KV Cache applicability: confirm the attention structure first
Before estimating KV Cache, confirm the model’s attention structure. The formula and worked example below apply to standard MHA or GQA-style structures. If the model uses MLA, low-rank KV compression, or another non-standard cache structure, you must recalculate using the official configuration and the inference framework implementation. Do not apply this article’s example directly.
KV Cache ≈ layers × KV heads × head dim × K/V copies × token count × bytes per element × concurrency
3. 128K, single concurrency: fp16/bf16 versus fp8
Assume 93 layers, 64 KV heads, head dimension 128, both K and V cached, context length 131072, and concurrency 1. With a 2-byte KV cache precision, that is fp16 or bf16, the engineering estimate for single-concurrency KV Cache is about 371.9GiB, or roughly 399GB in decimal terms. Spread across 16 cards, that is about 23.2GiB per card, which can be rounded to roughly 25GiB per card in capacity planning.
If fp8 KV Cache is enabled and the calculation is redone at 1 byte, single-concurrency KV Cache is about 185.9GiB, or roughly 200GB in decimal. Across 16 cards, that is about 11.6GiB per card, which can be rounded to roughly 13GiB per card in planning.
If concurrency rises to 8, estimated at 2-byte fp16/bf16:
Per-card KV Cache ≈ 371.9GiB × 8 ÷ 16 ≈ 186GiB per card
That 186GiB is approximately 200GB. Using the decimal approximation common in engineering planning, combining 97.5GB of per-card weight with roughly 186GB of per-card KV gives a total pressure of about 283GB. With strict binary conversion, the result is slightly higher, around 297GB. Under either convention, this exceeds a 192GB single-card capacity, so you need to lower concurrency, add cards, enable fp8 KV Cache, compress context, or adopt more aggressive paged scheduling.
If the context grows from 128K to 1 million tokens, single-concurrency KV Cache approaches the 3.2TB range. Configurations like this cannot go live under a normal production standard.
4. Configuration selection: 8 GPUs or 16 GPUs
The following is a rough assessment at 128K context, using ideal per-card weight averaging. Actual figures vary with parallelism and expert offloading.
| Configuration | Total GPU VRAM | Ideal per-card weight | 128K single-concurrency KV pressure | Feasibility conclusion |
|---|---|---|---|---|
| 8×80GB | 640GB | ~195GB per card | Total VRAM below 1.56TB; CPU or storage expert offload required | Not recommended for production; weight and runtime pressure too high |
| 8×141GB | 1128GB | ~195GB per card | Total VRAM below 1.56TB; CPU or storage expert offload required | Suitable only for small-scale validation, with short context and low concurrency |
| 8×192GB | 1536GB | ~195GB per card | Weight already exceeds single-card capacity; expert offload or memory pooling needed | Usable for controlled validation, not suitable as a production default |
| 16×192GB | 3072GB | ~97.5GB per card | ~25GiB per card at fp16, ~13GiB per card at fp8 | A better private-production starting point, but concurrency still needs control |
| 16×80GB | 1280GB | ~97.5GB per card | Total VRAM below 1.56TB; expert offload and rate limiting required | Suitable for controlled production or experimentation, with demanding latency and stability targets |
The core conclusions from this table: 8×80GB is not suitable for direct production; 8×192GB is not “naturally sufficient,” because the weights alone approach or exceed a single card’s capacity; 16×192GB is a better starting point for production, though concurrency and context still need control; and 16×80GB requires more aggressive offloading and rate-limiting strategies.
5. Deployment architecture: from validation to production
Option 1: a small-scale validation environment
Do not chase high throughput during validation. The goal is to confirm whether the model is usable and correct. Recommended checks: whether the weights load completely, whether a single inference succeeds, whether a 128K context works, whether image input works, whether output is stable, and whether peak VRAM stays within bounds. At the same time, confirm the vLLM version, model format, the security boundary of trust_remote_code, and whether the license hash and review records have been retained.
The objective at this stage is to determine “does it run,” not “can it handle high concurrency.”
Option 2: a production-grade private inference cluster
For production, 16 cards is the recommended starting point. The following is a structured deployment sketch. Note that this is not a native vLLM configuration file; it only illustrates the parallelism and context strategy.
# Structural illustration only; not a vLLM config filedeployment:model: /models/openweight-moeparallel:tensor_parallel: 8pipeline_parallel: 2context:max_model_len: 131072kv_cache_dtype: fp8cache:enable_prefix_caching: truemoe:top_k: 16shared_experts: 2expert_offload: false # vLLM has no native expert offload option; an external solution is needed
The corresponding vLLM launch command example follows, using vLLM 0.9.x. Check the official documentation for your deployed version for actual parameters:
pip install vllm==0.9.2vllm serve /models/openweight-moe \--served-model-name openweight-moe \--trust-remote-code \--tensor-parallel-size 8 \--pipeline-parallel-size 2 \--max-model-len 131072 \--kv-cache-dtype fp8 \--enable-prefix-caching
Some clarifications: tensor_parallel_size: 8 means 8-way tensor parallelism within one node; pipeline_parallel_size: 2 means pipeline parallelism across two nodes; the combination of 8 and 2 corresponds to a 16-GPU production configuration. max_model_len: 131072 means a 128K context, which is safer than a million-token context. Whether kv_cache_dtype: fp8 can be enabled depends on hardware, driver, framework version, and model configuration. Cross-node TP or PP requires high-speed interconnect or RDMA, otherwise communication may become the bottleneck.
What about expert offload commands?
vLLM itself does not necessarily provide a mature expert offloading capability. If expert offloading is mandatory, you can evaluate community options such as KTransformers or llama.cpp’s -ot flag, but you need to verify whether the model format is supported, whether the framework version supports it, the change in TTFT after offloading, the magnitude of throughput degradation, host memory bandwidth, and CPU memory capacity. Also prepare OOM and timeout troubleshooting plans. Expert offloading generally increases latency, especially when cold experts are loaded frequently from memory or storage. It is not a free rescue but a compromise when VRAM is insufficient.
6. Common errors and a troubleshooting checklist
1. OOM during startup
Likely causes: underestimated per-card weight, no allowance for KV Cache, max_model_len set too high, KV cache precision too high, ignored vision encoder or activation footprint, or imbalanced parallel partitioning. When troubleshooting, start with a shorter context, lower max_model_len, reduce gpu-memory-utilization, watch the VRAM curve in nvidia-smi, and pay attention to the loading and KV Cache allocation phases in the framework logs.
2. Timeouts after expert offloading
Likely causes: insufficient host memory, insufficient memory bandwidth, frequent loading of cold experts, uneven expert hit distribution, or storage reads becoming the bottleneck. Optimizations include pinning hot experts in cache, lowering concurrency, shortening context, separating hot and cold requests into tiers, and adding host memory or improving memory bandwidth.
3. fp8 KV Cache cannot be enabled
Likely causes: unsupported hardware, insufficient driver version, unsupported vLLM version, incompatible model configuration, or quantization calibration and dynamic range not meeting requirements. The recommendation is to fall back to fp16 or bf16 KV Cache first, then check the hardware generation and the vLLM version notes, and use load testing to confirm whether the quality regression is acceptable.
4. Abnormal behavior when combining TP, PP, and expert parallelism
Likely causes: the expert count cannot be partitioned correctly by the parallelism degree, the communication topology is unsupported, the framework imposes many restrictions on expert parallelism, or inter-node network bandwidth is insufficient. When troubleshooting, validate with pure tensor parallelism first, then enable pipeline parallelism step by step, then evaluate expert parallelism, while watching cross-node communication time.
5. Long-context VRAM over budget
Tier this by business need: cap interactive Q&A at 32K or 64K; set 128K for document understanding based on document length; offline batch processing can evaluate longer contexts but should be made asynchronous; million-token contexts are for experiments only and should not enter the default production configuration.
7. Key engineering optimizations
1. Prefix caching
In agent, coding assistant, and knowledge-base Q&A scenarios, many requests share the same system prompt, tool descriptions, or document fragments. In vLLM, look at enable_prefix_caching or the corresponding parameter for your version, and check the official documentation for the deployed version. Enabling prefix caching reduces redundant computation and lowers time to first token.
2. Request tiering
Tier requests by business value: high priority for interactive Q&A and real-time assistance; medium priority for background tasks and document generation; low priority for batch analysis and offline evaluation. When resources are constrained, protect high-priority requests first.
3. Context truncation and compression
Do not blindly max out the context window. For many workloads, sensible truncation, summary compression, and retrieval augmentation are more cost-effective than simply extending the context.
4. Expert heat governance
A long-running inference service should periodically collect expert hit statistics: keep high-frequency experts resident, load low-frequency experts dynamically, apply caching policies to hot experts, and apply eviction policies to cold experts.
5. Content safety and output review
Larger models generate more capable output, but it is also harder to fully predict. Recommended controls include input filtering, output review, sensitive-topic detection, high-risk intent blocking, log auditing, and human review.
8. How to calculate cost and return
Private deployment should not be judged on GPU procurement cost alone. Total cost of ownership also matters:
Total private deployment cost ≈ GPU depreciation + power with PUE + bandwidth + operations headcount + compliance review + model adaptation
GPU depreciation can be estimated as procurement cost divided by the depreciation period; power can be estimated as PUE × system power draw × electricity price; bandwidth can be estimated from dedicated-line or cloud egress costs; operations headcount should reflect actual staffing and labor cost; compliance review should include legal and security cost; and model adaptation should cover load testing, tuning, incident handling, and version upgrades.
You should also account for GPU utilization, idle depreciation, and replacement cycles. GPU depreciation should not be a simple average of procurement cost; it should be adjusted for utilization. Idle depreciation can be estimated from procurement cost, depreciation period, and the utilization gap. Replacement cycles should consider hardware generations, framework upgrades, and model evolution.
If the business does not demand high model capability, or concurrency is small, a general-purpose cloud inference service may be more economical. A very large open-weight model becomes more attractive only when privacy, compliance, customization, and long-term cost are core requirements.
9. Phased launch checklist
| Phase | Checklist items |
|---|---|
| POC acceptance | Weights load completely, single inference succeeds, 128K context works, image input works, peak VRAM under control, license records retained |
| Production readiness | SLA met, monitoring integrated, content safety in place, failure degradation defined, license review complete, capacity load test complete |
| Post-launch monitoring | TTFT, TPOT, throughput, peak VRAM, expert hit distribution, refusal rate, error rate, cost |
10. FAQ
1. Can open-weight models be used commercially?
It depends on the specific license text. Open weights do not equal standard open source, and they do not automatically permit commercial use, redistribution, or publishing derivative models. Before any commercial use, review the full official license and have legal confirm whether internal use, external serving, redistribution, and derivative model release are allowed.
2. Is 8-GPU VRAM enough for a 2.8T MoE model?
Stable production is very unlikely. 8×80GB totals 640GB, which cannot hold 1.56TB of weights, nor can it hold a 128K KV Cache under conventional settings. Even with expert offloading enabled, latency, throughput, and stability will suffer. A safer approach is 16 cards or more, or 192GB-class cards combined with offloading and KV quantization.
3. Can 8 GPUs with 192GB each run it?
It can serve as a small-scale validation or controlled experiment setup, but per-card weight is about 195GB, which already exceeds single-card capacity, so it must rely on expert offloading, memory pooling, or resources from adjacent nodes. If you require a 128K context plus some concurrency, the pressure grows further.
4. Should I choose 8 cards or 16?
For production-grade private deployment, evaluate 16 cards first. 8 cards suit validation, low concurrency, short context, or scenarios that tolerate higher latency. 16×192GB is more stable than 16×80GB because the per-card VRAM headroom is larger and the pressure from KV Cache and runtime overhead is smaller.
5. How do I estimate VRAM for an MoE model?
Use: weight footprint + activations + KV Cache + runtime overhead + communication buffers. Weight footprint can be estimated initially as total weight divided by the number of cards to get an ideal per-card value; KV Cache can be estimated from layer count, KV head count, head dimension, token count, bytes per element, and concurrency. Actual deployments must be corrected using framework logs and load-test results.
6. Is a million-token context practical?
It is theoretically possible but very expensive. 1 million tokens significantly amplifies KV Cache, VRAM footprint, and latency. Production typically limits context to 128K or lower first, then controls cost with KV quantization, prefix caching, retrieval augmentation, and request tiering.
11. Final recommendation: validate, then comply, then scale
The value of a very large open-weight MoE model lies not in “it has a lot of parameters” but in whether it can run stably, controllably, and affordably in a specific business. The more rational adoption path: review the license first and confirm commercial use, redistribution, derivative models, and output liability; then calculate VRAM, separating weights, KV Cache, activations, and communication overhead; then run a small-scale validation on 16 GPUs or a controlled 8-GPU setup; once validation passes, do capacity load testing, monitoring integration, and content safety; and only then decide whether to scale into production. Open-weight models are not download-and-go tools. They are a systems engineering effort that requires hardware, framework, algorithm, compliance, and operations to work together.
