Deploying Very Large Open-Weight Models Locally: A Practical VRAM and Hardware Sizing Guide

    This guide explains why "you need 8 GPUs" is a conditional example rather than a rule for running very large open-weight models on your own hardware. Using a worked example of 2.8T total parameters, 104B activated parameters, 896 experts, 1.56TB of weights, and a 1M-token context window, it shows how to build a VRAM ledger (weights, KV cache, activations, framework overhead, communication buffers, safety margin), estimate KV cache growth, reason about MoE routing and expert parallelism, evaluate quantization safely, and choose hardware. It also includes reference configuration tables, a troubleshooting table, and a pre-launch checklist covering observability, fallback strategies, security, and compliance.

    1. Start With the Conclusion: “8 GPUs” Is a Conditional Example, Not a Formula

    As soon as a very large open-weight model is released, the question developers care about is usually not “can I download the weights?” but “can I run this reliably?” Weight files reach terabyte scale, parameter counts can reach the trillions, and the model is often not a single dense network but a composite system with expert routing, sparse activation, long-context caching, a vision encoder, and tool-calling capability. Local deployment is therefore not a simple file load. It is systems engineering that touches storage, VRAM, compute, interconnect, the inference framework, scheduling, observability, and security compliance.

    Two questions need to be separated from the start: open weights solve model availability; local deployment solves resource fit. An open license determines whether you can obtain the weights at all. Hardware topology, runtime architecture, quantization strategy, and your business SLA determine whether you can keep serving traffic.

    So “you need 8 GPUs to run this locally” should not be read as a universal rule. The more accurate framing is: under specific hardware configurations, precision strategies, context lengths, and concurrency targets, an 8-GPU-class machine may be the minimum viable or recommended configuration. If 1.56TB of weights must be fully resident in VRAM, 8 GPUs means roughly 200GB per card at minimum. If a single card cannot hold that much, you must bring in quantization, layered loading, CPU offloading, expert sharding, or multi-node scaling. Otherwise, “it runs” only means an experimental launch, not production readiness.

    2. Pin Down the Numbers First: 2.8T, 104B, 896 Experts, 1.56TB, 1M Context

    To avoid mistaking example figures for general facts, label the source, model version, precision convention, and runtime framework before you deploy. The 2.8T total parameters, 104B activated parameters, 896 experts, 1.56TB of weights, and 1M-token context used here are illustrative. Actual deployment should follow the real model release notes, weight format, inference framework, and load-test results.

    Metric Example value What it means Deployment impact Boundary you must confirm
    Total parameters 2.8T All expert and shared-module parameters combined Drives storage, host memory, and weight-residency pressure Is this the official release? Does it include the vision encoder and tool-calling modules?
    Activated parameters 104B Parameters that actually compute per token Reduces per-token compute, but not storage or routing cost Is the activation strategy fixed? Does it depend on dynamic routing?
    Expert count 896 experts, 16 selected per token, plus 2 shared experts Typical MoE structure Increases expert scheduling, caching, and communication complexity Are experts evenly distributed? Is expert parallelism supported?
    Weight footprint 1.56TB Weight files, or estimated size after loading The first constraint is often storage and host memory, not the GPU Precision, quantization, compression, framework memory layout
    Context length 1M tokens Long-context capability KV cache can become the dominant VRAM variable Does it rely on sparse attention? Does the runtime support it?

    The value of this table is the reminder not to look at “total parameters” or “activated parameters” alone. Total parameters drive storage and VRAM. Activated parameters drive compute volume. Expert structure drives scheduling and communication. Context length drives KV cache. Multimodal capability drives vision encoding and input-processing overhead.

    3. Can 8 GPUs Actually Run It? Reference Configurations

    For “8 GPUs” to become an actionable conclusion, it has to be bound to precision, weight footprint, per-card VRAM, quantization, CPU offloading, context length, concurrency target, and first-token latency requirements. The following table replaces a single answer with a set of scenarios.

    Option Weight and quantization basis Per-card VRAM needed GPUs Quantized CPU offload Target context Target concurrency First-token latency Suitable for
    A: full precision, fully resident 1.56TB weights ~200GB+ per card 8 No No Up to 8K 1–2 Low, but the prerequisites are extremely strict Short text, low concurrency, experimental validation
    B: 8-bit or FP8-class quantization ~0.8TB of weights ~100–120GB per card 8 Yes No Up to 32K 4–8 Moderate Production candidate, subject to accuracy regression testing
    C: 4-bit or layered expert loading ~0.4–0.6TB resident in VRAM 80GB+ per card, depends on host memory 8 Yes Yes Up to 8K 1–2 Higher Offline inference, low concurrency, tolerable latency
    D: multi-node scaling 1.56TB weights 80–141GB per card 16 or 32 No, or as needed No, or as needed 64K–100K High Depends on interconnect and parallelism strategy Long context, high concurrency, enterprise serving

    One point deserves emphasis: the “per-card VRAM needed” column only reflects the amortized weight footprint. It does not include KV cache, activations, framework overhead, communication buffers, or safety margin. Real deployments typically reserve an additional 10–20% or more. If the target is a million-token context or high concurrency, VRAM demand can far exceed the amortized weight figure.

    If each of 8 accelerators has 80GB, total VRAM is 640GB and 1.56TB of weights cannot be fully resident. At 141GB per card, the total is roughly 1,128GB, still not enough to cover weights plus runtime overhead. At 192GB per card, the total is roughly 1,536GB, which only barely covers the weights and still requires strict control over context, concurrency, and cache strategy. An 8-GPU deployment must therefore be evaluated together with the precision, quantization, and scheduling plan.

    4. The VRAM Ledger: Weights Are Not the Only Variable

    The first step in deploying a very large model locally is not picking a GPU. It is building a VRAM ledger. A simplified estimate:

    Total VRAM demand ≈ resident weights + KV cache + activations and temporary tensors + framework overhead + communication buffers + safety margin

    Resident weights must be split according to tensor parallelism, pipeline parallelism, expert parallelism, and layered loading. KV cache is the part most often underestimated in long-context scenarios. Activations and temporary tensors grow substantially with long sequences, multimodal inputs, and high-concurrency batching. Framework overhead includes the runtime, memory allocator, compilation cache, and CUDA graphs or equivalent mechanisms. Communication buffers scale with tensor, expert, and pipeline parallelism.

    KV cache can be understood with a general formula:

    KV cache ≈ 2 × layers × KV heads × head_dim × bytes per element × batch size × sequence length

    The 2 accounts for Key and Value caches. More layers means a larger cache; longer sequences mean a larger cache; larger batches mean a larger cache; fewer bytes per element means a smaller cache. Note that KV heads are not the same as attention heads. In classic MHA, KV heads usually equal attention heads. In MQA, KV heads equal 1. GQA shares multiple query heads over fewer KV heads. Structures such as MLA may use compressed latent representations, so the classic formula does not apply directly.

    For example, assume 60 layers, 8 KV heads, head_dim of 128, BF16 precision, one token per request, and a 1M-token context:

    2 × 60 × 8 × 128 × 2 × 1 × 1,000,000 ≈ 245.8GB

    Raise the batch size to 8 and the KV cache approaches 2TB. A million-token context is therefore not a simple length extension; it is a high-risk VRAM management scenario. Real numbers are also affected by paged caching, block alignment, allocation strategy, sparse caches, compressed caches, vision-encoder caches, and runtime implementation details, with error margins that can reach 20–50% or more.

    So when assessing million-token context support, do not just ask “does the model support it?” Also ask: does the runtime support sparse attention, is the KV cache paged, is cache compression supported, are requests properly isolated, and does it OOM under long-context concurrency?

    5. MoE Architecture: The Real Bottleneck Is Routing, Caching, and Communication

    For a MoE model with 2.8T total parameters and 104B activated parameters, deployment complexity does not come only from parameter count. With 896 experts, 16 selected per token, and 2 additional shared experts, every token triggers dynamic expert selection. The system must handle expert load balancing, expert caching, routing strategy, communication scheduling, and fault isolation.

    A common MoE inference mistake is treating activated parameters as the VRAM requirement. 104B activated parameters lower per-token compute, but they do not remove the storage pressure created by 2.8T total parameters. Every expert still has to reside on callable media. Otherwise, once an expert has to be loaded across devices or nodes, latency rises noticeably.

    Expert parallelism can improve scheduling efficiency, but it introduces additional All-to-All communication. If inter-card or inter-node bandwidth is insufficient, compute units spend their time waiting for data synchronization and throughput drops. At low concurrency, latency may come mainly from expert loading and routing; at high concurrency, VRAM capacity, VRAM bandwidth, and communication bandwidth become the primary bottlenecks.

    MoE deployment therefore needs close monitoring of expert load distribution, routing hit rate, expert cache hit rate, communication time, and hotspot devices. If certain experts consistently land on a few devices, you get local VRAM pressure and compute hotspots that reduce overall throughput.

    6. Long Context and Multimodality: Runtime Complexity Rises Further

    A 1M-token context matters greatly for long documents, codebase analysis, document Q&A, and agent tasks, but it also changes deployment complexity substantially. Under classic attention, both attention compute and KV cache grow noticeably with sequence length. Without careful cache management in the runtime, VRAM can be exhausted midway through a request.

    Replacing part of the full attention with sparse attention reduces wasted compute and cache pressure on long sequences. But sparse attention is not a simple flag at the model level; it requires the inference runtime to provide matching kernels, cache policies, and memory layout support. If the deployment environment lacks a compatible implementation, the design advantage may never materialize.

    Multimodal capability adds resource overhead too. The vision encoder participates in image preprocessing, feature extraction, and cross-modal alignment. The higher the image resolution and the more images per input, the more preprocessing time and VRAM are consumed. Agent capability adds multi-turn reasoning, tool invocation, state maintenance, and permission boundary management. Compared with single-turn Q&A, agent serving has higher engineering complexity, requiring monitoring across input parsing, vision encoding, model inference, tool calls, cache hits, and response delivery.

    7. How to Choose Hardware: Answer Six Questions First

    Before selecting hardware, answer the following:

    1. Can the weights be fully resident in VRAM? If not, will you accept quantization, layered loading, or CPU offloading?
    2. Is total machine VRAM sufficient? Is per-card VRAM sufficient? Is multi-node mandatory?
    3. Does inter-card interconnect meet the needs of tensor and expert parallelism? Is there high-speed intra-node interconnect, and InfiniBand or RoCE between nodes?
    4. Is CPU memory large enough for expert or layer offloading? Can PCIe bandwidth support the target latency?
    5. Can storage hold 1.56TB of weights? Does load speed affect cold start and scale-out?
    6. Does the inference framework support MoE, quantization, paged KV cache, sparse attention, multimodality, and observability?

    Hardware selection cannot look at peak compute alone. For very large models, VRAM capacity, VRAM bandwidth, inter-card interconnect, CPU memory, storage throughput, and framework ecosystem usually matter more than single-card FLOPS.

    8. Is Quantization Acceptable? Smaller Files Are Not the Whole Story

    Quantization is not just file compression. Once weight size drops, you must re-evaluate accuracy, stability, long-context behavior, tool-calling correctness, and multimodal quality. At least three layers of validation are advisable:

    Layer 1 — Offline quality evaluation. Compare pre- and post-quantization behavior on business datasets across Q&A, code, long documents, tables, image understanding, and tool calls.

    Layer 2 — Boundary testing. Focus on very long contexts, very long inputs, high concurrency, multi-turn conversations, malformed input, empty requests, and very long outputs. Watch for VRAM overflow, response timeouts, and quality degradation.

    Layer 3 — Production canary. Roll out to a small traffic share first and observe first-token latency, per-token latency, P95/P99 latency, error rate, VRAM watermark, GPU utilization, and expert load distribution. If a key metric crosses its threshold, pause the ramp-up and roll back.

    A common approach is to keep sensitive layers at higher precision while quantizing some non-sensitive layers, combined with mixed-precision storage. Be especially careful with routing layers, shared experts, and the vision encoder, since they can affect overall output quality and cross-modal alignment.

    9. Common Pitfalls and Troubleshooting

    Common pitfall Typical symptom Likely cause How to investigate and fix
    Treating 104B activated parameters as the VRAM requirement OOM right after weight loading Ignoring 2.8T total parameters and expert residency Recompute VRAM from total parameters and weight footprint
    Looking at total VRAM only, not per-card headroom One card OOMs first Uneven expert distribution or uneven sharding Check expert parallelism, load distribution, and per-card headroom
    Load-testing short text only Long requests OOM after launch KV cache grows with sequence length Cap maximum context, load-test long requests, enable paged or compressed cache
    Testing a single request only Latency spikes under high concurrency Insufficient batching, communication, or VRAM bandwidth Add concurrency-gradient load tests, watch P99 and VRAM watermark
    Ignoring interconnect bandwidth Slow tensor parallelism, low throughput Inter-card communication is the bottleneck Check NVLink, PCIe, InfiniBand, and RoCE configuration
    Blindly using CPU offloading Very high first-token latency, low throughput Insufficient PCIe or memory bandwidth Lower concurrency, optimize expert layering, or switch to a stronger single-node or multi-node setup
    Overlooking the vision encoder Image requests are noticeably slower Image preprocessing and vision encoding consume extra resources Cap resolution, image count, and batch size; monitor the vision path separately
    Running functional tests only Cannot recover after a failure No degradation, fallback, or isolation mechanisms Add timeouts, circuit breakers, standby instances, and request throttling

    10. Pre-Launch Checklist

    • Resource profile: confirm total parameters, activated parameters, weight footprint, context length, multimodal support, target concurrency, and average input/output lengths.
    • Precision acceptance: confirm the quantization plan passes business evaluation and that a rollback version exists.
    • VRAM estimate: confirm explicit headroom for weights, KV cache, activations, framework, communication buffers, and safety margin.
    • Load testing: cover short text, long text, multi-turn conversations, high concurrency, image input, and malformed input.
    • Observability: collect first-token latency, per-token latency, P95/P99, VRAM watermark, GPU utilization, expert load, and error rate.
    • Degradation strategy: when VRAM runs short or a node fails, be able to cap context, reject oversized requests, and switch to a smaller model or a standby instance.
    • Security and compliance: apply access control, masking, and retention management to weight files, logs, session state, and tool-call data.

    11. Enterprise Rollout: The Engineering System Matters More Than the Hardware

    In enterprise settings, deploying a very large model is rarely an isolated GPU task. It is a full engineering system. Common building blocks include object storage for weights and intermediate artifacts, a container platform to orchestrate inference instances, a logging service to record call traces, monitoring and alerting for VRAM, latency, and error rate, and elastic resource scheduling for peak traffic. Cloud providers such as Baidu AI Cloud offer this class of general capability; enterprises can equally build a comparable setup from open-source components such as Kubernetes, object storage, Prometheus, Grafana, ELK, or OpenTelemetry.

    The value of this architecture is that it turns an unpredictable large-model runtime into a manageable distributed service: weights have unified storage and version control, inference instances scale elastically, call traces are traceable, resource watermarks trigger alerts, and failure recovery follows a standard process. For workloads that need to run continuously, this matters more than a one-off experimental deployment.

    12. Data Security and Compliance

    Local deployment of very large models also raises data security concerns. Weight files should have access permissions and encryption policies; logs should be masked; session state should be isolated; tool calls should be audited; and sensitive data should have a defined retention period. If the model will process an enterprise knowledge base, code repositories, or customer conversations, define data boundaries, permission boundaries, and compliance requirements up front so that deployment and operations do not introduce additional risk.

    Conclusion: From “Downloadable Weights” to “Sustainable Service”

    2.8T total parameters, 104B activated parameters, 896 experts, 1.56TB of weights, a 1M-token context, and a vision encoder together form a complex inference system. Open weights lower the barrier to obtaining the model; they do not lower the barrier to running it. 1.56TB of weights makes storage and VRAM the first constraint. The MoE structure makes scheduling and communication the key bottleneck. Long context means the KV cache must be managed precisely. Multimodality and agent capability mean a longer serving path with more failure points.

    So “you need 8 GPUs to run this locally” should be read as an engineering example, not a general rule: under a specific precision, context length, concurrency target, and topology, multi-GPU coordination is what makes a viable deployment possible. The mature approach is to close the loop across five layers: resource profiling, quantization evaluation, parallelism design, observable operations, and security governance. Once those engineering questions are resolved, a very large open-weight model moves from being downloadable weights to being a production service that can sustainably deliver capability.