Open-Source LLM Local Deployment: How to Choose Among Four Options and Avoid Pitfalls

    This guide walks through how to pick a local deployment approach for open-source large language models (LLMs). It explains when self-hosting makes sense versus staying on cloud APIs, the three core constraints (VRAM capacity, KV cache and context length, and MoE architecture), and four deployment options — quantized single/dual-GPU inference, single-node multi-GPU tensor parallelism, multi-node hybrid parallelism, and cloud-elastic containerized deployment. Along the way you get VRAM and KV estimation formulas, engine choices (vLLM, SGLang, TensorRT-LLM, llama.cpp/Ollama), configuration and command examples, serving and optimization techniques, acceptance metrics, a TCO cost framework, and a FAQ covering common pitfalls.

    1. First, Decide Whether to Self-Host: Cost, Compliance, and Latency

    Open-source LLM parameter counts are moving from the hundreds of billions toward the trillions, while context windows expand to support long documents, agent memory, and multi-turn tool calls. Cloud APIs bill separately for input and output tokens, so in high-frequency, long-context, or fixed-system-prompt workloads, cost grows roughly linearly with call volume. Once the monthly bill climbs noticeably, model selection stops being purely a technical question and becomes a financial one.

    The core value of local deployment is converting a token-based operating expense into a controllable hardware investment. Its advantages include: data stays within your own domain, which simplifies compliance; latency and rate limits are more controllable; and after stable large-scale use, per-inference cost can drop. But self-hosting does not mean zero cost — you still need to account for hardware depreciation, power, cooling, data-center space, networking, ops staff, and spare parts. So the real question compares “large-scale cloud spend” against the “total cost of ownership (TCO) of self-hosting.”

    Before deploying, answer three common questions: what to do when VRAM is insufficient, how many GPUs a 284B model needs, and what mistakes commonly occur in local deployment. This article gives estimation methods, selection boundaries, and a pitfall checklist for each.

    2. Three Core Constraints Before Deployment

    1. VRAM Capacity: The First Threshold

    Weight VRAM can be roughly estimated as:

    1. Weight VRAM ≈ total parameters × bytes per parameter

    BF16/FP16 uses about 2 bytes per parameter, INT8/W8A8 about 1 byte, and INT4/W4A16 about 0.5 byte. For a 284B model:

    • INT8: about 264.5 GiB;
    • INT4: about 132.2 GiB.

    This means a 284B model exceeds common single-GPU capacity even at INT4; a trillion-parameter model still needs about 745 GiB after INT4, requiring multi-GPU or even multi-node setups.

    MoE models activate only some experts per token, but expert weights usually must stay resident in VRAM. So VRAM capacity depends mainly on total parameters, not activated parameters. Expert parallelism (EP) distributes different experts across GPUs, scaling VRAM and compute horizontally.

    Priority order when VRAM is insufficient:

    1. Prefer INT4 quantization, and evaluate W8A8 for throughput-sensitive cases;
    2. Increase TP degree, or move from single-node to multi-node;
    3. Lower max_model_len and concurrency;
    4. Enable FP8/INT8 KV cache;
    5. For low-frequency or offline scenarios, try MoE expert CPU offloading, but latency rises and CPU memory bandwidth becomes the bottleneck.

    2. Context Length and KV Cache

    The KV cache grows linearly with sequence length:

    1. KV cache ≈ 2 × layers × KV heads × head dim × sequence length × concurrency × bytes per element

    The “×2” accounts for one copy each of K and V. GQA models have far fewer KV heads than attention heads, so their KV cache is markedly smaller.

    For a typical lightweight model — 60 layers, 8 KV heads, head dim 128, FP8 KV — the full KV per token is roughly:

    1. 2 × 60 × 8 × 128 × 1 Byte ≈ 0.117 MB

    A single 131072-token context uses about:

    1. 0.117 MB × 131072 ≈ 15.4 GiB

    With TP=8, the KV is sharded across 8 GPUs — about 0.015 MB per token per card, and about 1.9 GB per 128k context. Assuming 80 GB cards and loading a 284B INT8 model at TP=8, weights take about 33 GiB per card; at 0.9 VRAM utilization, about 72 GB is usable, leaving roughly 39 GB. By capacity alone, 39 GB ÷ 1.9 GB ≈ 20 concurrent 128k sessions. But this is only a theoretical ceiling — you must also subtract activations, temporary buffers, communication buffers, and VRAM fragmentation, so real concurrency must be validated against load-test logs.

    If a single card carries the full KV, reserving about 20% VRAM (~16 GB) per card supports only about 1 session of 128k, not “20 sessions.” So plan long-context concurrency by layers, KV heads, head dim, precision, and sequence length — not just a flat 20% VRAM per card.

    Common mitigations for KV bloat include FP8/INT8 KV quantization, PagedAttention, and Prefix Caching. Note that FP8 KV usually depends on newer hardware and engine versions (Hopper or Ada are better supported); on older cards, verify official compatibility before enabling it.

    3. MoE Deployment Characteristics

    “A trillion total parameters, tens of billions activated” is a typical MoE structure: each token passes through only some experts, so compute is close to the activated-parameter scale and speed does not slow proportionally to total parameters — but all expert weights still must fit in VRAM.

    This leads to three conclusions:

    • The VRAM threshold depends on total parameters;
    • Throughput depends on activated parameters and VRAM bandwidth;
    • Expert parallelism (EP) is the key to horizontal VRAM scaling for MoE.

    MoE deployment usually configures both dense-layer parallelism and expert-layer EP. Get basic parallelism working first, then gradually enable EP and tune.

    3. Four Deployment Options

    Option 1: Quantization + Single/Dual-GPU Inference

    Applies to: dense models of 70B and below, aggressively quantized lightweight MoE, proof-of-concept, workstations, or edge devices. This option is low-cost and fast to launch, but not suitable for high-concurrency, long-context production serving.

    Common engines:

    • llama.cpp / Ollama: GGUF quantization and fast low-resource validation;
    • vLLM: single- or multi-GPU serving with tensor parallelism support;
    • SGLang: multi-turn dialogue and high concurrency with fixed prefixes;
    • TensorRT-LLM: low latency and high throughput on NVIDIA hardware.

    vLLM single-GPU example:

    1. pip install "vllm==0.6.3"
    2. vllm serve /models/qwen3-32b-awq \
    3. --host 127.0.0.1 --port 8000 \
    4. --max-model-len 32768 \
    5. --gpu-memory-utilization 0.90 \
    6. --kv-cache-dtype fp8

    Dual-GPU example:

    1. vllm serve /models/qwen3-32b-awq \
    2. --host 127.0.0.1 --port 8000 \
    3. --tensor-parallel-size 2 \
    4. --max-model-len 32768 \
    5. --gpu-memory-utilization 0.90 \
    6. --kv-cache-dtype fp8

    Note: these commands bind only to the localhost address 127.0.0.1 by default. If you must expose the service, place it behind an internal load balancer or API gateway with authentication, rate limiting, and access control enabled. Do not expose it directly to the public internet.

    To verify, send a test request:

    1. curl http://127.0.0.1:8000/v1/chat/completions \
    2. -H "Content-Type: application/json" \
    3. -d '{"model":"qwen3-32b-awq","messages":[{"role":"user","content":"Introduce yourself in three sentences"}]}'

    Limitation: a 284B model at INT4 is about 132 GiB. A standard dual 80 GB setup (160 GB total) is very tight once you account for utilization, KV, activations, and communication buffers. This scale is better served by a single 96 GB+ card or a dual high-VRAM configuration. Trillion-scale models are not suited to this option.

    Option 2: Single-Node Multi-GPU + Tensor Parallelism

    Applies to: ~284B-class lightweight MoE, such as some 235B specs and GLM-4.5-Air. This is the mainstream production option that balances performance and ops cost.

    The core technique is tensor parallelism (TP): split each layer’s weight matrices across multiple GPUs, compute in parallel, then merge results via AllReduce. Within a node, GPUs are usually connected by NVLink or high-bandwidth PCIe, so communication overhead is relatively controlled.

    Example: a 284B INT8 model on 80 GB cards:

    TP Weight per card (approx.) Usable VRAM (approx.) Notes
    2 132 GiB 72 GB Not feasible
    4 66 GiB 72 GB Only ~6 GB left; lower bound for short context and low concurrency
    8 33 GiB 72 GB ~39 GB left; better for long context and mid-to-high concurrency

    Key constraints:

    • TP is usually set to the number of GPUs in a single node;
    • TP must evenly divide the number of attention heads and KV heads;
    • If KV heads are fewer than TP, some engines may replicate KV heads, increasing KV VRAM — check official documentation and startup logs;
    • MoE models also need the expert intermediate dimension to be divisible by TP;
    • Whether GPTQ/AWQ support TP=4 or TP=8 depends on vLLM, SGLang, TensorRT-LLM, and the specific quantization implementation. The safe approach is to check the official support matrix and validate TP=1/2 first; for high TP prefer FP8 or BF16, and optionally try dynamic INT4 if the engine supports it — but always measure;
    • Deploy TP8 within the same NVLink domain; in PCIe environments, set parameters like NCCL_P2P_DISABLE=1 to troubleshoot P2P issues.

    Example configuration:

    1. model_path: /models/llm-284b
    2. precision: fp8
    3. tensor_parallel: 4
    4. max_seq_len: 131072
    5. kv_cache_dtype: fp8
    6. gpu_mem_util: 0.90

    Do not max out gpu_memory_utilization — reserve room for KV cache, activations, CUDA context, and communication buffers.

    Option 3: Multi-Node Multi-GPU + Hybrid Parallelism

    Applies to: trillion-scale flagship MoE that a single node cannot host, requiring multi-node multi-GPU.

    The core combination includes:

    • Intra-node TP: splits weight matrices, suited to NVLink domains;
    • Cross-node PP: pipeline parallelism, placing different layers on different nodes;
    • EP: expert parallelism, distributing MoE experts across GPUs.

    Parallelism must satisfy:

    1. total GPUs = num_nodes × gpus_per_node
    2. dense part: TP × PP × DP = total GPUs

    For example, 4 nodes × 8 GPUs = 32 total. You can use TP8 × PP4 × DP1. EP applies to MoE layers and requires the total GPU count to be divisible by EP; the expert count should also be divisible by EP where possible. An EP group is usually formed from GPUs within the same PP stage; if the EP group is split across stages, engine support requirements are higher, and beginners should avoid it.

    Example configuration:

    1. model_path: /models/llm-1.6t
    2. tensor_parallel_size: 8
    3. pipeline_parallel_size: 4
    4. expert_parallel_size: 8
    5. num_nodes: 4
    6. gpus_per_node: 8
    7. interconnect: RDMA / high-speed Ethernet

    Rollout advice: disable EP first and get TP8 × PP4 working; then gradually enable EP2, EP4, and EP8, load-testing with identical requests at each step to compare throughput, latency, and VRAM.

    On networking: keep TP within a node, and use RDMA or 100 Gbps+ Ethernet for cross-node communication. Otherwise, PP transfer, EP All-to-All, and KV migration all become bottlenecks. As nodes increase, the probability of single points of failure rises, so pair the setup with health checks, automatic restarts, and monitoring/alerting.

    Option 4: Cloud Elasticity + Containerized Deployment

    Applies to: teams without a self-built data center, that need elastic scaling and quick start/stop, or that want a disaster-recovery backup for a self-hosted cluster. It carries a lower ops burden and suits bursty workloads, but is not ideal for scenarios chasing extreme low unit cost at long-term full load.

    Typical architecture:

    1. Access layer: load balancer / API gateway — authentication and rate limiting
    2. Service layer: containerized inference instances with autoscaling
    3. Storage layer: object storage for weights, logging service for access logs
    4. Ops layer: monitor GPU utilization, VRAM, latency, and error rate

    Rollout checklist:

    • Use a least-privilege sub-account for object storage, granting read access only to the specified bucket;
    • Use STS for temporary credentials, and pull weights through VPC internal endpoints;
    • Place inference workloads in a private subnet, exposed via load balancer or gateway;
    • Install the NVIDIA driver, containerd, and nvidia-container-toolkit on nodes;
    • Pin engine, CUDA, and PyTorch versions in the base image;
    • Store weights on shared storage or node-local disks to avoid each Pod re-downloading them;
    • Base autoscaling on queued requests or concurrency rather than GPU utilization alone;
    • Warm up before peaks; drain traffic before scaling down, then terminate instances.

    4. Serving, Optimization, and Acceptance

    For the serving interface, use an HTTP API compatible with the OpenAI format for easy client integration. Example:

    1. curl http://127.0.0.1:8000/v1/chat/completions \
    2. -H "Content-Type: application/json" \
    3. -d '{"model":"local-llm","messages":[{"role":"user","content":"Introduce yourself"}],"stream":true}'

    General optimization techniques include:

    • Continuous batching: dynamic batching to raise GPU utilization;
    • PagedAttention: on-demand KV paging to reduce fragmentation;
    • Prefix Caching: reuse fixed system prompts or multi-turn dialogue prefixes;
    • Speculative decoding: a small draft model plus large-model verification to lower decode latency;
    • KV quantization: reduce the cost of long-context concurrency.

    Recommended acceptance metrics:

    Metric Meaning Reference target
    TTFT Time to first token ~1 s at P95 for short inputs
    TPOT Time per output token Set according to interaction requirements
    Total throughput tokens/s under concurrency Baseline by peak concurrency
    VRAM usage Peak VRAM / per-card capacity Keep below 90% to avoid OOM
    Stability Sustained load test At least 2 hours with no OOM and no marked error-rate increase

    For load testing, use a fixed request set covering short Q&A, long documents, multi-turn dialogue, and different concurrency levels. You can use tools such as vllm bench serve or genai-perf; with max_model_len fixed, find the maximum usable concurrency under the target SLA.

    Security recommendations:

    • Enable API Key, JWT, or gateway authentication on external interfaces;
    • Do not bind to a public address by default;
    • Mask PII such as phone numbers, ID numbers, and emails in logs, and set retention periods;
    • Comply with model licenses — pay attention to commercial-use, redistribution, and naming requirements.

    5. Selection Advice and TCO Cost Framework

    Comparison of the four options:

    Option Hardware threshold Applicable scale Ops complexity Recommended scenario
    Single-GPU quantization Low 70B and below, or lightweight MoE validation Low Proof-of-concept, edge deployment
    Single-node multi-GPU Medium ~284B class Medium Mainstream production first choice
    Multi-node multi-GPU High Trillion-scale High Flagship model self-hosting
    Cloud elastic Pay-as-you-go Flexible Low Elasticity, disaster recovery, ops-free

    TCO calculation:

    1. Self-hosted annual cost = hardware investment / depreciation years + annual power + data center & bandwidth + ops staff + maintenance & spares
    2. Annual power = total system power (kW) × PUE × 8760 h × electricity price (per kWh) × utilization rate
    3. Cost per token = self-hosted annual cost / (daily effective tokens × 365 × utilization)

    Note: effective tokens should exclude failed requests, retries, cache hits, and duplicate requests, or you will mis-estimate the unit cost.

    Example: an 8-GPU server at about 1.2 million CNY, depreciated over 4 years, is 300,000 CNY/year. If the whole system draws about 10 kW, PUE is 1.3, electricity is 0.7 CNY/kWh, and utilization is 80%, annual power is roughly:

    1. 10 × 1.3 × 8760 × 0.7 × 0.8 ≈ 64,000 CNY

    Adding about 30,000 CNY for data center and bandwidth and about 100,000 CNY for ops staff, the total is about 494,000 CNY/year.

    Decision path: first validate the model’s value cheaply with the single-GPU quantization option; once confirmed, use single-node multi-GPU for lightweight specs and multi-node hybrid parallelism for flagship specs; add the cloud containerized option if you need elasticity, ops-free operation, or disaster recovery.

    6. FAQ

    What to do when VRAM is insufficient? Try in order: INT4 quantization, raise TP or go multi-node, lower max_model_len and concurrency, enable FP8/INT8 KV cache, and finally consider MoE expert CPU offloading.

    Local deployment vs. cloud API? High-frequency, stable, data-sensitive, long-term workloads lean toward self-hosting; low-frequency, bursty workloads that want fast launch and ops-free operation lean toward cloud.

    Does quantization hurt quality? W4A16 has limited loss on most general tasks, but for reasoning tasks like math and code, run your business eval set first and consider QAT calibration if needed.

    What are common local-deployment mistakes? Weight format not matching TP, TP × PP × DP not equaling the total GPU count, EP not dividing evenly, cross-node traffic over ordinary Ethernet, maxing out VRAM utilization, driver and engine version drift, and overly optimistic KV concurrency estimates.

    What if throughput is low? First check TP communication and cross-node networking, then tune continuous batching and concurrency limits, then look at prefix cache hit rate, and finally evaluate speculative decoding.

    NCCL errors in multi-node deployment? Check the NIC and RDMA configuration, for example NCCL_IB_HCA and NCCL_SOCKET_IFNAME, and confirm firewall ports and driver versions. Validate single-node multi-GPU first, then scale to multi-node.