Local LLM Deployment: A Practical Guide to Model Selection and VRAM Planning
This guide explains how to choose and deploy large language models locally by turning model selection into a constraint-driven decision. It covers task definition, VRAM estimation including KV cache and concurrency, parameter-class choices by scenario, quantization trade-offs, deployment models, model provenance and supply-chain governance, a validation-to-production workflow, evaluation, operations, and common pitfalls. The conclusion is a practical selection sequence and a balanced approach based on runnability, evaluation, maintainability, and compliance rather than maximum parameter count.
Local LLM Deployment: A Practical Guide to Model Selection and VRAM Planning
Deploying a large language model locally is not simply a matter of finding a set of model weights and copying it to a machine. It involves four questions at the same time: first, whether the model can run; second, whether quality meets the business scenario; third, whether cost and latency are acceptable; and fourth, whether the system can be maintained, audited, and kept compliant over time. Many teams focus only on parameter scale and download channels during model selection, while ignoring context length, concurrency, quantization loss, inference runtime overhead, licensing, and supply-chain governance. As a result, the validation environment works but production does not, or a short demo is possible but long-term maintenance is difficult.
A more reliable approach is to turn “which model do you recommend?” into “how should models and deployment options be selected under given constraints?” The following sections discuss tasks, hardware, quantization, deployment forms, model sources, evaluation, and operations.
1. Define the Problem First: Inference, Fine-Tuning, or Knowledge QA
Local LLM deployment has at least three typical use cases, and their resource models are completely different.
Inference only: This includes general chat, summarization, information extraction, classification, structured output, coding assistance, and retrieval-augmented question answering. Resources are mainly determined by model weights, KV cache, activations, concurrent batches, and runtime overhead. This is the form most teams deploy first.
Fine-tuning or continued training: This includes domain adaptation, instruction tuning, and parameter-efficient fine-tuning. Training requires not only weights but also gradients, optimizer states, intermediate activations, and checkpoints. VRAM requirements are usually much higher than for inference. Even with parameter-efficient fine-tuning, space must be reserved for long samples, larger batches, and validation-set inference.
Retrieval-augmented and tool-calling orchestration: The model itself may not be large, but the system also includes embedding models, rerankers, document parsing, retrieval services, access control, and audit logs. In this case, “local LLM deployment” is a system problem and cannot be estimated solely from a single LLM.
Therefore, the first step in model selection is not to ask “which model is strongest?” but to clarify task type, input and output length, concurrency, latency targets, data boundaries, whether fine-tuning is needed, whether internet access is allowed, whether multimodality is required, whether structured output is needed, and whether long-term auditing is required. Write these constraints into an acceptance checklist, and evaluate every model candidate against that checklist.
2. VRAM Estimation: Parameter Scale Is Only the Starting Point; KV Cache and Concurrency Are Often Underestimated
Whether a model can be deployed locally is usually constrained by accelerator VRAM, but VRAM is not simply “parameter count multiplied by precision.” For inference-only scenarios, peak VRAM can be roughly broken down into:
- Model weights;
- KV cache;
- Activations and temporary workspace;
- Runtime and framework overhead;
- VRAM fragmentation and peak fluctuations;
- Additional communication buffers introduced by multi-GPU parallelism.
The weight portion alone can be roughly estimated as follows:
weight VRAM ≈ number of parameters × bytes per parameter16-bit precision: about 2 bytes per parameter8-bit quantization: about 1 byte per parameter4-bit quantization: about 0.5 bytes per parameter
For example, a 70B–80B parameter model requires about 140GB–160GB for 16-bit weights, about 70GB–80GB for 8-bit, and about 35GB–40GB for 4-bit. But this is only the weights. It does not yet include KV cache growth from longer context, concurrent batches, long-input activations, and runtime reserve. In practice, deployments usually need a 20%–50% safety margin, and long-context, high-concurrency scenarios may need even more.
The table below gives rough weight-only estimates for initial screening. It cannot replace real load testing.
| Parameter class | Approx. 16-bit weight-only range | Approx. 8-bit weight-only range | Approx. 4-bit weight-only range | Typical use note |
|---|---|---|---|---|
| 1B–3B | 2GB–6GB | 1GB–3GB | 0.5GB–1.5GB | Edge devices, classification/extraction, low-concurrency assistants |
| 7B–8B | 14GB–16GB | 7GB–8GB | 3.5GB–4GB | Personal development, small-team general chat, summarization, RAG |
| 13B–14B | 26GB–28GB | 13GB–14GB | 6.5GB–7GB | Quality-oriented small services, complex instruction following |
| 30B–34B | 60GB–68GB | 30GB–34GB | 15GB–17GB | Multi-GPU or high-VRAM devices, complex analysis and generation |
| 70B and above | 140GB+ | 70GB+ | 35GB+ | Enterprise multi-GPU/multi-node, core scenarios after rigorous evaluation |
It is especially important to note that long contexts significantly amplify KV cache pressure. The same 7B–8B parameter model may run on consumer-grade VRAM with short context and low concurrency; once context expands to tens of thousands of tokens, concurrency rises, and batches grow, VRAM and bandwidth pressure increase rapidly. KV cache is usually related to the number of layers, hidden dimension, context length, concurrent batch size, and cache precision. Grouped-query attention, multi-query attention, or cache quantization can reduce pressure, but they also introduce trade-offs in quality and implementation complexity.
In addition, VRAM does not simply add up in multi-GPU deployments. Tensor parallelism, pipeline parallelism, communication buffers, load balancing, and failure recovery all introduce extra costs. Multiple GPUs are suitable for improving throughput or hosting larger models, but one should not assume that “two GPUs always equal twice the usable VRAM.”
3. Choose Parameter Classes by Scenario: Do Not Use One Model for Everything
In real engineering, a more reasonable strategy is to build a portfolio of models rather than pursuing one “all-powerful largest model.”
3.1 Personal Development and Edge Validation
Models in the 1B–3B parameter class are suitable for prototype validation, simple classification, information extraction, offline summarization, and low-concurrency assistants. They can run in quantized form on CPU-only machines, unified-memory devices, or devices with small VRAM, with low cost and fast iteration. However, these models are usually limited in complex reasoning, long-document consistency, and multi-turn tool calling. They are suitable as process validation or specialized small-task models, not as direct owners of high-value complex QA.
If more general chat and summarization capability is needed, consider 4-bit or 8-bit quantized versions of 7B–8B parameter models. These models usually have relatively friendly VRAM requirements and are suitable for personal workstations and small-team validation, but context length, concurrency, and output stability still need to be measured.
3.2 Small-Team Knowledge QA and Business Assistance
A 7B–8B parameter model combined with embedding models, rerankers, document parsing, and access control can build local knowledge QA, customer-support assistance, ticket summarization, and policy retrieval systems. The recommendation logic here is not “the larger the model, the better,” but rather prioritizing retrieval quality, traceable citations, structured-output validation, and low latency. If knowledge-base documents have complex formats, parsing and reranking may have a greater impact on final quality than increasing model parameter scale.
For scenarios with higher quality requirements and low concurrency, evaluate 4-bit quantized or multi-GPU 16-bit options for 13B–14B parameter models. These models usually have more headroom for complex instruction following, long-document summarization, and multi-step analysis, but they must be validated against a domain evaluation set to confirm whether they truly outperform smaller models. Parameter scale alone is not enough.
3.3 Enterprise Production and Complex Analysis
Models above 30B parameters usually require multiple GPUs or high-VRAM devices and are suitable for complex analysis, high-quality generation, coding assistance, and cross-document reasoning. Mixture-of-experts (MoE) architectures can activate only a subset of parameters during inference, reducing computation, but weight storage remains close to the total parameter scale. Do not mistake “fewer activated parameters” for “lower VRAM requirements.” Enterprise selection should focus on peak concurrency, p95 latency, long-context stability, post-quantization quality, failure fallback, version rollback, and total cost of ownership.
Models at 70B and above should generally be attempted only by teams with clear business benefits, multi-GPU/multi-node operations capability, and a rigorous evaluation loop. If the core requirement is only internal knowledge QA, using a medium-sized model, optimizing retrieval and reranking, and strengthening prompts and structured constraints is often more cost-effective than moving directly to an extremely large model.
3.4 Multimodal and Specialized Models
Capabilities such as image understanding, speech recognition, speech synthesis, and OCR do not necessarily need to be handled by one extremely large model. A more reliable engineering approach is to split by capability: deploy specialized vision encoders, speech models, and language models separately and orchestrate them through a gateway. This makes independent scaling, independent evaluation, and fault isolation easier, and avoids concentrating all load in the same resource pool. Multimodal models also require additional consideration of preprocessing and caching overhead from image resolution, video frame count, and audio duration.
4. Quantization Is Not One-Click Compression: Use an Evaluation Loop to Trade for VRAM
4-bit and 8-bit quantization can significantly reduce weight VRAM, but it is not lossless. Quantized models may degrade in factuality, long-tail knowledge, format stability, multilingual capability, math, and coding tasks. The degree of degradation depends on the quantization method, calibration data, runtime implementation, and task type. In engineering, follow these principles:
Define the quantization goal first: Is it to fit the model into existing hardware, improve throughput, or reduce memory-bandwidth pressure? Different goals require different quantization strategies and acceptance metrics.
Compare full precision and quantized versions on a domain evaluation set: Cover at least core tasks, long context, structured output, refusal, and safety scenarios. Do not judge with only a few general QA examples.
Separate weight VRAM from cache VRAM: 4-bit weights can reduce model residency memory, but with long contexts the KV cache may become the main bottleneck. When necessary, evaluate cache quantization, context truncation, segmented summarization, and retrieval augmentation.
Keep a fallback path: Production environments should retain a higher-precision or smaller-model fallback so the system can switch when the quantized version produces abnormal output, latency rises, or resources are tight.
Record complete artifacts: Model weights, tokenizer, quantization configuration, inference runtime version, evaluation-set version, and evaluation results should be archived together to ensure reproducibility.
5. Deployment Forms: Local, Hybrid, and Edge–Center Collaboration
Local deployment does not have only one form: “everything offline.” Common forms include:
- Pure local single machine: Data does not leave the internal network. Suitable for validation, personal workstations, and small-scale services. Pay attention to single points of failure, backup, and upgrade management.
- Local server or cluster: Suitable for team-shared inference services. Build a unified platform through containerization, artifact repositories, gateways, authentication, auditing, and monitoring.
- Edge–center collaboration: The edge runs small models for low-latency preprocessing and privacy filtering, while the center runs larger models for complex analysis. Design offline fault tolerance, model synchronization, and data masking.
- Cloud validation, local production, or hybrid deployment: When hardware is lacking, use isolated validation environments built from general cloud compute, containers, object storage, logs, and monitoring before migrating. For example, general-purpose compute, containers, object storage, logging, and monitoring/alerting services from a cloud provider such as Baidu AI Cloud can be used to build a validation environment. Production decisions should still be based on data boundaries, cost, service-level objectives, and measured results when choosing self-hosted, cloud, or hybrid deployment. This article makes no commitments about specific APIs, billing, or performance.
Regardless of the form, it is advisable to split the inference service into layers: model artifacts, inference runtime, service gateway, authentication and auditing, monitoring and alerting, and evaluation regression. Model artifacts should be immutable, with trackable versions and hashes. The gateway handles rate limiting, timeouts, retries, and fallback. Monitoring covers resources, latency, throughput, error rate, and security alerts.
6. Model Sources and Supply-Chain Governance: Available Does Not Mean Production-Ready
Public model repositories and distribution channels can solve the “acquisition” problem, but production environments should not treat public downloads as a default process. Build a controlled artifact repository:
- Obtain models only from approved sources, and record version, hash, license, model description, and data-source description;
- Perform integrity verification on model weights, tokenizers, configurations, inference runtimes, and container images;
- Maintain a software bill of materials (SBOM) that records dependencies, vulnerability scan results, and license obligations;
- Prohibit production environments from directly pulling unreviewed public artifacts; when necessary, use offline import or controlled image synchronization;
- Run regression evaluation and canary releases for model updates, and retain the ability to roll back to old versions;
- Clarify commercial licensing, redistribution, ownership of fine-tuned weights, responsibility for output content, and data-processing boundaries.
“Downloadable” does not mean “commercially usable,” and “runs successfully” does not mean “auditable.” Especially in enterprise scenarios, model-source governance and license review should move forward to the selection stage rather than being remediated after launch.
7. From Validation to Production: A Reusable Process
7.1 Define Acceptance Metrics
First write down business metrics and engineering metrics: task accuracy, citation correctness, structured-output pass rate, hallucination rate, correct refusal rate, first-token latency, token throughput, p95 response time, concurrency capacity, peak VRAM, cost per request, and failure fallback time. Without acceptance metrics, model selection degrades into subjective experience.
7.2 Build a Domain Evaluation Set
The evaluation set should include normal questions, boundary cases, long documents, multi-turn context, adversarial inputs, privacy-sensitive inputs, format constraints, and failure cases. Combine automated evaluation with human blind review, and record model version, prompts, parameters, and evaluation environment.
7.3 Do Resource Screening First, Then Real Load Testing
The following pseudocode can be used for rough estimation, but final decisions must be based on load testing with real input lengths and concurrency:
function estimate_memory(total_params, weight_bits, context_len, batch_size,kv_bytes_per_token, activation_bytes,runtime_overhead_ratio, safety_factor):weight_bytes = total_params * weight_bits / 8kv_bytes = context_len * batch_size * kv_bytes_per_tokenbase = weight_bytes + kv_bytes + activation_bytesreturn base * (1 + runtime_overhead_ratio) * safety_factor
Here, kv_bytes_per_token should be obtained from actual model configuration and runtime measurement, not replaced by a rule-of-thumb constant. Long-context, high-concurrency, and dynamic-batch scenarios especially require measured peaks.
7.4 Configuration Example and Canary Strategy
Below is a neutral deployment configuration sketch to show which key items should be recorded:
model_artifact:name: internal_approved_candidateversion: locked_by_hashlicense: reviewedprecision: quantized_or_full_precisionruntime_policy:max_context_tokens: 8192max_concurrency: 4kv_cache_policy: measure_then_limitstructured_output_validation: enabledservice_policy:timeout_ms: based_on_measured_p95_with_safety_marginrate_limit: enabledaudit_log: enabledfallback_model: readyrollback_version: previous_stableevaluation:offline_set: domain_cases_v1online_canary: small_percentagemetrics: [quality, first_token_latency, p95_latency, error_rate, safety]
Launch with a small-traffic canary, verify latency, VRAM, and error rate under the real request distribution, and then expand gradually. Model updates must trigger evaluation regression and must not directly replace production artifacts.
8. Evaluation System: Quality, Performance, Security, and Cost Together
Quality evaluation: Cover task accuracy, factual consistency, long-document summarization coverage, citation traceability, structured-output parse rate, and multi-turn context retention. For knowledge QA, evaluate retrieval recall and reranking quality separately; do not attribute all system problems to the LLM.
Performance evaluation: Focus on first-token latency, token throughput, p95 latency, concurrency limit, peak VRAM, and long-context degradation. Good average latency does not mean production readiness; tail latency and timeout rate are more critical.
Security evaluation: Include prompt injection, boundary-breaking inducement, sensitive-information leakage, unauthorized tool calls, malicious file parsing, and output-content compliance. Tool-calling scenarios need allowlists, parameter validation, and execution sandboxes.
Cost evaluation: Calculate not only hardware purchase but also power, cooling, operations, model updates, evaluation labor, failure costs, and opportunity cost. A small model plus optimized retrieval is often more economical than full generation with a large model.
9. Operations and Common Pitfalls
Production environments should monitor resource utilization, request volume, error rate, timeout rate, first-token latency, token throughput, cache hits, model version, and audit logs. Set up resource isolation, rate limiting, timeouts, retries, fallback, and rollback. Model serving should not run co-located with training tasks without isolation; otherwise training spikes can easily cause inference timeouts.
Common pitfalls include: looking only at parameter scale and not at context and concurrency; selecting hardware only by weight VRAM while ignoring KV cache and runtime overhead; directly using an unevaluated quantized model; pulling public artifacts directly in production; treating full fine-tuning as a task within the inference budget; ignoring licenses and data boundaries; using one model for all tasks; lacking fallback models and rollback versions; and having no real business evaluation set, making decisions only on demo performance.
10. Conclusion: Recommend a Decision Method, Not a Fixed Model Name
Returning to the question “what LLM can be deployed locally?”, the answer depends on hardware VRAM, context length, concurrency, quantization precision, task complexity, and compliance requirements. An executable recommendation sequence is:
- First use 1B–3B parameter models to validate the workflow and edge feasibility;
- Use 7B–8B parameter models for personal development, small-team knowledge QA, and general business assistance;
- When quality is insufficient, then evaluate quantized or multi-GPU options for 13B–14B parameter models;
- Enter 30B+ parameter or multi-node deployment only when there is clear business benefit and operations capability;
- Build in parallel a model artifact repository, domain evaluation set, monitoring and alerting, canary and rollback, and security auditing.
The core of local deployment is not chasing the largest parameter count, but finding the balance point where the system is runnable, evaluable, maintainable, and compliant. For most teams, a smaller model combined with high-quality data, retrieval augmentation, structured constraints, and rigorous evaluation is often more reliable than blindly moving to a very large model, and it is easier to iterate continuously.
