Running Open-Source LLMs Locally: A Guide to Hardware Selection and Cost Estimation

    A practical guide to self-hosting open-source large language models. It covers why teams deploy locally, how to estimate GPU memory, three hardware tiers, deployment steps, and a three-year total cost of ownership comparison against cloud APIs.

    1. Why More Teams Are Considering Local Deployment

    Over the past two years, open-source large language models have closed the performance gap with closed-source models. A steady stream of high-quality open-weight models has turned “running the model on our own hardware” from a hobbyist project into a serious engineering decision. Companies usually have three motives:

    • Data security and compliance: Sensitive data stays inside the internal network, which avoids the compliance risks of sending data to third-party APIs.
    • Cost control: In high-frequency usage, pay-per-token API fees can add up over the years and eventually exceed the one-time cost of hardware.
    • Freedom to customize and integrate: Local deployment lets you fine-tune models, modify the inference pipeline, and integrate deeply with internal systems.

    However, local deployment is not as simple as downloading a weight file and running it. It is a combined hardware and cost engineering problem. The sections below break it down step by step.

    2. Start with the GPU Memory Budget: How Much Space the Model Needs

    GPU memory is the first barrier to local deployment and the most important cost variable. The commonly used estimation method is as follows.

    1. GPU memory for model weights

    1. Weight memory ≈ parameter count × bytes per parameter
    2. - FP16/BF16 (half precision): 2 bytes per parameter
    3. - INT8 quantization: 1 byte per parameter
    4. - INT4 quantization: 0.5 bytes per parameter

    Take a model with 70 billion parameters as an example:

    Precision Weight Memory Notes
    FP16 About 140 GB Does not fit on a single GPU. Requires multi-GPU parallelism.
    INT8 About 70 GB Feasible on two high-end GPUs.
    INT4 About 35 GB Fits on a single 48 GB GPU.

    2. Don’t forget runtime overhead

    During actual deployment, reserve memory for the following in addition to the weights:

    • KV cache: The longer the context, the more memory it uses. For long-context workloads (such as 32K tokens and above), it may use an additional 10% to 30% of GPU memory.
    • Activations and framework overhead: Typically reserve 5% to 10%.
    • Concurrency headroom: With multiple concurrent users, the KV cache grows in proportion to the number of sessions.

    Rule of thumb: GPU memory required ≈ weight memory × 1.2 to 1.5. Many failed deployments come from calculating only the weights and ignoring runtime overhead.

    3. Hardware Tiers: Three Typical Configurations

    Tier 1: Entry-Level Experimentation (Budget: RMB 10,000 to 30,000)

    • Configuration: One or two consumer GPUs with 24 GB of memory, paired with 32 GB or more of system RAM and a high-speed NVMe SSD.
    • What it can run: High-precision versions of models with 7 to 14 billion parameters, or INT4-quantized versions of 70-billion-parameter models.
    • Suitable for: Personal learning, technical proof of concept, and low-concurrency internal tools such as a personal knowledge base Q&A system.
    • Limitations: Limited concurrency and context length. Performance drops noticeably on long-text tasks.

    Tier 2: Team Production (Budget: RMB 150,000 to 400,000)

    • Configuration: Two to four professional GPUs with 48 GB of memory each, connected via NVLink or another high-speed interconnect, with 64 GB or more of system RAM, and redundant power supplies and cooling.
    • What it can run: INT8 or INT4 quantized deployments of 70-billion-parameter-class models, supporting daily use by a team of 10 to 50 people.
    • Suitable for: Production workloads at small and mid-sized companies, such as customer support chatbots, code assistance, and document processing.
    • Key point: This tier is the price-performance sweet spot. For most business use cases, the quality loss from 4-bit quantization is acceptable.

    Tier 3: Flagship On-Premises Deployment (Starting at RMB 1 million)

    • Configuration: A cluster of data center accelerator cards, networked across multiple machines, with dedicated cooling and power upgrades.
    • What it can run: Full-precision models with hundreds of billions to trillions of parameters.
    • Suitable for: Core private deployments at large organizations with strict requirements to keep data within their own infrastructure.
    • Practical note: Beyond hardware, this tier requires data center renovation, an operations team, and electricity. The total investment is far above the hardware price.

    4. Key Deployment Steps

    The standard deployment process is broadly similar across all tiers.

    Step 1: Obtain and verify the weights

    Download model weights from official sources. Always verify the file hashes to avoid strange errors caused by corrupted or incomplete weight files.

    1. # Illustrative workflow: verify file integrity after download
    2. sha256sum model-00001-of-000xx.safetensors
    3. # Compare each checksum with the one published by the official source

    Step 2: Choose an inference framework

    Mainstream open-source inference frameworks now support multi-GPU tensor parallelism, quantized inference, and OpenAI-compatible APIs. When choosing one, evaluate three things: whether it supports your GPU interconnect setup, how broad its quantization format support is, and how active its community is. Here is a general deployment configuration:

    1. # Illustrative config: multi-GPU quantized deployment
    2. model_path: /models/open-llm-70b
    3. quantization: awq # or gptq / fp8, depending on framework support
    4. tensor_parallel_size: 2 # Set parallelism based on the number of GPUs
    5. max_model_len: 32768 # Maximum context length
    6. gpu_memory_utilization: 0.92

    Step 3: Expose the service endpoint

    Most frameworks provide a server endpoint compatible with the common Chat API format out of the box, so business systems can usually connect with little or no modification.

    1. curl http://localhost:8000/v1/chat/completions \
    2. -H "Content-Type: application/json" \
    3. -d '{"model": "open-llm-70b", "messages": [{"role": "user", "content": "Hello"}]}'

    Step 4: Load test and tune

    Run concurrency load tests before going live. Focus on three metrics: time to first token (TTFT), throughput (tokens per second), and peak GPU memory usage. Adjust max_model_len, the concurrency limit, and the quantization strategy based on the results.

    5. Cost Estimation: Buying Hardware vs. Renting Compute

    The cost of local deployment goes well beyond the purchase price of hardware. It is recommended to evaluate it using a three-year total cost of ownership (TCO) model.

    1. Hardware depreciation: Server equipment is typically depreciated over three to five years. Accelerator cards are updated quickly, so a three-year schedule is a reasonable assumption.

    2. Power and cooling: A four-GPU server often draws more than 2,000 W at full load. Running it 24/7 for one year can cost tens of thousands of RMB in electricity alone. Data center cooling costs are separate.

    3. Operations staffing: At least one engineer familiar with inference services needs to maintain the system part-time. This includes troubleshooting, driver upgrades, and model updates.

    4. Opportunity cost: Buying hardware locks in a technology path. If a more cost-effective GPU or model appears within a year, the sunk cost cannot be recovered.

    Decision framework:

    Scenario Recommended Approach
    Low daily call volume, occasional use Call a cloud-hosted model service directly
    Sensitive data, stable call volume, team of 20 or more Local deployment, starting at Tier 2
    Sensitive data, but highly variable call volume Hybrid approach: a private cloud container platform handles sensitive workloads, and public cloud elastic compute covers peak demand

    For teams in the middle ground, mainstream cloud platforms offer container services and GPU instances. Teams can first validate how well a model fits their business on pay-as-you-go instances, then decide whether to move to owned hardware. Cloud elasticity keeps the cost of trial and error as low as possible.

    6. Closing Thoughts

    Deploying open-source LLMs locally is fundamentally a trade-off between data sovereignty, cost structure, and technical flexibility. Three practical recommendations:

    1. Quantize first, scale later: Start with an INT4-quantized version to validate results for your use case. Only evaluate higher precision or larger models once the benefits are confirmed.
    2. Overprovision GPU memory: Take the estimate from the formula and add 30% to the budget to leave room for concurrency and long contexts.
    3. Decide on TCO, not the purchase price: Include three years of electricity, operations staffing, and the risk of hardware becoming obsolete. The conclusion often differs sharply from what the sticker price suggests.

    Once you work through these numbers, “should we deploy locally?” becomes a question with a precise engineering answer, not a matter of guesswork.