A Practical Guide to Selecting and Quantizing Open-Source LLMs for On-Premises Deployment

1. The Decision Starting Point: What Does On-Premises Deployment Actually Solve?

Before discussing “which open-source LLM to install,” answer a more fundamental question: does your scenario really require on-premises deployment? The essence of local deployment is to consolidate inference compute and data entirely within your own environment. The benefits fall into three dimensions:

1. Data privacy and compliance. For sensitive industries such as finance, healthcare, and legal, sending raw data to external services means an uncontrollable exposure surface. Local inference guarantees at the architectural level that “data never leaves the domain,” satisfying the principle of minimal exposure and making it easier to pass compliance and industry regulatory audits.

2. Controllable marginal cost. Cloud APIs bill by token, and usage grows linearly with business volume; with local inference, after a one-time hardware investment, the marginal cost per request approaches zero. For high-throughput, 24/7 batch processing, monitoring, and daily report tasks, an on-premises solution typically has a clear break-even point on the cost curve.

3. Low latency and availability. Inference completes within the same machine, so network jitter, service rate limiting, and regional failures no longer affect the链路 (link); for intranet tools, offline environments, and edge nodes, local deployment may even be the only option.

Conversely, if the workload is low-frequency, bursty, and requires very large parameter-scale models, cloud elastic compute (for example, GPU cloud servers and model inference services offered as general categories by Baidu AI Cloud) may be more economical. A pragmatic architecture is often a hybrid of “local-first, cloud fallback.”

2. Five Core Dimensions of Model Selection

Once localization is confirmed, selection can be broken down into five quantifiable dimensions to avoid falling into “leaderboard worship”:

1. Parameter scale and architecture. Mainstream open-source models span the 7B to 100B+ parameter range, with architectures divided into dense and Mixture-of-Experts (MoE). Dense models have a simple structure, predictable inference behavior, and a mature quantization ecosystem, making them a solid starting point for local deployment; MoE spreads computation by “activating only some experts on demand,” but VRAM must hold all parameters, which is actually less friendly to local hardware. Rough rules of thumb: 16GB of VRAM can carry a 14B-class quantized model and barely accommodate a 27B-class one (depending on fine-grained quantization); 24GB can comfortably carry 32B; 70B-class models generally require 48GB or more of VRAM or multi-GPU parallelism.

2. License. Be sure to confirm that the model weights license permits commercial use and redistribution. Permissive licenses such as MIT and Apache 2.0 are the most worry-free; some models attach restrictions on monthly active user scale or industry use, which must be checked clause by clause before commercial use.

3. Context length and modality. Long document analysis and codebase understanding require 128K-class context; note that the longer the context, the larger the KV Cache VRAM footprint, which must be included in the budget. When mixed image-text input is involved, the additional overhead of the vision encoder must also be evaluated.

4. Toolchain compatibility. Prioritize weight formats and tokenization schemes widely supported by mainstream inference frameworks and quantization toolchains—ecosystem compatibility means problems have already been answered before, and future migration costs are lower.

5. General and domain capability. Public benchmarks only reflect general-level performance; what the business ultimately cares about is domain performance. It is recommended to build your own evaluation set of 50–200 real samples covering the actual input distribution, to serve as the final arbiter.

3. VRAM Budget: An Arithmetic Problem

VRAM usage can be estimated with an approximate formula:

Weight VRAM ≈ parameter count × bytes per parameter
  FP16/BF16  → 2 bytes/parameter
  INT8       → about 1 byte/parameter
  4-bit quantization → about 0.5 bytes/parameter (plus 10%–20% after group scaling and high-precision layers)

Take a 27B-parameter dense model as an example: FP16 is about 54GB, which consumer GPUs cannot hold; 8-bit is about 27GB, still exceeding a 24GB GPU; 4-bit starts at about 13.5GB, and after overhead is about 15–17GB, which is very tight for a 16GB GPU and leaves ample headroom on a 24GB one.

In addition to weights, reserve space for: KV Cache (grows linearly with context length and concurrency, and may take several more GB in long-context scenarios), runtime buffers, and framework overhead (about 1–2GB). The budgeting principle is to keep total usage within 80%–90% of VRAM; otherwise VRAM swapping is triggered and throughput drops sharply.

4. Quantization: The Key Technique for Fitting a 27B Model onto a 16GB GPU

Quantization is an unavoidable part of local deployment, and the core tension is how to balance compression ratio against accuracy loss.

1. Uniform quantization. The traditional approach applies the same precision to all layers (e.g., all 4-bit); it is simple to implement and has good generality, but “one-size-fits-all” ignores differences in quantization sensitivity across layers. For a 27B model, a common 4-bit scheme produces an on-disk size of about 17GB, which a 16GB GPU struggles to hold directly.

2. Layer-wise mixed quantization. A more refined approach allocates precision differentially based on layer sensitivity: embedding layers and output heads are most sensitive to precision and retain higher bit widths; attention projections and FFN backbones are robust to quantization and can be compressed to 4-bit or even lower; channels with significant outliers use group scaling or per-channel scaling to suppress error amplification caused by outliers. According to the general conclusions of public research and industry practice, a carefully designed mixed 4-bit scheme can keep perplexity and downstream task metric loss to around 1%, while compressing a 27B dense model to about 13GB—20%–25% smaller than uniform 4-bit. This directly changes the hardware threshold: a 16GB consumer GPU goes from “can’t run it” to “can run it.”

3. Accuracy validation. Regardless of the quantization method used, regression validation should be performed: compare the FP16 baseline against the quantized version on a fixed evaluation set (perplexity, task accuracy, manual spot checks of key cases), and confirm the loss is within business tolerance before going live. The principle of quantization is “validated is what counts,” not “compressed and done.”

5. Domain Capability: A General Model ≠ a Qualified Industry Model

Take the financial scenario as an example: its tolerance for hallucination is extremely low—a fabricated metric definition or a made-up regulatory clause carries a cost far higher than ordinary stylistic flaws. Therefore, tasks such as financial Q&A, quantitative programming, and tool calling require domain work layered on top of a general foundation:

1. Continued domain corpus training and instruction fine-tuning: inject market data, research reports, regulations, code, and other corpora so that the model’s prior distribution aligns with domain vocabulary and reasoning paradigms;
2. Tool-calling alignment: financial tasks rely heavily on real-time data and deterministic computation; the model needs to stably output structured calls (querying market data, executing calculations, reading reports), leaving “computation” to programs and “understanding and orchestration” to the model;
3. Domain evaluation set as a guardrail: regression testing after each fine-tuning and quantization to prevent the “seesaw effect” where general capability rises while domain capability degrades.

In practice, at the same parameter scale, a domain-tuned model often significantly outperforms a larger but unadapted general model on vertical tasks.

6. Inference Quality Engineering: Let the Model “Self-Check Before Answering”

The quality of a single generation has an upper bound; a common engineering compensation is the reflection iteration loop: complex tasks automatically enter a multi-round “execute—evaluate—revise—verify” process, and output is returned to the user only after self-verification; simple Q&A answers directly without wasting compute. A practical scheduling pseudocode is as follows:

def answer(task):
    plan = decompose(task)                # break down the problem
    if is_simple(plan):                   # answer simple tasks directly
        return generate(task)
    for round in range(MAX_ROUNDS):       # complex tasks enter the loop
        draft  = execute(plan)            # execute
        report = evaluate(draft, plan)    # evaluate correctness and completeness
        if report.passed:
            return verify_and_output(draft)  # output only after verification passes
        plan = revise(plan, report)       # reflect and adjust the plan
    return escalate_to_human(task)        # escalate to a human if rounds are exhausted

Key design points: trigger conditions must be decidable, using rules or classifiers (whether multi-step reasoning is involved, whether computation and external data are involved) to decide whether to enter the loop; the evaluator must be concrete, with checks landing on verifiable signals (numbers match the calculator, citations actually exist, code passes static checks); set a round limit and a fallback path, escalating to a human or explicitly reporting failure when the limit is exceeded—this is more responsible than “confidently giving a wrong answer.” This mechanism essentially injects the workflow of a senior engineer—decompose, evaluate, execute, reflect, optimize—into the inference process, trading controllable extra compute for a significant improvement in accuracy and executability on complex tasks.

7. Serving and Operations: From “It Runs” to “It’s Reliable”

1. Standardized interface. Wrap inference as an industry-standard chat-completion-style REST interface, shielding underlying model switches and making it easy for upstream applications to integrate;
2. Resource governance. Set concurrency limits, request queues, and timeout circuit breakers to avoid long-context requests blowing up VRAM; the general scheduling approach of continuous batching can significantly improve throughput;
3. Observability. Integrate logging, monitoring, and alerting: QPS, first-token latency, generation throughput, VRAM watermark, and quantized output anomaly rate are all must-watch metrics;
4. Version management. Bind weights, quantization configuration, and evaluation reports to the same version number; any change must first pass evaluation-set regression to eliminate “silent model swaps”;
5. Elastic fallback. When local compute is insufficient or a larger parameter scale is needed, part of the traffic can be shifted to cloud compute (such as the GPU compute and model service categories of Baidu AI Cloud), forming a hybrid architecture where local deployment preserves privacy and the cloud supplements compute.

8. Deployment Checklist

• Clarify the drivers for localization: privacy / cost / latency—hit at least one and quantify it;
• Match parameter scale with VRAM budget: weights + KV Cache + overhead ≤ 90% of VRAM;
• License checked and confirmed to permit commercial use and the expected distribution method;
• Quantization scheme has passed evaluation-set regression, with accuracy loss within business tolerance;
• Domain evaluation set (50–200 real samples) established and incorporated into the change process;
• Reflection iteration loop enabled for complex tasks, with simple tasks answered directly;
• Serving interface, monitoring and alerting, and version management in place;
• Cloud elastic fallback or degradation path designed.

Deploying open-source LLMs on-premises is not as simple as “download the weights and run them”—it is a complete engineering chain consisting of requirements assessment, model selection, quantization, domain adaptation, inference quality, and serving. Do each link solidly, and consumer-grade GPUs can equally support a low-cost, high-privacy, 24/7 stable private AI service.