Open-Source LLM Selection and Deployment Guide: Estimate VRAM First, Then Measure Hallucination Rate

    This guide presents a practical workflow for selecting and deploying open-source LLMs: separate total parameters from active parameters, estimate VRAM from weights, KV cache, and runtime overhead, build an internal evaluation set to measure quality and hallucination rate, complete pre-launch verification, and plan capacity with a tiered architecture. It includes formulas, parameter logging templates, checklist items, rollback steps, and capacity calculations. Numerical values are illustrative assumptions and should be replaced with measurements on the target models and hardware.

    The first step in selecting and deploying an open-source LLM is to break down the parameter numbers on the release page and then replace estimates with measured data. This article is organized around four parts: definitions and VRAM estimation, internal evaluation and hallucination rate, pre-launch verification, and capacity planning with a tiered architecture. Readers can use Part 1 to calculate a VRAM lower bound, Part 2 to build an evaluation set, and the last two parts to make launch decisions. Most numbers in this article are assumed examples; during formal selection, replace them with measured results for the target model and target hardware.

    1. Consistent Definitions: Total Parameters, Active Parameters, and VRAM

    Parameter numbers on the release page need to be understood separately. For Mixture-of-Experts (MoE) models, total parameters mainly determine weight size, while active parameters determine the compute per token. They consume different resources and should be estimated and measured separately.

    Metric Main contributing factors Estimation or measurement method Common pitfall
    Weight size Total parameters, bytes per parameter Total parameters multiplied by precision bytes: about 2 bytes for FP16, 1 byte for INT8, and 0.5 bytes for INT4 Assuming fewer active parameters means lower VRAM
    VRAM usage Weights, KV cache, activations, framework overhead Read actual VRAM after startup, then stress test at target concurrency and context length Estimating only from weight size
    Output speed (decode) Memory bandwidth, active parameters, KV cache reads, concurrency Fix input length, output length, batch, concurrency, quantization precision, and hardware model; count output tokens per second Assuming a purely linear relationship with active parameters
    Time to first token (prefill) Input length, compute throughput Measure the time from request sent to first token returned Assuming it depends on output length

    This article uses binary units, GiB (1 GiB = 2^30 bytes), for VRAM. Disks and release pages often use decimal GB (1 GB = 10^9 bytes). The conversion is 1 GiB ≈ 1.074 GB.

    The weight size formula is:

    1. Weight size ≈ total parameters × bytes per parameter
    2. Example (assumed values): 100 billion total parameters, INT4 quantization
    3. Weight size ≈ 100 × 10^9 × 0.5 bytes = 5 × 10^10 bytes ≈ 46.6 GiB (about 50 GB)

    The VRAM lower bound must also include KV cache and runtime overhead. The KV cache formula is:

    1. KV cache bytes ≈ 2 × layers × KV heads × head dimension × context length × concurrent requests × bytes per element

    The coefficient 2 corresponds to the K and V caches. Layers, KV heads, and head dimension must be read from the model configuration file; do not apply generic values.

    Assume a model has 60 layers, 8 KV heads, and a head dimension of 128, using an FP16 cache. The KV cache per token is 2 × 60 × 8 × 128 × 2 bytes = 245,760 bytes, or 240 KiB. A single 32K (32,768-token) context request takes about 7.5 GiB, and 8 concurrent requests take about 60 GiB. Long context and high concurrency can make VRAM requirements far exceed the weight size.

    The common online rule of “weight size multiplied by 1.2” applies only to rough estimates with short context and low concurrency. Under long-context or high-concurrency conditions, this factor severely underestimates VRAM requirements. Use the KV cache formula and measured results instead.

    Output speed and time to first token should be evaluated separately. The prefill phase processes the entire input at once and mainly consumes compute; the longer the input, the higher the time to first token. The decode phase generates one token per step and must read weights and KV cache each step, so it is usually limited by memory bandwidth. In MoE models, batch requests are routed to different experts, and as concurrency increases, the number of experts that must be read also increases, raising bandwidth pressure. With expert parallelism (EP) or tensor parallelism (TP), cross-GPU communication overhead must also be considered. Therefore, fewer active parameters do not necessarily mean faster performance under high concurrency; generation speed must be measured at the target concurrency.

    Quantization can reduce VRAM but may affect accuracy, especially for mathematical reasoning, code generation, and long-chain tasks. Before putting INT4 into production, run an A/B comparison against an FP16 or INT8 baseline on the internal evaluation set, record the quality drop, and keep a fallback to the original precision. The target hardware must also support the corresponding quantization operators; otherwise, the performance benefit may not materialize.

    2. Internal Evaluation and Hallucination Rate Measurement

    Leaderboard scores are suitable for coarse screening, not for final decisions. General leaderboards focus on general question answering and code, while finance, medical, legal, and other domains require separate validation. Different leaderboards use different prompt templates, sampling temperatures, context lengths, and tool switches, so results are not directly comparable across leaderboards. If training data includes evaluation questions, scores will also be inflated.

    2.1 Evaluation Set Composition

    An internal evaluation set should contain at least 300 samples across three categories:

    1. Standard samples: Cover typical business questions, sampled by business subcategory, with no fewer than 30 samples per subcategory.
    2. Boundary samples: Requests with missing input, contradictory information, very long text, or malformed formats.
    3. Adversarial samples: Requests that induce the model to fabricate citations, fake APIs or data, or execute instructions beyond its authority.

    Each sample should record an ID, task type, input, reference answer or scoring rule, source, and version.

    2.2 Inference Parameter Logging Template

    During evaluation, inference parameters must be fixed and recorded; otherwise, results cannot be reproduced. Each evaluation should record at least the fields in the table below:

    Field Example Description
    model_name Model name Match the release page version
    model_version Version number or commit hash Record the exact version
    weight_sha256 Weight file hash Confirm the file was not replaced
    precision FP16 / INT8 / INT4 Quantization format
    framework_version Inference framework and version Retest after upgrading the framework
    temperature 0 to 0.2, etc. Set by task type
    top_p Value Sampling truncation parameter
    max_output_tokens Value Maximum output length
    stop_sequences Stop word list Affects the output truncation point
    context_length Value Actual context limit used
    system_prompt_version Version identifier Record system prompt changes
    tool_calling On or off Tool-calling switch
    seed Random seed Explain if it cannot be fixed
    concurrency Concurrency level Affects latency and VRAM

    Temperature should be set by task type. Classification, extraction, and factual question answering can start from 0 to 0.2. Creative generation can use a higher temperature, but the same input should be sampled multiple times and averaged. These values are only starting points; internal evaluation results should govern.

    2.3 Scoring Rules and Pass Thresholds

    Scoring has three layers: rule validation, model review, and human review.

    Rule validation criteria are as follows:

    • Format: The output follows the required structure. For tasks requiring JSON, the JSON parse success rate must be 100%.
    • Field matching: Key fields exactly match the reference answer or match within an agreed synonym table.
    • Numeric error: Numeric results are within the allowed error, for example, a relative error of no more than 1%.
    • Citation verification: Literature, links, and API names in the output must be findable in the knowledge base or tool returns; if not found, they are judged as fabricated.

    Model review is used only for open-ended answers that rules cannot cover. The review model should come from a different vendor or version than the tested model, and review results should be sample-calibrated.

    Recommended human review ratios are: review all samples failed by rules; review all adversarial samples and high-risk domain samples; randomly sample 10% of the remaining passed samples.

    There are two common definitions of hallucination rate, and the report must state which one is used:

    • Sample-level hallucination rate = number of samples containing at least one hallucination ÷ total evaluation samples
    • Assertion-level hallucination rate = number of factual assertions judged as hallucinations ÷ total factual assertions in the outputs

    Assertion-level hallucination rate is more granular and suitable for long-text answers. Sample-level hallucination rate is more direct and suitable for short question answering.

    Evaluation pseudocode:

    1. Input: candidate model list M, internal evaluation set D, fixed inference parameters P, scoring function score
    2. Initialize result table R
    3. For each model m in M:
    4. For each sample d in D:
    5. output = m.generate(d.prompt, P)
    6. s = score(output, d.reference, d.rule)
    7. Append (m.name, m.weight_sha256, d.id, d.type, output, s) to R
    8. Aggregate R: compute pass rate, sample-level hallucination rate, and assertion-level hallucination rate by model and task type

    Statistical error must be stated. When estimating a hallucination rate of about 10% from 300 samples, the normal approximation gives a 95% confidence interval of about ±3.4 percentage points. If the difference between two models is smaller than this interval, it is not possible to determine which is better; increase the sample size or use a paired test.

    Pass thresholds should be set by the team according to business risk. The following are starting values:

    • Standard sample pass rate is no lower than a comparable baseline minus 3 percentage points.
    • Adversarial sample-level hallucination rate is no higher than the baseline, and no unauthorized execution may occur.
    • JSON parse success rate is 100%.
    • Hallucination rate in high-risk domains is no higher than in general domains.

    3. Pre-Launch Verification Checklist

    Before a model enters production, confirm each item below and record the result in the selection review sheet.

    Check item Verification content Pass criteria
    Weight hash Compare the SHA256 provided by the publisher with the local file Exact match
    License Confirm commercial use, redistribution, modification, and usage restrictions Complies with legal opinion
    Tokenizer and template Tokenizer, chat template, and weight version are consistent Versions match and test cases exist
    Inference framework Framework version and quantization operator support Same as the version used for evaluation
    Remote code Whether trust_remote_code is required If required, manually review the code first
    Safety filtering Input and output filtering, refusal policy, sensitive information detection Adversarial sample tests pass
    Data and privacy Whether request logs contain user data and the retention period De-identified and compliant
    Monitoring and alerting P95 latency, error rate, VRAM utilization, hallucination spot-check rate Alert rules configured
    Rollback plan Previous version weights, configuration, and routing switch Drilled and acceptable duration

    Recommended rollback steps:

    1. Use a canary rollout at 5%, 20%, 50%, and 100%, observing at least one business cycle at each level.
    2. Stop the rollout immediately if any of the following occurs: sample-level hallucination rate exceeds the baseline threshold, error rate exceeds the agreed cap, or P95 latency exceeds the agreed cap.
    3. During rollback, switch routing back to the previous version and retain logs and samples from the faulty version for analysis.
    4. Keep the previous version’s weights and configuration until the new version has run stably for at least one month.
    5. Drill rollback once per quarter and record the actual duration.

    4. Capacity Planning and Tiered Architecture

    4.1 Tiering Strategy

    When compute is limited, not all requests should go to the largest model. A common approach uses three tiers:

    • Tier 1: Rules and small models handle low-risk requests such as format conversion, simple classification, and keyword extraction.
    • Tier 2: Medium-sized models handle routine question answering and summarization.
    • Tier 3: Large models handle complex reasoning, code generation, and high-risk domain requests.

    The router assigns requests based on task type, input length, and confidence. When the Tier 1 model has low confidence or its output fails rule validation, the request is escalated to the next tier. Each tier needs its own evaluation set and thresholds.

    4.2 Concurrency, VRAM, and Latency Estimation

    Using the example model above, weights at INT4 are 46.6 GiB, and the KV cache for a single request with 32K context is 7.5 GiB. The table below calculates only weights and KV cache, excluding runtime overhead:

    Concurrency Total KV cache Total VRAM (weights + KV) After 15% headroom
    1 7.5 GiB 54.1 GiB 62.2 GiB
    4 30 GiB 76.6 GiB 88.1 GiB
    8 60 GiB 106.6 GiB 122.6 GiB
    16 120 GiB 166.6 GiB 191.6 GiB

    The 15% headroom is only a starting point to cover memory fragmentation and traffic spikes. The actual headroom should be determined from VRAM fluctuations during stress testing. Calculating KV cache by the longest context is a conservative estimate. If the business’s average context is shorter, the average can be calculated from the actual distribution, but the peak should still be validated using the longest context.

    Latency metrics must be filled in through stress testing. Recommended stress-test records include: input length, output length, concurrency, batch setting, quantization precision, GPU model and count, time to first token (P50 and P95), and output tokens per second. Latency numbers without these conditions cannot be used for capacity decisions.

    4.3 Capacity Planning Steps

    1. Estimate peak concurrency and average input/output lengths from business traffic.
    2. Build a stress-test baseline for each model tier using the method in Section 2.
    3. Calculate the VRAM required for each tier using the formulas above, plus runtime overhead and headroom.
    4. Determine the number of replicas per tier so that P95 latency at peak concurrency meets business requirements.
    5. After launch, continuously compare actual load with planned values and review once per month.

    Capacity planning should be based on measurements. Calculation results can only serve as a preliminary basis for procurement or scaling.

    5. Conclusion

    The order for open-source LLM selection is: first separate total parameters and active parameters, then calculate VRAM according to target concurrency and context length, then measure quality and hallucination rate with an internal evaluation set, pass the launch checklist, and finally plan capacity with a tiered architecture. Every step should keep reproducible records, including weight hash, inference parameters, and stress-test conditions. Readers can use the tables and formulas in this article to build their own selection review sheet and replace the assumed values with measured results on the target hardware.