Open-Source LLM Post-Training Selection: 5 Variables to Check First

    Selecting an open-source large language model (LLM) requires looking beyond base model names and parameter counts. Post-training significantly impacts a model's ability to follow instructions, use tools, and manage multi-turn conversations. This article outlines five critical variables to evaluate during the selection process: total versus active parameters in MoE architectures, benchmark measurement conditions, hallucination rate metrics, VRAM estimation formulas, and small-scale traffic validation strategies. By understanding these variables, developers and architects can make informed decisions and avoid common vendor pitfalls.

    1. Why Post-Training Affects Model Selection

    While pretraining determines a model’s knowledge base, post-training dictates its ability to follow instructions, call tools, and engage in multi-turn conversations. Common post-training phases include supervised fine-tuning (SFT), instruction tuning, preference-based reinforcement learning (RLHF), and tool-use data mixing.

    The same base model can perform drastically differently depending on its post-training. Therefore, selection should not rely solely on the base model name or parameter scale. Before choosing, verify these five variables: total vs. activated parameters, benchmark measurement conditions, hallucination rate metrics, VRAM requirements, and small-scale traffic validation results.

    2. MoE: Separating Total and Active Parameters

    Mixture of Experts (MoE) models distribute parameters across multiple expert subnetworks, activating only a fraction during inference. For instance, a recently released open-source flagship model reportedly has 2.8 trillion total parameters but activates 104B per inference. These figures are unverified but illustrate a key point.

    The two numbers serve different purposes:

    • Total parameters determine the storage footprint of the weights. Even unactivated expert weights must reside in VRAM or system memory, or be swapped in and out by scheduling mechanisms.
    • Active parameters determine the computation volume per token, affecting the inference time per token.

    Therefore, VRAM estimation for MoE models should be based on total parameters. Throughput and latency can reference active parameters, but actual latency is also affected by routing overhead, inter-device communication, and batching strategies, requiring empirical testing.

    3. Multimodality and Long Context: Defining Capability Boundaries

    Multimodal capabilities typically refer to a model’s ability to process images, text, and sometimes audio. Long context refers to the maximum number of tokens a model can process at once. Both are easily marketed as “supported,” but support does not guarantee stable performance.

    When verifying, answer three questions:

    1. What input types are supported? Is the vision encoder provided separately? What is the maximum image resolution?
    2. What is the nominal context length, and does retrieval or Q&A accuracy drop significantly when approaching this limit?
    3. Do long-text tests include real business documents, or only synthetic “needle in a haystack” tests?

    4. Edge and Browser Benchmarks: Check Measurement Conditions

    Models running on edge devices and browsers are highly sensitive to quantization methods, inference engines, and hardware. The same model may score differently across devices.

    Whether benchmark gaps are meaningful depends on consistent measurement conditions. The table below lists items to record and compare.

    Checklist Item Information to Record Why It Matters
    Device Model, VRAM/RAM capacity, driver version Determines if the model fits and limits speed
    Quantization Precision (e.g., FP16, INT8, INT4) and method Quantization alters precision and output quality
    Context Length Maximum context used during testing Long inputs significantly increase VRAM usage and latency
    Sampling Params temperature, top_p, max output length Randomness alters scores
    Tool Calling Whether tool use is allowed, and the toolset Differences in tool use affect final results
    Scoring Script Script source, answer parsing rules, human review Different parsing rules can yield different scores for the same answer
    Version Model weight version, inference framework version Minor updates can change scores
    Repetitions Runs per question, average or max value used Single runs fluctuate; using max values overestimates ability

    If two datasets are inconsistent across multiple items, the score gap between them may have no practical significance.

    5. Hallucination Rate: Define the Metrics First

    Hallucination rate refers to the proportion of model outputs that contradict facts or input materials. However, definitions of “hallucination” vary widely. Some evaluate only factual errors, while others include fabricated citations or context-contradicting answers.

    When checking hallucination rates, confirm at least these four points:

    • Source and scale of the test set, and whether samples are public.
    • Judgment method: human annotation, model scoring, or rule matching.
    • Whether refusals to answer count as correct. If a model refuses more often, hallucination rates may drop, but usefulness also suffers.
    • Whether the report includes response coverage or usefulness metrics.

    6. VRAM Estimation: Formulas and Steps

    Use the following formulas for initial judgment. Estimates must be verified with actual testing.

    Weight VRAM ≈ Parameters × Bytes per parameter

    • FP16 or BF16: ~2 bytes per parameter
    • INT8: ~1 byte per parameter
    • INT4: ~0.5 bytes per parameter

    KV Cache VRAM ≈ 2 × Layers × KV Heads × Head Dimension × Context Length × Concurrent Requests × Bytes per element

    Total Estimate ≈ (Weight VRAM + KV Cache VRAM + Activation & Framework Overhead) × Safety Factor

    The “2” accounts for both K and V caches. Activation and framework overhead is empirically set to 10% to 20% of the Weight VRAM; this is an experience-based value, not a universal standard. The safety factor can be 1.1 to 1.2.

    Example (hypothetical): A 70B model using INT4 quantization requires approximately 70×10⁹×0.5 bytes for weights, or 35GB. With KV Cache and overhead added, actual requirements will be higher.

    Estimation steps:

    1. Calculate weight VRAM using total parameters (not active parameters).
    2. Determine target context length and concurrency to calculate KV Cache VRAM.
    3. Add activation and framework overhead, multiply by the safety factor for a preliminary conclusion.
    4. Load the model on the target device and record peak VRAM. If the actual value deviates from the estimate by more than 20%, check KV Cache configurations and batching settings.

    7. Small Traffic Validation: Steps and Rollback Conditions

    Small traffic validation routes a fraction of real requests to the new model, comparing it in parallel with a baseline. The following steps serve as a template; specific thresholds should be decided by your team.

    1. Set a comparison baseline (current production model or a known reference model).
    2. Ramp up traffic incrementally: start with 1% to 5%, then expand to 10%, 30%, and 50% after sufficient observation.
    3. Select metrics: task success rate, human-sampled hallucination rate, time to first token (TTFT), tokens per second (TPS), cost per request, and escalation or complaint rates.
    4. Estimate sample size. For example, to detect a 3-percentage-point difference from an 80% baseline success rate (using a standard two-sided test with 0.05 significance and 0.8 power), you need approximately 2,600 requests per group. Recalculate based on your specific formula and parameters.
    5. Define rollback conditions beforehand (e.g., success rate drops below a certain threshold, P95 latency exceeds limits, or sampled hallucination rate rises). Rollback conditions must be locked in before traffic ramp-up, not decided afterward.

    8. Conclusion and Risk Warnings

    Post-training and MoE architectures undeniably affect model performance. However, claims like “multiple times improvement” and specific figures for total/active parameters or hallucination rates should be treated as unverified vendor claims without comparison baselines, measurement metrics, and independent reproduction. The original claim that post-training brings multiple times the improvement of the base model lacks baselines and methods, so this figure is not adopted as a selection criterion here.

    Risk warnings:

    • Unverified benchmarks should not be written into procurement commitments or external promotions.
    • Rely on actual testing for VRAM estimates.
    • Retain complete logs during small traffic validation for post-incident analysis.
    • This article provides a verification framework and does not replace vendor documentation or independent evaluations. Final decisions should be based on test data from your own environment.