Open Model Selection Guide: VRAM Calculation, Quantized Deployment, and License Verification

    Choosing open-weight models requires careful evaluation, as downloadable weights do not guarantee local deployment, and vendor-reported scores do not ensure reproducibility. This guide provides a structured "verify first, estimate next, decide last" approach for evaluating MoE, dense, and world models. It covers score validation, VRAM calculation, quantization impacts, license compliance, and a step-by-step decision process.

    Open-weight models are being released increasingly frequently, and teams often face two challenges: downloadable weights do not mean they can run locally, and vendor-reported scores do not guarantee they can be reproduced on your specific tasks. This article organizes evaluation methods for three categories of models in the order of “verify first, estimate next, decide last.” All performance data mentioned are self-reported by the publishers and have not been independently reproduced by third parties.

    1. Verify First: Self-Reported Scores and License Terms

    Self-reported scores require item-by-item verification. First, confirm the evaluation dataset version, as different versions of the same evaluation may have different questions and scoring scripts. Second, verify the prompt template and the number of few-shot examples. Third, check the decoding parameters, including temperature, maximum generation length, and inference budget. Fourth, confirm whether additional inference computation, such as long thinking modes, is enabled. Fifth, check if reproduction scripts are available. If any of these are missing, the score should only serve as a reference. Additionally, check whether evaluation questions might appear in the model’s training data. If data contamination cannot be ruled out, rely on the results from your team’s privately built task set.

    License terms also require careful review, focusing on the following points: Is commercial use allowed? Can derivative models retain the original name? Is redistribution permitted? Are there user count or revenue thresholds for serving use? Are there regional, export, or domain-specific restrictions? Are there explicit declarations regarding the sources of weights and training data? The constraints of standard open-source licenses versus vendor-defined agreements vary significantly; the final conclusion should be confirmed by the legal team.

    2. Trillion-Parameter MoE: Large Total Parameters vs. Small Activated Parameters

    MoE models essentially trade storage for compute. The total parameters determine the model capacity and loading cost, while the activated parameters determine the computation per token. Taking publicly available data as an example, a model might have approximately 2.8 trillion total parameters, activating about 100 billion parameters per token, representing an activation ratio of about 3% to 4%. However, “activating 100 billion” does not mean you only need the resources for 100 billion parameters; routing, shared experts, and the inference framework all introduce overhead.

    Before selection, confirm the statistical scope of the activated parameters, i.e., whether it includes attention layers, shared experts, and embedding layers. Different scopes can lead to significant differences in compute estimation. It is advisable to request a scope description from the publisher; if unavailable, estimate using the most conservative scope.

    VRAM calculation can be done using the following method:

    • Weight volume = Total parameters × Bytes per parameter. FP16 is 2 bytes, INT8 is 1 byte, and 4-bit is about 0.5 bytes.
    • 2.8 trillion parameters are about 5.6TB in FP16 and about 1.4TB in 4-bit.
    • Quantization schemes also include metadata such as scaling factors, so the actual volume will be slightly higher than the theoretical value.

    1.4TB exceeds the VRAM configuration of a standard single server and typically requires a multi-GPU cluster. If CPU offload is used, some weights remain in host memory, restricting inference to memory bandwidth and PCIe bandwidth limits, which significantly reduces throughput. Therefore, “weights are downloadable” and “can be deployed in your team’s environment” must be judged separately.

    Long context capabilities must also be tested independently. When a publisher states a model supports million-level context, this refers to an acceptable input length and does not mean output quality is stable at that length. Tests should design long-form Q&A and information retrieval tasks, recording accuracy at different lengths like 32K, 128K, and 1M, while also logging KV cache usage and Time to First Token (TTFT). Without this test, long context capability should not be used as a selection criterion.

    3. 4-bit Dense Models: Weights, KV Cache, and Runtime Overhead

    Dense models with around 65 billion parameters support 4-bit quantized deployment. The weight volume is estimated as follows: 65 billion × 0.5 bytes ≈ 32.5GB, while in FP16 it is about 130GB. If quantization with a group size of 128 is used, each group includes a 16-bit scaling factor, adding about 0.125 bit per parameter, bringing the weight volume to roughly 34GB.

    4-bit quantization only indicates that weights are stored in 4 bits; it does not mean the entire inference pipeline runs in 4-bit. The following must be verified item by item: the precision used for activations (some schemes maintain FP16 or BF16); whether the KV cache is quantized (e.g., FP8 or INT8); the quantization method and group size used, as different methods have varying precision performance; whether the content and size of the calibration dataset align with your business; and the accuracy gap on the target task before and after quantization.

    The VRAM for the KV cache can be estimated using the following formula:
    KV cache bytes = 2 × Number of layers × Number of KV heads × Head dimension × Context length × Batch size × Bytes per element

    Assuming a model has 80 layers, 8 KV heads, a head dimension of 128, and the KV cache is stored in FP16, each token will occupy about 320KB. A 32K context request takes about 10GB, and 8-way concurrency takes about 80GB. These are illustrative numbers; actual calculations should be based on the parameters in the model configuration file.

    Deployment boundaries can be judged as follows:

    • Single 80GB GPU: 34GB of weights plus KV cache and runtime overhead can accommodate a small number of concurrent medium-length requests, but with limited headroom.
    • Dual GPUs or more: More suitable for higher concurrency or longer contexts.
    • Runtime overhead (CUDA contexts, activation buffers, VRAM fragmentation) generally requires reserving several GBs, and it is recommended to leave a certain percentage of margin.

    Throughput testing can be executed following these steps:

    1. Fix the quantization and inference framework versions, and record the configuration.
    2. Build a test set using the input and output length distribution of your team’s real requests.
    3. Test TTFT and output tokens per second under both single concurrency and target concurrency.
    4. Record peak VRAM usage.
    5. Compare task metrics with an FP16 or higher precision baseline to calculate the accuracy loss.
    6. If the loss exceeds the business threshold, fall back to INT8 or higher precision, or switch to a larger deployment tier.

    4. World Models: Autonomous Driving and Robotics Weight Audits

    Several sets of world model weights tailored for autonomous driving and robotics have recently been opened. World models typically predict future scenes or states based on current observations and actions. They are often used for simulation, data augmentation, and planning result evaluation. Using them directly for control carries significantly higher risks than these other use cases.

    Before deployment, the following five points must be verified:

    • Input and Output: Clarify which sensor inputs the model receives (e.g., cameras, LiDAR, or poses), and whether the output is video frames, occupancy grids, or state vectors. Mismatched input/output formats will lead to integration failures or misinterpreted results.
    • Simulation Boundaries: Good offline metrics do not mean good closed-loop performance. Test the accumulation of long-term errors in closed-loop simulations, and track the occurrence rate of events such as collisions, lane departures, and instability.
    • Real-Vehicle or Real-Machine Boundaries: Initially run in shadow mode, only recording model outputs without issuing control commands. Once stability is confirmed, conduct small-scale tests in restricted scenarios. Control outputs must pass through an independent safety layer, such as speed limits, emergency stops, and rule validation.
    • Safety Fallback: Predefine a fallback strategy for when the model fails, and regularly verify that this strategy can be triggered correctly.
    • License and Data Compliance: Verify whether the license permits commercial and derivative use. Confirm the source of the training data, specifically checking whether personal information like faces or license plates has been desensitized. If data involves local collection or cross-border transmission, it must also comply with the data security and intelligent connected vehicle regulations of the respective region.

    5. Selection Decision Sequence

    During model selection, you can judge step-by-step in the following order. If any step fails, stop or adjust the solution:

    1. VRAM and Throughput: Can the target hardware accommodate the weights, KV cache, and runtime overhead while achieving the target latency and concurrency? If not, consider a smaller model, more GPUs, or a lower quantization bit-width if precision permits.
    2. Quantization Loss: On your team’s task set, are the metrics after quantization within an acceptable range? If not, fall back on precision or change the model.
    3. License and Data Compliance: Does the license cover commercial use, serving, and redistribution? Is the data source acceptable? If not, the model cannot enter the production environment.
    4. Score Reproducibility: Can the vendor-reported scores be reproduced in your team’s environment? If not, rely on the results of your private task set rather than public leaderboards.

    By following this sequence, teams can allocate limited testing resources to the aspects that truly impact deployment.