Very Large Open-Weight Models in Production: Licensing, Compute, and Deployment

    This article explains how to evaluate and deploy very large open-weight models. It distinguishes open weights from open source, outlines common license types and review points, describes MoE architecture and parameter metrics, provides storage and VRAM sizing formulas, and discusses low-precision trade-offs and long-context/sparse attention evaluation.

    1. First, Separate “Open Weights” from “Open Source”

    Being able to download weights only means the files are accessible. Whether you can use them commercially, modify them, and redistribute them depends on the accompanying license terms. Confusing the two is one of the most common reasons enterprises misuse models.

    Common license types fall into roughly three categories:

    • Permissive licenses: allow free use, copying, modification, distribution, and commercial use, usually requiring only retention of copyright notices.
    • Licenses with additional terms: the main terms are permissive but impose limits on specific scenarios. For example, if an entity providing model services has revenue above a threshold, or a commercial product has monthly active users above a certain scale, a separate agreement may be required.
    • Restrictive licenses: limit purpose, region, or users; some clauses may prohibit use for training other models.

    When reviewing a license, check the following item by item:

    1. Scope of grant: are use, modification, fine-tuning, and distribution all permitted?
    2. Commercial thresholds: are there revenue, user-count, or deployment-scale thresholds? How are the measurement period and definition specified?
    3. Obligations: are attribution, retention of notices, or disclosure of modifications required?
    4. Prohibitions: are certain industries or uses prohibited?
    5. Change risk: can the license terms be modified unilaterally? After a new version is released, does the old version remain valid?

    Suppose a license states that entities providing model services externally and with cumulative revenue over the past twelve months above a certain amount must sign a separate agreement before commercial use. The team must first calculate revenue according to the contract definition, then determine whether legal review is needed. The measurement window and revenue definition directly affect the conclusion; rely on the original contract text, not impressions.

    2. Understanding the Structure of Very Large Parameter Models

    Mixture-of-Experts (MoE) splits the feed-forward portion into many expert subnetworks, and each token is routed to only a few experts for computation. This creates two metrics that need to be distinguished:

    • Total parameter count: determines the size of the weight files and the upper bound of memory capacity that must be resident.
    • Activated parameter count: determines the computation and latency of a single inference step.

    Routing is dynamic, and any expert may be called by a token, so all expert weights must be loaded into accessible memory. A smaller activated parameter count does not mean a lower hardware barrier.

    Common structural designs also include:

    • Shared experts: every token passes through them; they handle general features and reduce redundant learning across experts.
    • Multimodal encoders: convert images into vectors and feed them together with text into the backbone network.
    • Long-context mechanisms: support longer inputs through position encoding and attention optimization.

    Understanding these structures helps determine where the model consumes resources and explain capability differences observed in evaluations.

    3. Capacity Estimation: Storage and VRAM

    Weight footprint can be roughly estimated with the following formula:

    1. Weight footprint ≈ total parameters × average bytes per parameter × (1 + metadata overhead ratio)

    At different precisions, bytes per parameter are roughly: 16-bit floating point is 2 bytes, 8-bit is 1 byte, and 4-bit is about 0.5 bytes. Block-scaling formats introduce a small amount of additional overhead, and the metadata overhead ratio is usually only a few percent.

    Assume the model has 2 trillion total parameters and uses 4-bit quantization:

    1. 2 × 10^12 × 0.5 bytes ≈ 1 TB
    2. After adding metadata overhead, about 1.05 TB

    VRAM also needs to reserve space for:

    • KV Cache: grows linearly with context length and number of concurrent requests. In long-context scenarios, it is often the main VRAM consumer.
    • Activations and runtime buffers: related to batch size and sequence length.
    • Communication and fragmentation: in multi-GPU parallelism, sharding and communication both consume extra space.

    When planning, use the following process to calculate the number of GPUs:

    1. Required total VRAM = weight footprint + KV Cache footprint + runtime buffers
    2. Usable VRAM per GPU = nominal VRAM per GPU × usable ratio (empirical value about 0.9)
    3. GPUs = ceil(required total VRAM / usable VRAM per GPU)

    As a rule of thumb, reserve 20% to 30% of total required VRAM as headroom, and confirm whether tensor parallelism or expert parallelism is split evenly. If sharding is uneven, one GPU may run out of VRAM before the others.

    4. Trade-offs Between Low Precision and Quality

    Quantization can significantly reduce storage and bandwidth pressure, at the cost of precision loss. Two common paths are:

    • Post-training quantization: convert an already trained full-precision model. Cost is low, but quality degradation may be more obvious at low bit widths.
    • Quantization-aware training: simulate low-precision computation during training so the model adapts to precision loss in advance. At the same bit width, quality is usually more stable.

    If the publisher only provides a single quantized version, you need to validate its performance on your target tasks yourself. Do not assume it matches the full-precision version, especially for numerically sensitive tasks such as mathematical reasoning and code generation.

    5. Long Context and Sparse Attention

    Standard attention computation grows quadratically with sequence length, while KV Cache grows linearly with length. Sparse attention computes relationships only among some positions, which can reduce computation and memory pressure, but it may miss some long-distance dependencies.

    When evaluating long-context capability, it is not advisable to look only at scores on short questions. More effective methods include:

    • Embed key information in long text and check whether the model can retrieve it accurately.
    • Design questions that require crossing multiple paragraphs to answer and check the model’s integration ability.
    • Gradually increase input length and observe quality.