Open-Weight Giant Models in Production: Parameters, Licensing, and Deployment

    Open-weight releases make inference weights available, but they do not settle licensing, hardware sizing, or serving architecture. This article explains how to separate total parameters from activated parameters in a 2.8T-parameter mixture-of-experts model, estimate a 1.56TB weight footprint and an eight-GPU baseline, plan parallel and long-context inference, review custom license thresholds, and evaluate vendor benchmark claims before committing to private deployment.

    1. Clarify the Terms: Open Weights Are Not Open Source

    For very large models, “open weights” usually means only that the parameter files required for inference can be downloaded publicly. It does not mean that training code, training data, data mix, training logs, and the full toolchain are all open. For developers, this solves the question of whether they can obtain the weights and run private deployments. For enterprises, it raises further questions: whether the license permits commercial use, whether it permits offering the model as an external service, and whether scaling the business requires a separate agreement.

    Therefore, evaluating an open-weight model should not focus only on parameter count. Three things matter at the same time: model architecture and weight format, the hardware and engineering stack required for deployment, and the license constraints on the business model. Ignoring any one of them can lead to rework after launch.

    2. 2.8T Total Parameters Does Not Mean 2.8T Activated

    Publicly available materials indicate that this class of very large model uses a mixture-of-experts architecture: about 2.8T total parameters, but only about 104B parameters activated per token. Its routing configuration is commonly described as 896 experts, with 16 experts selected per token, plus 2 shared experts; about 93 layers; a context window of up to 1 million tokens; and a vision encoder to support multimodal input. Two accounting views need to be separated:

    • Compute accounting: inference compute is mainly estimated from activated parameters. 104B activated is far smaller than 2.8T total, which is the key reason the mixture-of-experts design lowers per-token compute cost.
    • Memory accounting: weights must cover all experts, and memory usage is estimated from total parameters and quantization bit width. It cannot be inferred simply from 104B.

    Architecturally, this type of model may also replace some full-attention layers with sparse attention to reduce the attention compute cost of million-token long contexts. Sparse attention is valuable for long documents, codebase-level context, and multi-turn agent tasks, but it is complex to implement. Whether the inference framework supports it efficiently and whether it can maintain stable throughput over long sequences require empirical testing.

    For quantization, public weights may be the output of quantization-aware training, such as 4-bit floating-point weights and 8-bit floating-point activations, with only a single quantized version provided and no higher-precision or alternative quantized versions. This has two implications. First, the memory barrier is lower, but accuracy loss and long-context stability must be validated per task. Second, the inference toolchain must support the corresponding quantization format; otherwise, “the weights can be downloaded” does not mean “the service can run.”

    3. The Engineering Bill: 1.56TB of Weights and an Eight-GPU Starting Point

    The model weights are about 1.56TB, containing 96 tensor shards and 112 files. Based only on static weight occupancy, splitting across 8 GPUs gives about 195GB per GPU. This does not yet include KV cache, activation buffers, communication buffers, framework overhead, or the extra memory required by long contexts. Therefore, “8 GPUs to start” should be understood as one of the minimum hardware configurations, not a threshold that consumer devices can attempt. Actual deployment also depends on per-GPU memory, inter-GPU interconnect bandwidth, host memory, storage throughput, and network bandwidth.

    A simplified deployment plan can be abstracted as follows:

    1. deployment_plan:
    2. model:
    3. total_params: 2.8e12
    4. active_params: 104e9
    5. weight_storage_tb: 1.56
    6. quantization: weight_4bit_act_8bit
    7. parallel:
    8. tensor_parallel: 8
    9. expert_parallel: auto
    10. pipeline_parallel: auto
    11. runtime:
    12. max_context_tokens: 1000000
    13. kv_cache: paged
    14. continuous_batching: true
    15. observability:
    16. metrics: [latency, throughput, device_memory, expert_load]

    Downloading and verification are also major engineering tasks: 1.56TB requires stable storage, shard verification, resumable downloads, and mirror synchronization. Self-hosted data centers need to consider power, cooling, and rack space. For cloud deployment, object storage can hold weight shards, container platforms can orchestrate inference services, and logging and monitoring services can observe runtime status. For example, general-purpose object storage, container platforms, logging, and monitoring services from a cloud provider can serve as hosting options, but exact specifications still need to be selected according to memory, interconnect bandwidth, and throughput targets.

    4. Inference Deployment: Parallel Strategy Matters More Than “It Loads”

    Inference optimization for very large mixture-of-experts models usually involves tensor parallelism, expert parallelism, pipeline parallelism, and sequence parallelism. 896 experts are suitable for splitting across multiple GPUs or machines by expert, but expert routing has hotspot issues: some experts are selected frequently, causing uneven load across GPUs. Engineering work needs to focus on capacity factors, expert replication, routing balance, and communication overlap.

    Long context is another main line. A 1 million token context means the KV cache may become a major memory consumer. It requires general techniques such as paged management, prefix caching, KV quantization, and continuous batching. Although sparse attention reduces compute, if the implementation is insufficient, it may still be limited by memory bandwidth. During service operation, monitoring should also cover time to first token, time per output token, throughput, memory watermark, expert load, and request queue length.

    5. License Compliance: Open Weights Still Have Scale Boundaries

    This type of model usually uses a custom license rather than a standard permissive open-source license. Its main terms may allow free use, copying, modification, distribution, fine-tuning, and commercial use, but may add conditions for large-scale external business. For example, when offering model-as-a-service externally and cumulative revenue over 12 months exceeds $20 million, a separate agreement with the publisher may be required before commercial use; a commercial product reaching a certain monthly active user scale may also trigger additional terms. Specific thresholds and interpretations should be based on the original license text and legal advice.

    An enterprise compliance checklist should at least cover: whether the model is directly offered as an external service; how relevant revenue over the past 12 months is counted; whether commercial product monthly active users reach the terms; whether the license and notices are retained when distributing derivative models; the boundary between internal use and external service; and whether advance authorization is needed. Open weights lower the barrier to access, but they do not eliminate license risk.

    6. Capability Evaluation and Task Selection

    Vendor-reported benchmarks in public materials usually show that this class of model is strong in coding, terminal operation, and agent tasks, while it may be weaker than some closed reference models in scientific reasoning and difficult reasoning. In the cited materials, 16 of 37 benchmarks exceed the comparison model, a mixed result. A third-party intelligence index may also give a reference value such as 55 points and fourth place globally, but benchmark scores are affected by prompts, tool calls, contamination, and evaluation configuration. They cannot replace evaluation on business-specific datasets.

    Task selection should be graded. Code agents, tool calling, and long-document multimodal understanding can be validated first at small scale. Scientific computing, rigorous reasoning, and high-reliability decisions require additional human review and domain evaluation. For latency-sensitive scenarios, focus on measuring tail latency under long context and expert routing.

    7. Can a Typical Team Adopt It?

    The conclusion is that it can be studied, but the barrier should not be underestimated. Local inference requires at least 8 high-memory accelerators to start, and total memory must cover weights, KV cache, and runtime overhead. Fine-tuning is harder than inference. Full-parameter fine-tuning is basically unrealistic, and parameter-efficient fine-tuning must also address quantization compatibility and expert routing. A more prudent path is to first validate task benefits through a controlled service, then evaluate the cost of private deployment. If cloud hosting is chosen, weight storage, service orchestration, logging and monitoring, access control, content moderation, and cost alerts should be included in the same plan.

    Open weights bring controllable deployment and room for secondary development, but “open” has boundaries, and “downloadable” does not mean “low-cost to run.” Calculate the parameter bill, memory bill, license bill, and evaluation bill before deciding whether to invest. That is the rational path for putting very large models into production.