Ultra-Scale Open-Weight Models: Architecture, Licensing, and Deployment Guide

    Open-weight models are scaling rapidly toward the trillion-parameter range. This guide offers a more actionable framework than parameter count alone. It explains how Mixture-of-Experts (MoE) sparse activation lets trillion-parameter models actually run, why sparse attention and hybrid designs enable very long context, why many models ship only low-precision weights, how to read custom licenses before any commercial use, and how to size deployment hardware. It also includes a pre-deployment checklist, answers to common questions, and guidance on choosing between self-hosted and cloud GPU resources.

    A note up front: the parameter counts, expert counts, activated parameters, and quantization formats discussed below are presented as illustrative figures so the calculation and evaluation methods are clear. For any real deployment, rely on the corresponding model card, technical report, license text, and inference framework documentation, and avoid treating these examples as universal facts.

    1. Why Trillion-Parameter Models Can Run: MoE Sparse Activation

    Ultra-scale open-weight models typically adopt MoE (Mixture-of-Experts) architectures. Using the example figures from a public technical report, total parameters can reach around 2.8T, yet only about 100B parameters are actually activated per token during inference. Such a model may contain roughly 896 experts, with a router dynamically selecting about 16 of them per token, while keeping 1 to 2 shared experts reserved to handle common knowledge that all tokens may rely on.

    The key value of this design is that it decouples “knowledge capacity” from “per-inference compute cost.” Total parameters determine the breadth of knowledge the model may hold, while activated parameters determine the compute resources actually used in each forward pass. As a result, a trillion-parameter model can require per-token computation close to that of a roughly ten-billion-parameter dense model, which is the fundamental reason it still has a realistic chance of entering production deployment.

    When evaluating these models, it helps to record four metrics first: total parameters, activated parameters, number of experts and routing strategy, and shared-expert design. Don’t just ask “how large are the parameters”—also ask “how many parameters are actually activated per step.” Together, these two figures determine both model quality and deployment cost.

    2. Long Context: Sparse Attention and Hybrid Designs

    Top open-weight models often claim support for very long contexts, such as the million-token level. But standard full attention scales its compute and memory cost quadratically with sequence length. Enabling full attention across every layer would drive up the cost of long-context scenarios very quickly. For this reason, some models introduce sparse attention, such as DSA.

    A note on terminology here. DSA is often explained as DeepSeek Sparse Attention, a specific technical implementation within the broader category of sparse attention; “dynamic sparsity” instead emphasizes adaptively choosing compute paths based on input, task, or intra-layer relevance. The two are related but should not be treated as equivalent. In practice, sparse attention is also rarely applied across an entire model. Instead, a hybrid design is common: most layers keep full attention to preserve modeling quality, while a minority switch to sparse attention to relieve the compute and memory pressure of long contexts. The exact layer ratio, implementation, and triggering conditions should be confirmed against the model documentation.

    When evaluating models, don’t just look at “how many tokens are supported.” Also confirm whether long context is on by default, requires extra configuration, or depends on a specific inference framework. Context length is not a paper specification—it directly affects memory, latency, and concurrency as a real resource item.

    3. Why Many Open-Weight Models Ship Only Low-Precision Weights

    If ultra-large models were released entirely in high precision, both storage and loading costs would be extremely high. Using 2.8T parameters as an example, a rough conversion looks like this:

    • At 4-bit weights, the weight footprint alone is about 1.4TB.
    • At 16-bit, it’s about 5.6TB.
    • At 8-bit, it’s about 2.8TB.

    Adding embedding layers, quantization scale factors, metadata, and more, the actual on-disk size will be even larger. The roughly 1.5TB figure mentioned earlier is typically a rough estimate of 4-bit weights plus necessary metadata, and does not include all runtime overhead during inference.

    This is why many models ship only low-precision versions, for example weights in formats like MXFP4 and activations in MXFP8. The direct benefit is lower thresholds for storage, download, and loading. But whether low precision hurts quality cannot be generalized. Some models use quantization-aware training (QAT) to reduce the accuracy loss from low precision, while others may adopt post-training quantization or other optimization strategies. Whether a model uses QAT, and whether there are official accuracy evaluations, should be confirmed from the model provider’s release notes—QAT should not be assumed to be a universal fact for all low-precision models.

    For the party doing the deployment, a quantization format is not simply “better the simpler.” Three things matter: weight footprint, single-GPU memory usage, and whether the inference framework supports the format natively. If framework support is poor, a low-precision model may end up costing more due to compatibility issues.

    4. Do Open Weights Equal Open Source? Read the License Clause by Clause

    Open weights mean you can obtain the weight files, but not that you can use them unconditionally in any commercial scenario. Many open-weight repositories do not use MIT or Apache 2.0 licenses; instead they use custom licenses, such as license: other. The substance of such a license may be close to an MIT-style grant, permitting free use, copying, modification, distribution, fine-tuning, and commercial use, but it appends restrictions targeting large-scale commercialization at the top.

    The common restrictions fall into two categories. First, if you offer the model as a service and your revenue over any consecutive 12-month period exceeds a high threshold—for example, $20 million—your company may need to sign a separate agreement with the model provider. Second, if a commercial product’s monthly active users exceed a certain scale, additional authorization may also be required. The exact amounts, user scale, and triggering conditions must be confirmed against the original license text.

    For most small and mid-sized teams and internal R&D, these clauses may have no direct impact. But for vendors preparing to offer services at very large scale externally, a license compliance review must be completed before launch. At minimum, check the following: whether commercial use is allowed, whether external service is restricted, whether there is a revenue or MAU threshold, whether attribution is required, whether specific industries or regions are restricted, and how derivative models and output content ownership are defined.

    The conclusion is clear: before evaluating any open-weight model, the first step is always to read the license clause by clause, not to look only at parameter count.

    5. Sizing Deployment: Why 8 GPUs Is Often the Starting Point

    During inference, the model weights should ideally stay resident in GPU memory. Using 4-bit weights as an example, a simplified formula gives a rough estimate:

    • Weight size ≈ total parameters × bytes per parameter
    • For 2.8T parameters at 4-bit: 2.8 × 10^12 × 0.5 Byte ≈ 1.4TB

    If 8 accelerators carry the model, each card needs about 1.4TB / 8 ≈ 175GB for the weights alone. And that is just the weights—it excludes KV Cache, activations, communication buffers, framework overhead, and the extra memory from long contexts. Relying on a single low-memory card is therefore unrealistic, and 8 GPUs is often merely a rough starting threshold rather than a sufficient configuration.

    If the context is longer and concurrency higher, the KV Cache will keep increasing memory usage. At this point you can consider KV Cache quantization, such as FP8 or INT8 KV Cache. Here it’s worth distinguishing the concepts: KV Cache quantization is an optimization technique that lowers cache precision, whereas Paged Attention is a memory paging mechanism that improves KV Cache reuse and memory utilization through paging. Paged Attention is not quantization and the two should not be conflated.

    For deployment, you can also combine distributed strategies such as expert parallelism, tensor parallelism, and pipeline parallelism to reduce per-card pressure. Long context, high concurrency, and batch size need to be balanced: a larger batch size may increase throughput, but it also raises memory pressure. A truly deployable configuration must come from capacity calculations and load testing, not from looking only at total parameters.

    6. Pre-Deployment Checklist: From Model Card to Capacity Planning

    Before formal deployment, build a checklist to avoid looking only at “can I download it.”

    1. Parameters and activated parameters: Confirm total parameters and per-token activated parameters, so you don’t misestimate compute using total parameters alone.
    2. Quantization format: Confirm weight and activation precision, such as 4-bit, 8-bit, BF16, FP8, MXFP4, MXFP8.
    3. Weight footprint: Estimate using total parameters times bytes per parameter, and reserve space for metadata.
    4. License risk: Confirm whether commercial use is allowed, whether there are revenue or MAU thresholds, and whether external service is restricted.
    5. Context length: Confirm whether the target context is natively supported by the model or extended by the inference framework.
    6. Concurrency and batch size: Define average request length, peak request volume, batch size, and queuing strategy.
    7. GPU count and per-card memory: Look beyond weights to reserve KV Cache, activations, and framework overhead.
    8. Storage and loading speed: Pulling, caching, and hot-loading TB-scale weights affect time to launch.
    9. Monitoring metrics: Memory usage, throughput, time to first token, long-context latency, error rate.
    10. Fallback plan: Prepare a smaller-parameter model, a low-precision version, or a cloud elastic-scaling option.

    The value of this checklist is that it breaks “can it run” into multiple verifiable questions. If any single item has no answer, deployment risk will surface later rather than sooner.

    7. FAQ: Can Open-Weight Models Be Used Commercially? How Large a Model Can 8 GPUs Run?

    Q: Can open-weight models be used commercially?
    A: Usually yes, but it depends on the license. If the license is close to an MIT-style grant, commercial use is often allowed; but if there are revenue thresholds, MAU thresholds, or external-service restrictions, you may need to obtain authorization in advance.

    Q: Can 8 GPUs definitely run a trillion-parameter model?
    A: Not necessarily. 8 GPUs is only a rough starting configuration for certain 4-bit models. In practice it depends on per-card memory, context length, concurrency, KV Cache, and framework overhead. If memory is insufficient, you may need more cards, lower-precision cache, or a smaller model.

    Q: How should the revenue and MAU thresholds be understood?
    A: A common formulation is that if providing the model as an external service is your core business, and your cumulative revenue over 12 consecutive months exceeds a certain amount, or monthly active users exceed a certain scale, you must sign a separate agreement. The exact amounts and user scale are governed by the original license text.

    Q: What are common deployment mistakes?
    A: Looking only at total parameters and not activated parameters; looking only at weight size and not KV Cache; treating Paged Attention as quantization; overlooking commercial restrictions in the license; underestimating the memory footprint of long context; going live without capacity load testing.

    Q: How do I choose between local and cloud deployment?
    A: Teams doing temporary validation, small-to-medium launches, or handling obvious traffic fluctuations and wanting to iterate quickly are generally better off starting with cloud resources. For long-term, stable, high-load scenarios, you need to compare the total cost of a self-built cluster against cloud resources.

    8. Choosing Cloud Hosting: Which Teams Does It Suit?

    For most enterprises, building a self-hosted 8-GPU starter cluster is not always the optimal choice. Teams doing temporary validation, small-to-medium launches, or handling obvious traffic fluctuations and wanting to iterate quickly are generally better off starting with cloud GPU resources. Cloud solutions can convert a one-time hardware investment into pay-as-you-go operating expense and use elastic scaling to handle peaks.

    Taking a general-purpose cloud platform such as Baidu AI Cloud as an example, four capability areas are worth evaluating closely. First, whether high-memory GPU instances meet the need to keep weights resident in memory. Second, whether object storage can hold TB-scale weights and support fast pulling. Third, whether the container platform can orchestrate inference services and support distributed deployment such as expert parallelism and tensor parallelism. Fourth, whether monitoring and alerting can observe memory, throughput, time to first token, and long-context performance in real time.

    But cloud is not a cure-all. Long-term, stable, high-load scenarios need a comprehensive comparison of the per-token cost of a self-built cluster versus cloud on-demand, reserved, and batch instances. Scenarios with strict data compliance requirements need to confirm whether storage, network, and model weights meet intranet or private-cloud requirements. Whether to choose cloud should be based on utilization, compliance requirements, and total cost, not on the per-hour unit price alone.

    Conclusion: Think Through Three Things Before Deciding to Deploy

    The real barrier to ultra-scale open-weight models is not “can I download it,” but three things: first, understanding the compute math introduced by the MoE sparse architecture; second, evaluating the commercial boundaries of custom licenses clause by clause; third, choosing the right deployment model based on memory calculations. Put parameters, activations, quantization, licensing, context, concurrency, and GPU count into a single checklist, and even a trillion-parameter model can be deployed reliably and affordably.