A Panoramic View of Open-Weight LLMs: Architecture, Quantization, and Evaluation

1. From “Public Weights” to “Complete Openness”: The Standard Trio

Over the past two years, the release strategy of leading open-weight models has been standardizing. A key signal is that “open” no longer just means releasing weight files; it now forms a complete trio:

1. Complete Weights: Contains all parameter tensors, allowing third parties to fully load, run inference, and fine-tune.
2. Model Card: Systematically discloses architectural parameters (total parameters, active parameters, context window, quantization format), evaluation methods, and benchmark results.
3. Full License: Clearly defines the boundaries for commercial use, redistribution, and derivative models.

For enterprises, this trio directly determines “whether it can be used, how to use it, and what risks are involved.” It is crucial to note: open weights do not equate to free use in the open-source software sense. Many custom licenses come with naming conventions, user scale thresholds, and redistribution conditions. These must be reviewed clause by clause before integration, bringing in legal assessment if necessary. Compared to API calls, private deployment offers data sovereignty, predictable costs, and deep customization, but it also requires taking full responsibility for operations and compliance—a trade-off that must be clearly calculated.

2. Sparse Architecture: Decoupling Total and Active Parameters

Ultra-large open-weight models almost universally adopt the Mixture-of-Experts (MoE) architecture. Its core philosophy can be summarized as: total parameters determine what the model “knows,” while active parameters determine how much computation is required per inference.

The specific mechanism is as follows:

• Each layer consists of a router and multiple parallel expert networks (usually FFNs).
• For each token, the router calculates scores across all experts and activates only the top-k scoring experts.
• The selected experts perform weighted calculations, while the remaining experts are completely bypassed in the forward pass.

In common industry configurations, active parameters typically account for only 3% to 10% of the total. This means a trillion-parameter model can have a single-token computation cost similar to a model with tens of billions of parameters. However, the trade-offs are equally clear:

• VRAM: The weights of all experts must be loaded into VRAM (or distributed across multiple cards via expert parallelism), regardless of whether they are “activated” for a specific token.
• Load Balancing: If the router sends too many tokens to the same expert, the card hosting that expert becomes a bottleneck. Mechanisms like capacity factors and auxiliary balancing losses are needed to mitigate this.

Therefore, MoE inference systems typically combine Tensor Parallelism (TP), Expert Parallelism (EP), and Pipeline Parallelism (PP). Schematic pseudocode:

def moe_forward(x, experts, router, top_k=2):
    # x: [tokens, hidden]
    scores = router(x)                               # [tokens, num_experts]
    probs, topk_idx = topk_softmax(scores, k=top_k)
    out = zeros_like(x)
    for e in range(num_experts):
        hit = topk_idx == e                          # Whether the token's top-k includes expert e
        w = (probs * hit).sum(dim=-1, keepdim=True)  # Corresponding fusion weights
        selected = w.squeeze(-1) > 0
        if selected.any():
            out[selected] += experts[e](x[selected]) * w[selected]
    return out                                        # Unselected experts perform zero computation

3. Million-Token Contexts: The VRAM Bill Behind the Capability

Million-token context windows are becoming standard for flagship models, enabling capabilities like “entire repository code understanding,” “long document review,” and “long-horizon agent trajectories.” However, the primary cost of long context is usually not the attention computation itself, but the KV cache:

def kv_cache_bytes(layers, kv_heads, head_dim, seq_len, dtype_bytes=2):
    return 2 * layers * kv_heads * head_dim * seq_len * dtype_bytes  # Factor 2 corresponds to K and V
# Example: 80 layers, 8 KV heads, head_dim=128, 1 million tokens, FP16
# ≈ 2 * 80 * 8 * 128 * 1_000_000 * 2 bytes ≈ 305 GiB (for a single request!)

Without optimization, the KV cache of a single million-token request can max out several high-end GPUs. Common industry optimization techniques include:

• GQA/MQA: Multiple query heads share a single set of KV heads, compressing the KV cache by an order of magnitude.
• KV Cache Quantization: Storing K/V in 8-bit or even 4-bit formats.
• Sparse/Hierarchical Attention: A hybrid design combining sliding window local attention with a few global tokens.
• Position Encoding Extrapolation: Using scaling and interpolation of Rotary Position Embeddings (RoPE) to allow models trained on shorter contexts to extrapolate stably to longer sequences.
• Context Parallelism: Splitting ultra-long sequences across multiple GPUs to distribute attention computation and caching.

For capacity planning, it is recommended to calculate using this formula first: target concurrency × average context length × KV cache coefficient. This is the true basis for VRAM planning, rather than just looking at the theoretical “maximum supported tokens” metric.

4. Post-Training Reinforcement Learning: The Main Engine for Capability Leaps

A common phenomenon is that new generation models largely retain the underlying architecture of their predecessors with limited changes on the pre-training side. Capability improvements primarily stem from increased reinforcement learning (RL) during the post-training phase. There are three reasons for this:

1. Maturation of Verifiable Rewards: Tasks like math and coding can be automatically graded. Reward signals no longer rely on expensive manual annotation, allowing RL training to scale up significantly.
2. Structured Nature of Agentic Tasks: Real-world workflows are multi-step: plan → call tools → observe results → correct. This long-horizon capability is difficult to fully learn via supervised fine-tuning mimicry. RL, through complete task episodes and turn-level rewards, enables the model to learn strategies for “how to recover after an error.”
3. Evaluation-Driven Iterative Loop: Using real-world evaluation results as targets to direct targeted RL investment in corresponding capabilities, forming an “evaluation—training—re-evaluation” flywheel.

The takeaway for enterprises is that when evaluating models, one cannot just look at architectural parameters. Post-training investment intensity and evaluation system quality are often better predictors of real-world performance.

5. Low-Precision Quantization: Compressing Multi-TB Weights into a Deployable Scale

Weight storage is the first hurdle for deploying ultra-large models. Taking the 2 to 3 trillion parameter open-weight models that have appeared in the industry as an example, let’s calculate using 2.8 trillion parameters:

• 16-bit precision (BF16/FP16): 2 bytes per parameter, requiring about 2.8T × 2B = 5.6TB.
• MXFP4 (Microscaling 4-bit float): About 4.25 bits per parameter (4-bit elements + a shared intra-block scale amortized at ~0.25 bits), requiring 2.8T × 4.25 / 8 ≈ 1.5TB. With a small number of retained high-precision layers and metadata, this totals about 1.5~1.6TB.

The key design of MXFP4 is block-level microscaling: weights are divided into blocks of 32 elements, sharing an 8-bit power-of-two scaling factor within the block, while the elements themselves are quantized to 4-bit floats. This retains the compactness of a 4-bit format while adapting to the dynamic range of each block through the block-level scale, mitigating precision collapse caused by outliers.

def mxfp4_quantize(w, block=32):
    blocks = w.reshape(-1, block)
    scale = pow2_round(amax(blocks) / max_fp4_value)  # One 8-bit power-of-two scale per block
    q = quantize_e2m1(blocks / scale)                 # 4-bit float elements
    return q, scale                                    # ≈ 4 + 8/32 = 4.25 bits/parameter

The benefits of quantization are immediate:

• VRAM Threshold: Estimating with 80GB VRAM GPUs, BF16 weights alone would require about 70 cards, while MXFP4 requires around 20 cards, making single-node multi-GPU deployment feasible.
• Bandwidth and Throughput: The decoding phase is memory-bandwidth bound. Reducing the weight volume by approximately 3.7 times significantly boosts inference throughput.
• Synergy with MoE: Sparse architecture saves compute, and low precision saves VRAM. The combination of the two is what makes interactive deployment of trillion-parameter models possible.

It is important to note that quantization is not lossless. Standard practice retains high precision for a few sensitive layers (such as certain attention projections, embeddings, and the output layer). For quality-sensitive scenarios, one should also combine quantization-aware fine-tuning and run regression comparisons before and after quantization using a proprietary evaluation set.

6. Evaluation Methods: From Static Leaderboards to Real-World Agent Sessions

The “numbers” representing model capabilities are undergoing a methodological upgrade. Traditional static benchmarks face two old problems: data contamination (benchmark questions leaking into training data) and leaderboard saturation (loss of differentiation among high-scoring models). Therefore, the industry is placing more emphasis on real-world usage-based evaluation:

• Samples come from long-cycle agent sessions of real users, rather than artificially constructed questions, with scales needing to reach tens of thousands of interactions to suppress statistical noise.
• Bucketed statistics by scenario (e.g., work, coding, chat), avoiding the situation where “a single total score masks structural differences.”
• Pairwise comparison and net improvement rate: This characterizes the improvement margin of a new version over an old one in real tasks. Compared to absolute scores, the net improvement rate better reflects “how much real gain this iteration brought” and is naturally immune to leaderboard fluctuations.

Proprietary evaluations can draw on this approach: sample anonymized data from real business traffic, bucket by task type, use a fixed baseline model for A/B testing, report confidence intervals rather than single-point scores, and periodically rotate questions to prevent targeted overfitting.

7. Enterprise Deployment Checklist

Synthesizing the above, evaluating and deploying ultra-large open-weight models can proceed according to the following checklist:

1. Compute and VRAM Estimation: Weight VRAM ≈ parameter count × bytes per parameter; add KV cache (based on target concurrency × context length) and activations, reserving a 1.2~1.3x redundancy.
2. Parallelism Strategy: MoE models typically require a combination of TP + EP; plan network topology and inter-GPU bandwidth in advance.
3. Inference Engine Capability Check: Continuous batching, paged KV cache management, MXFP4/FP8 quantization operators, and context parallelism support for long contexts.
4. License Compliance Review: Commercial terms, redistribution obligations, naming and brand usage restrictions.
5. Proprietary Evaluation: Construct a private evaluation set using real business tasks for regression comparisons before and after quantization, and across model versions.
6. Infrastructure: If the cost of a self-built cluster is too high, leverage general capabilities of cloud platforms—such as GPU compute, container services, object storage, model serving, and monitoring—to build a three-stage pipeline: “weight storage—inference service—evaluation monitoring.”

8. Conclusion

The complete release of ultra-large open-weight models marks an industry shift from a “parameter race” to “systematic capabilities”: sparse architecture resolves the contradiction between capacity and compute, post-training reinforcement learning dictates real-world performance, low-precision quantization solves deployment economics, and real-world session-based evaluation redefines what it means to be “truly stronger.” For enterprise technology teams, understanding these four technical mainlines holds more long-term value than memorizing any leaderboard ranking.