New Model Weights Don't Equal Production Readiness: MoE Post-Training and Evaluation in Practice

    Shipping new model weights is not the same as having production-ready capabilities. This article walks through the engineering behind the latest generation of large open-weight models, separating what a base model contributes from what post-training adds. It explains how Mixture-of-Experts (MoE) architectures make trillion-parameter models practical to serve, how to read benchmark scores in a statistical context instead of treating small gaps as conclusions, and how to budget GPU memory and choose parallelism strategies. It closes with guidance on building a three-tier evaluation system and on governance topics such as licensing, data security, observability, and version management. Developers and technical decision-makers will find reusable methods for making explainable, reproducible, and maintainable choices.

    By watching how the flagship releases of open-weight models are shipped, we can draw a general conclusion from public benchmark and evaluation data: with the same base model plus more post-training compute, open models have closed much of the gap to top proprietary models on some agentic tasks, and hallucination rates have dropped noticeably. For developers and technical decision-makers, the more valuable questions are not “who came first” but three things: where capabilities come from, how to read benchmark scores, and how to deploy reliably.

    From a flagship release

    A few patterns in the industry are worth noting:

    1. Faster iteration cycles: The gap between a launch event and the release of weights is often one to two weeks. What developers receive are real files, and evaluation has shifted from “watching demos” to “inspecting weights.”
    2. Heavier post-training investment: Some flagship releases do not retrain the base model; instead they pour substantial compute into post-training on an existing base. So the capabilities come less from “retraining” than from “retraining on top.”
    3. Diverging metric structures: On agentic benchmarks that require real terminal interaction, the gap between open flagships and top proprietary models has narrowed to single digits. Reasoning capabilities (such as browsing/retrieval benchmarks) remain an area where leading products stay ahead. Some open releases have cut their hallucination rate by more than ten percentage points versus the previous generation.

    This article does not evaluate specific products. Instead, following the thread “base model → architecture → post-training → evaluation → engineering → governance,” it breaks down the general technical logic behind these signals and offers a reusable method for making explainable, reproducible, and maintainable decisions.

    One base model, two training logics

    To understand this release, first distinguish two concepts:

    • Base model: The capability ceiling determined by model architecture plus pretraining, including breadth of knowledge, language ability, and structural features such as MoE and native multimodality.
    • Post-training: On a fixed base model, using higher-quality supervision data and finer optimization objectives to turn latent capability into controllable, callable, and trustworthy real-world performance.

    The two are in a “store energy” versus “release energy” relationship. Once data mix, parameter scale, and training compute all reach a certain magnitude, further scaling pretraining yields diminishing capability gains; shifting more compute into post-training converts existing capability into task performance more efficiently. So when evaluating a model release, you should not only ask whether the base model was replaced, but also which post-training stages it went through.

    Post-training usually has three stages:

    1. Supervised fine-tuning (SFT): Using high-quality “instruction–response” samples to teach the model output formats, task workflows, and domain knowledge.
    2. Reward modeling or preference optimization (RM/DPO): Using human or model judgments to push outputs from “reasonable” to “better.”
    3. Online reinforcement learning (RL): Using automatically verifiable reward signals or reward models so the model repeatedly tries and learns more robust long-horizon decision strategies.

    If post-training covers real task scenarios such as terminal operation and tool calling, the reliability and consistency of model outputs improve markedly, rather than only pushing up scores on static multiple-choice questions. This also explains why “same base model, big capability jump” is plausible.

    MoE architecture: making trillion-parameter models affordable

    Sparse activation: decoupling total parameters from inference compute

    A Mixture-of-Experts (MoE) layer contains multiple parallel feed-forward subnetworks (experts), and a lightweight router decides which experts handle each token. The default routing parameters are:

    1. total_params = attention & embedding params + num_experts * expert_params + router_params
    2. active_params = attention & embedding params + top_k * expert_params + router_params

    Using public parameter configurations as a reference: about 2.8 trillion total, about 104B activated per token, top-k = 8. Substituting these values:

    • About 97% of parameters sit idle. Shared parameters such as attention make up a similar share in both active and total params; while the expert portion totals about 2.24T / 0.416T ≈ 5.4, on the same order as top_k=8 / (1 - 1/8) ≈ 9.1. This shows that most of the parameter count hides in unselected experts.
    • Each expert activates about 280B parameters, a “very large expert” design. The attention part is the “skeleton” shared by all tokens; the experts are “plug-in brains” that can be called on demand.

    From this we can draw three engineering conclusions:

    1. Compute allocated on demand: Fewer experts are routed to simple questions; complex questions activate more experts. The average number of activated parameters is the lower bound for capacity planning, while actual load fluctuates with the input distribution.
    2. Long-context cost is higher than model size suggests: In million-token contexts, the cost of attention and KV cache does not grow linearly with activated parameters, but grows with sequence length.
    3. Capacity and routing strategy determine real performance: The weaker the load-balancing constraint during training, the more uneven expert utilization at inference.

    Multimodality: input long tail from unified tokens

    Native multimodality means visual information is trained together with text during pretraining, rather than bolting on a vision module at inference time. A common approach splits an image into fixed-size patches, maps them through a vision encoder into vectors of the same dimension as text, then concatenates them with text tokens and feeds them into a unified Transformer.

    A generic implementation sketch (pseudocode):

    1. def build_multimodal_input(image, prompt, cfg):
    2. # 1. Preprocess: resize to a max side, then split into 14x14 patches
    3. patches, grid = patchify(resize(image, cfg.max_side), patch_size=14)
    4. # 2. Vision encoding: project each patch into a text-dimension vector
    5. visual = cfg.vision_encoder(patches) # [num_patches, dim]
    6. visual = cfg.projector(visual) # align to the language model dimension
    7. # 3. Insert visual tokens at the placeholder in the prompt
    8. text = cfg.tokenizer(prompt) # [1, seq_len]
    9. inputs = insert_at_placeholder(text, visual, marker=cfg.image_token_id)
    10. return inputs, grid # grid records image patch info

    Two pitfalls on this path:

    • Resolution determines token count: A high-resolution image can expand into thousands of visual tokens. In a million-token context budget, ten screenshots can consume a few percent of the budget, so for UI-analysis tasks you should compute the input budget first.
    • Modality alignment quality: Whether the vision encoder and language model are jointly trained sufficiently directly determines whether image understanding is reliable.

    How to read benchmark scores: putting “0.7 points apart” back into statistical context

    Terminal-interaction benchmarks: why open models catch up to proprietary ones

    Terminal tasks usually require the model to complete a set of operations in a real command environment—for example, fixing a code repository, running tests, locating errors, and modifying configuration—with pass/fail given by test scripts or a dedicated scorer. Compared with multiple-choice questions, these benchmarks are closer to real development scenarios.

    Open models approaching the leading level on such benchmarks indicates at least three things:

    1. Post-training is clearly effective: Through online RL and similar methods, the model can learn multi-step decision strategies like “read output → locate problem → retry.”
    2. Routing strategy can converge: In policy optimization, if multiple expert paths contribute similarly to the result, different trajectories activate different experts, making the task success rate more stable—effectively an implicit form of load balancing.
    3. Prompt sensitivity is reduced: A dedicated scorer replaces “model grading model,” so results are more objective and closer to engineering acceptance criteria.

    Hallucination rate: closer to business than benchmarks

    According to public information, a flagship release cut its hallucination rate from 23.4% in the previous generation to 8.3%. This change is often more meaningful than the score itself:

    • Error probability drops from about one in four to about one in twelve: In business, “wrong once” and “wrong four times” are completely different experiences.
    • Hallucination has several subtypes, which need separate treatment:
      • Factual hallucination (fabricating from nothing)
      • Contextual hallucination (not faithful to given material)
      • Instruction-conflict hallucination (answer contradicts set constraints)
      • Tool-call hallucination (fabricating nonexistent functions or fields)

    In practice, you can characterize hallucination with confidence calibration (such as ECE, expected calibration error), abstention rate and abstention accuracy, and citation coverage—rather than relying on a single ratio.

    Why 0.7 points cannot be an absolute conclusion

    On a terminal benchmark, if the leading model scores 66.4% and a model scores 65.7%, the gap is only 0.7 percentage points. Statistically, a gap of this magnitude needs careful interpretation:

    1. Sample noise: A single model output is itself stochastic; the same question may pass on one run and fail on another. When the evaluation set is small, results fluctuate with sampling.
    2. Environment differences: Container images, dependency versions, network availability, and timeout settings all change the pass rate.
    3. Scoring error: Rule-based or model-based scorers have their own misjudgments, and may not be comparable across versions.
    4. Question difficulty distribution: Even a slight change in the question bank shifts absolute values.

    A simple thought experiment illustrates the point: with a per-question pass rate of p, the probability of at least one success over n independent runs is 1 - (1 - p)^n.

    1. def pass_at_least_once(p: float, n: int) -> float:
    2. """Probability of at least one success over n runs when the per-run pass rate is p"""
    3. if not (0.0 <= p <= 1.0):
    4. raise ValueError("p must be in [0, 1]")
    5. if n < 1:
    6. raise ValueError("n must be >= 1")
    7. return 1.0 - (1.0 - p) ** n
    8. for p in (0.55, 0.60, 0.65):
    9. print(p, [round(pass_at_least_once(p, n), 3) for n in (1, 3, 5)])
    10. # 0.55 [0.55, 0.909, 0.982]
    11. # 0.60 [0.6, 0.936, 0.99]
    12. # 0.65 [0.65, 0.957, 0.996]

    When a model is allowed multiple attempts, two models whose per-question pass rate differs by only 0.05 can both exceed a 90% first-attempt success rate. To judge whether two models are “really” different, prioritize:

    1. Variance: the result distribution across multiple independent evaluations;
    2. Confidence intervals: for example, bootstrap-based confidence intervals;
    3. p-values: a paired test when treating two output sequences as paired samples.

    If a difference falls within noise, it should not be treated as a conclusion.

    Browsing/retrieval benchmarks: retrieval augmentation is not a cure-all

    Browsing/retrieval tasks test whether a model can locate key information in web pages or retrieval results and perform multi-step reasoning. Leading products still clearly lead on such benchmarks, usually thanks to two kinds of accumulated experience:

    • Retrieval-style training data: post-training makes heavy use of real retrieval trajectories as training signals;
    • A more mature data loop: the retrieve–read–reason–verify pipeline is refined more finely.

    For users of open models, this is exactly an opportunity: after connecting an enterprise’s own knowledge base, the gap in domain Q&A often shrinks substantially, because the model’s general retrieval ability transfers to enterprise corpora.

    Engineering deployment: from weights to a stable service

    Memory budget: estimate first, then choose parallelism

    A trillion-scale MoE model cannot do inference on a single GPU; you must split it with parallelism. A practical estimation flow:

    1. Weight memory: Estimate per-token compute from the number of activated parameters; estimate resident memory from the total parameters (including all experts). In FP16/BF16, about 2.8T total corresponds to about 5.6TB of weights; with 4-bit quantization, the theoretical value can drop to about 1.4TB, but in practice you must add quantization metadata and framework overhead.
    2. KV cache: With length 1 million, about 60 layers, GQA grouping to 8 KV heads, head dimension 128, in FP16 each token needs about 2 * 60 * 8 * 128 * 2 ≈ 245KB; a million-token context is about 245GB—effectively stacking on top of memory equivalent to about half the weight size.
    3. Concurrency and activation: account for concurrent requests, activation values, framework reservations, and communication buffers.
    1. def estimate_kv_cache_gb(seq_len, layers, kv_heads, head_dim, bytes_per_elem):
    2. # KV cache = 2 (K and V) * layers * kv_heads * head_dim * bytes per element * seq_len
    3. per_token = 2 * layers * kv_heads * head_dim * bytes_per_elem
    4. return per_token * seq_len / 1e9
    5. print(estimate_kv_cache_gb(1_000_000, 60, 8, 128, 2)) # ~245.76 GB
    1. Parallelism strategy:

      • Tensor parallelism (TP) suits multi-GPU within one machine;
      • Expert parallelism (EP) distributes different experts across devices and needs to handle All-to-All communication overhead;
      • Pipeline parallelism (PP) suits scaling across nodes.

      When combining the three, communication bandwidth should be the primary constraint.

    Five tuning points at the service layer

    1. Quantization: 4-bit/8-bit quantization can significantly compress memory usage but may cause precision loss. The difficulty for MoE models is uneven expert weight distributions, which requires per-layer calibration.
    2. Prefix caching: System prompts, tool definitions, and knowledge-base retrieval results recur across calls; reusing the prefix KV cache can significantly reduce first-token latency.
    3. Continuous batching: dynamically inserting new requests into batches that are already decoding improves GPU utilization.
    4. Routing monitoring: monitor how often each expert is activated and the token distribution to detect routing skew at inference.
    5. Context compression: summarize long conversations and use sliding windows to control KV cache growth.

    Cloud vs. private deployment: choosing between two paths

    • Private deployment: suits scenarios with sensitive data, deep customization needs, and stable call volume; the main cost is GPU resources and operations.
    • Cloud API calls: suits fast validation, large peak fluctuations, and small teams; the main cost is per-unit price and rate limits.

    Whichever path you choose, the underlying storage, logging, and monitoring systems need to be planned ahead. Baidu AI Cloud, for example, offers object storage for archiving model weights and evaluation samples, a logging service for recording inference requests and routing distributions, and a monitoring and alerting system for tracking latency and error rates. These are all general cloud infrastructure capabilities; refer to the official documentation for specific products and specifications.

    Evaluation system: turning benchmark scores into reusable decision input

    It is advisable for enterprises to build a “three-tier evaluation” system:

    1. Public benchmark tier: used to track capability changes across model versions, but only as a reference, not directly for selection.
    2. Internal evaluation tier: business experts write 100–500 real task samples covering typical scenarios and edge cases; fix the scoring scripts and run periodic regression.
    3. Online canary tier: launch with a small traffic share, collect user feedback and failure samples, and feed them back into the evaluation set.

    Recommended scoring dimensions include at least:

    • Task success rate: whether the end-to-end goal is achieved;
    • Hallucination rate: counted separately for factual, contextual, and instruction-conflict cases;
    • Average turns and token consumption: measuring efficiency;
    • Abstention rate and abstention accuracy: measuring the ability to “say I don’t know when you should say I don’t know”;
    • P95 latency: measuring user experience.

    Governance and risk control

    1. Licensing and compliance: open weights do not mean unrestricted commercial use; you need to check the model license, data-source compliance, and output-responsibility boundaries.
    2. Data security: for private deployment, evaluate network isolation and access control of the inference service; for cloud calls, clarify whether data is used for training.
    3. Observability: record input summaries, routing distributions, outputs, and scoring results for every inference, to support tracing and auditing.
    4. Version management: post-training versions iterate frequently; establish version registration and rollback mechanisms to avoid “new weights live means an outage.”

    Conclusion

    The signal from this flagship release is clear: the base model sets the ceiling, post-training sets usability, evaluation sets trustworthiness, and engineering sets cost.

    For developers, rather than chasing “who came first again,” it is better to focus on three things:

    1. Build your own evaluation set;
    2. Master the methods for estimating memory and parallelism;
    3. Bring hallucination governance into your daily workflow.

    When open models are already closing in on leading proprietary products on terminal tasks, what really creates the gap is often not the model itself, but the engineering capability and evaluation system built around the model.