Trillion-Parameter Sparse Models: Engineering Post-Training and Evaluation
Trillion-parameter MoE models have entered a "post-training-driven" iteration phase, shifting the industry's focus from switching base models to investing heavily in post-training. This article walks through what that shift means in practice: the division of labor between pre-training and post-training, how to read the memory and compute economics of sparse activation architectures, why long context and native multimodality reshape engineering tradeoffs, how to build an evaluation system that turns scores into decisions, and how to treat hallucination governance as a reliability discipline rather than a leaderboard chase. It closes with a practical checklist for measuring true per-task cost and rolling out models safely. The goal is to help engineers and teams choose models not just by headline benchmark scores, but by which capabilities actually match their business, how stable those capabilities are, and whether the total cost is sustainable.
1. From “Swapping the Base” to “Adding Post-Training”: An Important Fork in the Road
In the race among trillion-parameter large models, a new product iteration path is becoming increasingly common: keep the pre-trained base unchanged and pour most of the incremental investment into the post-training stage. A flagship model recently released by a Chinese team is a typical example. It launched in October and opened its weights in early November; the previous version in the same generation scored around 65.7% on terminal-operation benchmarks, while the new version reached about 66.4%, narrowing the gap to 0.7 percentage points. The new version did not swap the base. It has 2.8 trillion parameters, uses a Mixture of Experts (MoE) architecture, activates roughly 104 billion parameters per inference, supports a 1M-token context, and natively accepts text and image inputs. Beyond the benchmark numbers, one signal stands out: the hallucination rate dropped from about 23.4% to about 8.3%. For engineering teams, this shift is often closer to the real user experience than any aggregate score.
1.1 The Division of Labor Between Pre-Training and Post-Training
Pre-training learns general language and world knowledge from massive corpora. It determines the model’s “ceiling” and breadth of knowledge, but it is expensive, slow, and unforgiving. Post-training covers supervised fine-tuning (SFT), preference optimization, and reinforcement learning, and determines “how the model gets things done,” directly affecting:
- Instruction following and format constraints;
- Multi-step reasoning and planning;
- Tool calling and structured output.
You can think of pre-training as laying the foundation and post-training as the interior fit-out and final acceptance. What users directly perceive as “usable or not” is largely shaped during post-training.
1.2 The Trade-off Between “Swapping the Base” and “Adding Post-Training”
| Dimension | Swapping the Pre-Trained Base | Adding Post-Training on the Same Base |
|---|---|---|
| Training cost | High; requires re-running pre-training | More controllable; focuses compute on post-training |
| Iteration cycle | Long | Short; can release continuously by version |
| Capability ceiling | Can lift the model overall | Bounded by the base’s ceiling |
| Main risks | Training instability, emergent unknown capabilities | Catastrophic forgetting, capability collapse, preference overfitting |
| Suitable scenarios | Structural gaps in base capability | Improving experience and reliability for real tasks |
It is worth stressing that adding post-training compute is a mainstream industry approach, not something unique to any one team. It is better suited to solving “knows but can’t do it well” problems, such as unstable instruction following, tool-calling errors, and interrupted long workflows. If a model has structural deficiencies in foundational knowledge or reasoning depth, post-training alone will struggle to close them.
1.3 Three Engineering Signals From the New Version
- Faster iteration cadence. The short interval between release and open weights suggests the organization has already turned post-training into a pipeline. Data construction, training, evaluation, and release can proceed in parallel, so enterprise selection should focus on sustained iteration capability rather than a leaderboard rank at a single point in time.
- Improvement landed on the capability most sensitive at the application layer. Terminal-operation ability means the model can decompose a request into a sequence of commands, observe outputs after execution, locate errors, and correct them. This is precisely the core capability behind AI coding tools and operations assistants. It separates models more sharply than general Q&A and better reflects where post-training investment went.
- A large drop in hallucination rate. With the rate down to about 8.3%, roughly a 65% relative decline over the previous generation, this change often decides project success more than an aggregate score does for knowledge-base Q&A, business retrieval, and code Q&A.
1.4 The Prerequisites for Realizing the Gains
When fine-tuning on a post-trained version, common approaches include full-parameter fine-tuning, parameter-efficient methods such as low-rank adaptation (LoRA), and preference optimization and reinforcement learning methods such as direct preference optimization (DPO) and group relative policy optimization (GRPO). But whether incremental gains are retained depends on data mixing ratios and regression protection.
# Example: capability regression check before and after post-trainingdef release_gate(candidate, baseline, suite, threshold=0.0):"""candidate: fine-tuned model (same base)baseline : base modelsuite : general capability regression set, e.g., general knowledge,math reasoning, long-context understandingReturning False signals a capability-collapse risk; do not release."""report = {}for name, task in suite.items():base_score = evaluate(baseline, task)cand_score = evaluate(candidate, task)delta = cand_score - base_scorereport[name] = {"baseline": base_score,"candidate": cand_score,"delta": delta}regressed = [k for k, v in report.items() if v["delta"] < threshold]if regressed:print("The following capabilities regressed; adjust the data mix:", regressed)return report
The practical lesson: the general-capability regression set must be given equal weight to the task-specific evaluation set. Otherwise you easily end up with “specialized task scores rising while general capability declines,” which surfaces later in long-tail scenarios after launch.
2. Understanding Sparse Activation Architecture: Engineering Implications for Memory, Long Context, and Multimodality
Discussions of trillion-parameter models often fixate on parameter count, but from an engineering standpoint it is more useful to separate two different accounts: total parameters and activated parameters.
2.1 Two Accounts: Memory Depends on Total Parameters, Compute Depends on Activated Parameters
| Metric | Meaning | Primary Impact | Engineering Lever |
|---|---|---|---|
| Total parameters (2.8T) | Total weight volume across all experts and shared layers | Memory/VRAM capacity, weight loading time, parallelism slicing complexity | Tensor parallelism, expert parallelism, low-precision quantization |
| Activated parameters (~104B) | Experts actually involved in the forward pass per token | Per-token compute and latency | Batch throughput, operator fusion, serving concurrency tuning |
Engineering can be understood with a simplified single-GPU view. Real deployments inevitably involve multi-GPU parallelism, but the order-of-magnitude relationships below still hold:
- Memory bandwidth constraint. Each GPU can only use the portion of weights it holds. If weights are stored in BF16 (2 bytes per parameter), the bytes read from memory per forward pass is roughly number of weight parameters × 2.
- Arithmetic intensity. At batch size 1, only about 2 floating-point operations are executed per byte read, a classic memory-bound scenario. Throughput is then determined by memory bandwidth, and the GPU’s peak compute sits largely idle.
- Scaling with batch size. As concurrent requests grow, arithmetic intensity rises roughly linearly with batch size. As long as the compute for the activated experts is not saturated, throughput can increase significantly with concurrency, and the average per-request cost is amortized accordingly.
For MoE models, this means two typical phenomena:
- Memory capacity determines “whether it fits”;
- Concurrency scale determines “whether the unit cost is economical.”
2.2 Long Context: The Underrated Bandwidth and Memory Cost
Take 1M tokens as an example. In autoregressive generation, every new token must interact with the entire history. Under a naive attention implementation, single-step attention compute is roughly 2 × L² × d, where L is the sequence length and d is the model’s hidden dimension. When generating 2048 tokens, the sequence length grows from 1M to about 1.002M:
- The increase in attention compute is about 2 × 2048 / 1,000,000 ≈ 0.4%;
- But the KV cache grows linearly, adding storage for an extra 2048 × d × 2 elements.
In other words, the cost of long context comes mainly from memory capacity and bandwidth, not floating-point compute. Common mitigations in the industry include:
- Blocked attention (such as FlashAttention-style implementations) to reduce memory footprint;
- Sliding-window attention, KV-cache compression, or quantization;
- Retrieval first: recall the fragments relevant to the question, then feed them to the model.
Engineering advice: the context window is a “capability ceiling,” while retrieval quality determines “actual utilization.” Rather than stuffing an entire document library into the context, build a high-quality retrieval and reranking pipeline, then control what gets injected with a layered context strategy:
# Example: layered context assembly (pseudocode)def build_context(query, retriever, kv_store, max_tokens=32_000):# 1. Retrieval layer: recall the most relevant snippets and rerank themsnippets = rerank(retriever.search(query, top_k=50))[:8]# 2. Summary layer: rolling summary of conversation history,# keeping conclusions and key constraintshistory_summary = summarize(kv_store.recent_turns())# 3. Rerank layer: trim to budget, dropping lower-relevance items firstcontext = budget_fill(blocks=[{"role": "system", "text": SYSTEM_PROMPT},{"role": "assistant", "text": history_summary}] +[{"role": "system", "text": s.text} for s in snippets],max_tokens=max_tokens,)return context
2.3 Native Multimodality: Unified Representation and Context Budget
Native multimodality means the model encounters both text and images during the pre-training stage, rather than training a language model first and then attaching a vision encoder. Its engineering value shows up in two ways:
- Unified representation. Images and text share the same semantic space, making mixed image-text understanding more natural;
- End-to-end optimization. Cross-modal instruction following (such as “generate a corresponding query from this image”) is more stable.
However, image inputs consume large numbers of visual tokens. Two common approaches exist:
- Fixed-grid splitting, for example mapping each small block to one token;
- Compressing the visual representation through pooling.
A single high-resolution screenshot can easily occupy thousands of tokens. In a million-token window, the share of visual tokens can even exceed that of text, further intensifying memory and latency pressure. Engineering advice:
- Set resolution and compression strategies according to the task, avoiding “full-resolution input everywhere”;
- Cache reusable image descriptions;
- Keep a dedicated “mixed image-text task” section in the evaluation set, so pure-text evaluation does not mask multimodal weaknesses.
3. Designing a Trustworthy Evaluation System: Turning “Scores” into “Conclusions”
3.1 Leaderboard Scores Cannot Be Compared Directly
The “0.7 percentage point” mentioned at the start only shows that the two versions are very close on a terminal-operation benchmark. Whether that supports a judgment of superiority depends on variance and the evaluation protocol. Public leaderboards and your own scenarios differ in several ways:
- Different prompts and scaffolding. Few-shot examples, system prompts, and tool interface definitions can all significantly change scores.
- Different sampling parameters. Temperature, number of samples, and pass@k strategies make results incomparable.
- Different benchmark versions. Question sets are updated continuously, so cross-version comparisons need to note the version number.
- Different coverage domains. Public benchmarks cannot cover enterprise-private data and business workflows.
3.2 Reading Scores: Difficulty Type Matters More Than the Total
The same multiple-choice question is obviously too easy if half of them can be answered by random guessing. So analysis should focus on these metrics:
- Difficulty structure. Break results down by easy, medium, and hard tiers; differences hidden by the total score often live in the hard tier.
- Calibration quality. Reliability and confidence error (such as ECE) measure whether the model’s confidence in its own answers matches its accuracy, directly determining whether an abstention strategy works.
- Process metrics. In terminal and browser tasks, single-step success rate, tool-call legality rate, average number of interaction turns, and share of invalid operations locate problems better than final pass rate.
- Discrimination. If the old and new versions score nearly identically across all subsets, the evaluation set is insensitive to this round of improvements and needs to be expanded or redesigned.
3.3 Building Your Own Evaluation Set: A Four-Step Method
- Collect. Sample from real business logs, covering high-frequency, long-tail, and failure cases, with desensitization applied.
- Annotate. Provide reference answers for objective questions; for open-ended questions, use “reference answer + scoring rubric”; supplement with human spot checks to calibrate the automatic grader.
- Stratify. Build subsets by task type and difficulty, such as terminal operation, retrieval Q&A, code generation, multimodal understanding, and long-context understanding.
- Govern. Use version control, update regularly, prevent the test set from leaking into training data; for sampling-based evaluations, run multiple repetitions and report the mean and confidence interval.
# Example: unified evaluation and comparison across model versions (pseudocode)def compare(models, suite, repeats=3):rows = []for m in models:for subset_name, subset in suite.items():accs = [evaluate(m, subset) for _ in range(repeats)]rows.append({"model": m.name,"subset": subset_name,"accuracy_mean": mean(accs),"accuracy_std": std(accs),"latency_p95_ms": measure_latency(m, subset),"tokens_per_task": measure_tokens(m, subset),})return rows
| Subset | Core Capability | Key Metrics |
|---|---|---|
| Terminal operation | Command generation, error recovery, multi-step planning | Pass rate, average turns, invalid-operation rate |
| Retrieval Q&A | Information retrieval, evidence aggregation, multi-source comparison | Answer accuracy, hallucination rate, citation coverage |
| Code generation | Code correctness, maintainability | Compile/unit-test pass rate, number of static-check issues |
| Multimodal understanding | Image-text alignment, layout parsing | Field-extraction accuracy, format-compliance rate |
| Long context | Cross-paragraph reasoning, information localization | Needle-in-a-haystack success rate, cross-paragraph reasoning accuracy |
4. Hallucination Governance: Reliability Engineering Worth More Than Benchmark Chasing
Hallucination refers to content a model generates that looks plausible but does not match the facts or the provided material. It typically shows up in three forms:
- Fabricating nonexistent facts, APIs, fields, or citations;
- Distorting the given material;
- Forcing a definitive answer when information is insufficient.
Dropping the hallucination rate from about 23.4% to about 8.3%, a relative decline of about 65%, matters because:
- In retrieval-augmented Q&A, a substantial share of “errors” come from fabrication and distortion rather than retrieval failure;
- In terminal operations, the model may imagine nonexistent commands, paths, or configuration items, directly causing execution failures or even incidents;
- The hallucination rate correlates positively with human review cost, making it a major operational-expense variable.
A caution is warranted: a lower hallucination rate does not mean the risk is gone. Engineering must look at the following three metrics together:
- Coverage: the share of questions that should be answered but receive an answer;
- Accuracy: the share of provided answers that are correct;
- Over-refusal rate: the share of questions that should have been answered but were refused.
Usability can be approximated with the following relationship:
Usability index ≈ Coverage × Accuracy - λ × Over-refusal rate
Here λ is determined by the business’s tolerance for refusals. Hallucination governance should follow three lines of defense:
| Line of Defense | Typical Methods | Applicable Scenarios |
|---|---|---|
| Training layer | Factual preference data, abstention samples, tool-calling trajectories | Uniformly raising the baseline during version iteration |
| Inference layer | Retrieval augmentation, tool verification, self-consistency voting, structured-output constraints | Rapid mitigation after launch, tuning per scenario |
| Product layer | Citation display, confidence prompts, tiered human review | High-risk business and compliance requirements |
# Example: answer verification based on evidence checking and confidence (pseudocode)def guarded_answer(question, retriever, model, tau=0.60):evidence = retriever.search(question, top_k=8)if not evidence:return {"answer": None, "action": "abstain", "reason": "no_evidence"}draft = model.generate(prompt=build_rag_prompt(question, evidence),require_citations=True, # require per-sentence evidence numbering)checks = {"cited": all_citations_in_evidence(draft, evidence),"faithful": not_contradicts(draft, evidence),"confidence": estimate_confidence(draft, evidence) >= tau,}if all(checks.values()):return {"answer": draft.text, "action": "answer","citations": draft.citations}return {"answer": draft.text, "action": "human_review","failed_checks": [k for k, v in checks.items() if not v]}
4.1 “Rules First” for High-Risk Scenarios
In scenarios such as financial calculations, permission changes, and production-environment operations, you should not rely on the model’s judgment but should put rules first:
- Verifiable calculations. Hand these to a code execution environment or a dedicated tool; do not let the model “do the math in its head”;
- High-risk actions. Generate an execution plan and impact scope first, and dispatch only after human confirmation;
- Full audit trail. Record model outputs, tool calls, and execution results to enable traceability.
5. Cost and Deployment: Translating Leaderboard Rank into Budget
Model selection ultimately answers two questions: what is the cost per task, and can service quality stay stable?
5.1 Key Variables That Affect Cost
- Throughput and concurrency. Sparse activation lowers marginal cost in high-concurrency scenarios. For interactive applications, constrain both time-to-first-token and end-to-end duration.
- Context length. Long context significantly raises prefill-stage cost, so pair it with caching and layered assembly.
- Quantization and precision. Weight quantization reduces memory footprint and bandwidth pressure, but must be paired with a precision regression test.
- Task complexity. Terminal-type tasks often require multiple interaction turns, and actual token consumption can be tens of times that of simple Q&A.
- Cache reuse. Stable prefixes such as system prompts, long documents, and common knowledge are good candidates for cache reuse.
# Example: estimate unit cost by task profile (pseudocode)def estimate_cost(model_cfg, task_profile, qps, scale):# model_cfg: activated params, quantization bit width, KV-cache spec# task_profile: average input/output tokens per task, average turnsweights_bytes = scale["total_params"] * model_cfg["bits"] / 8kv_per_token = kv_cache_bytes_per_token(model_cfg)# Number of inference GPUs: must satisfy both weight capacity and KV-cache capacitycards_for_weights = ceil(weights_bytes * scale["replica"]/ model_cfg["gpu_memory"])cards_for_kv = ceil(qps * task_profile["avg_input_tokens"]* kv_per_token * scale["concurrency_factor"]/ model_cfg["gpu_memory"])cards = max(cards_for_weights, cards_for_kv)tokens_per_day = qps * 86400 * (task_profile["avg_input_tokens"]+ task_profile["avg_output_tokens"])return {"gpu_cards": cards,"tokens_per_day": tokens_per_day,"cost_per_1k_tokens": cards * model_cfg["card_hourly_price"] * 24/ (tokens_per_day / 1000),"notes": "KV cache is estimated with a typical concurrency factor; ""actual values need load-testing calibration",}
5.2 Comparing Deployment Forms
| Form | Advantages | Costs | Applicable Scenarios |
|---|---|---|---|
| Self-hosted inference cluster | Data control, deep customization, controllable unit cost at scale | Complex operations; you must build autoscaling and failover | Large, stable workloads; data-sensitive business |
| Managed model service | Ready to use, elastic scaling, no ops burden | Unit cost for large-scale long-term calls needs accounting | Fast validation; volatile-traffic business |
| Hybrid mode | Self-host the core path; offload peaks and long tail to managed | Requires unified routing, rate limiting, and observability | The end state for most mid-to-large enterprises |
5.3 Cloud Resource Combinations
In public cloud environments such as Baidu AI Cloud, inference services are typically paired with the following general cloud products:
- Object storage: archiving evaluation sets, dataset versions, and model artifacts;
- Message queues: load leveling for asynchronous evaluation tasks and batch inference;
- Container platform: orchestration, elastic scaling, and canary release of inference services;
- Logging and monitoring/alerting: collecting token consumption, latency distribution, error rates, and hallucination-related metrics, with threshold-based alerts.
Before launch, complete at least three types of testing:
- Capacity load testing. Determine each instance’s maximum concurrency and latency inflection point;
- Fault drills. Verify automatic migration and degradation after node failures;
- Cost accounting. Cover peak, average, and trough loads to avoid procuring resources based on the average alone.
6. Summary and Action Checklist
The entry of trillion-parameter sparse models into a “post-training-driven” iteration phase means enterprises now face not just “whose score is higher” but “whose capabilities better match the business, stay stable, and remain affordable.” Proceed along this checklist:
- Clarify the roadmap need. For an overall capability leap, watch base-model iteration; for task adaptation and experience improvement, prioritize evaluating post-training performance.
- Build your own evaluation set. Cover terminal operation, retrieval Q&A, code generation, multimodality, and long context; update regularly and manage with version control.
- Keep an eye on hallucination and calibration. Read hallucination rate, coverage, and over-refusal rate together; for high-risk scenarios, insist on rules-first and human review.
- Model cost by task profile. Estimate true unit cost using activated parameters, context length, concurrency scale, and quantization strategy.
- Insist on canary rollout and regression. Run the full regression set on every model or prompt change, monitor online metrics, and retain the ability to roll back quickly.
Model capability is shifting from “can it answer” to “can it complete the task reliably.” Only by building the three lines of evaluation, governance, and cost together can you keep realizing predictable business value across successive rounds of version iteration.
