Open-Source Trillion-Parameter MoE Post-Training, Benchmarking, Hallucination Evaluation, and Production Guide

    This article examines the rise of post-training as a decisive factor for open-source large models, using an open trillion-parameter MoE model as a case study. It covers the model’s key specifications, MoE architecture, long-context memory costs, post-training methods, benchmark interpretation, hallucination evaluation, and practical guidance for enterprises evaluating, deploying, and governing such models. It also provides code examples for evaluation, layered routing, and production considerations.

    1. Introduction: Post-training is becoming the main battleground for open models

    Over the past few years, the gap between open-source models and leading closed-source models was mainly defined by parameter scale, data quality, and pre-training compute. Teams with longer pre-training schedules and larger clusters were usually the first to release flagship models.

    Since the second half of 2025, this landscape has changed noticeably. One open-source team released a flagship model in late October and opened the weights in early November. The model has about 2.8 trillion total parameters, activates about 104 billion parameters per inference pass, supports a context window of 1 million tokens, and natively accepts text and image inputs.

    What is most notable is not that it was retrained from scratch, but that the team kept the original base model and shifted a large amount of compute into post-training.

    The results are representative: on an agentic terminal-operation benchmark, the gap with leading closed-source flagship models narrowed to 0.7 percentage points. At the same time, the hallucination rate dropped from 23.4% in the previous generation to 8.3%.

    This raises a more practical question for enterprise model selection: when benchmark scores become closer, how should teams judge real-world usability?

    Many teams ask three questions:

    • How can memory usage for a trillion-parameter MoE model be estimated?
    • How can hallucination be evaluated for open-source models?
    • Is post-training suitable for enterprise deployment?

    This article addresses these three questions.

    2. Version analysis: What kind of upgrade is this?

    2.1 Keeping the base model and increasing post-training

    Large model iteration generally follows one of two paths:

    • Retrain the base model: Pre-train from scratch or from an intermediate stage. This allows adjustment of data mixtures and architecture, and may raise the upper limit, but it requires enormous compute.
    • Post-training enhancement: Freeze or slightly adjust the base model, then add supervised fine-tuning, preference optimization, reinforcement learning, and other training stages.

    The model in this case follows the second path: the base model is unchanged, while compute is concentrated in post-training. This is a pragmatic engineering choice.

    First, the base model already has language understanding, knowledge coverage, and code capabilities. Repeating pre-training often brings diminishing marginal returns. Second, usability-related abilities, such as instruction following, tool calling, multi-turn consistency, and output formatting, are largely shaped by post-training. Third, keeping the original base allows existing ecosystems, fine-tuning results, and deployment experience to transfer smoothly, reducing upgrade cost.

    It is worth noting that open weights do not mean unrestricted commercial use. License terms, redistribution restrictions, and derivative model requirements should all be reviewed by enterprises before use.

    2.2 Open-source flagship: parameter scale does not equal inference cost

    Based on public information, the model has four key specifications:

    • Parameter scale: about 2.8 trillion total parameters;
    • Sparse structure: a mixture-of-experts architecture that activates about 104 billion parameters per inference pass;
    • Context length: supports 1 million tokens;
    • Modality capability: native multimodal support for text and images.

    These four specifications look simple, but they directly determine inference cost, memory requirements, and deployment patterns.

    The most important point is this: total parameters and activated parameters carry completely different costs.

    In a mixture-of-experts model, each token is routed to only a small number of experts, so computation is related to activated parameters. However, all expert weights must remain resident in memory. Otherwise, every routing step would require reloading parameters, causing latency to rise sharply.

    Compare the two numbers:

    2.8 trillion ÷ 104 billion is approximately 27, meaning that on average only 1 out of every 27 parameter copies participates in computation for a single token.

    This brings considerable savings in computation, but it does almost nothing to relieve memory pressure. This is also why a sparse model may still require multiple machines and multiple GPUs when deployed in the cloud.

    3. Architecture deep dive: MoE, long context, and native multimodality

    3.1 MoE: capacity through on-demand experts

    The core idea of mixture-of-experts architecture is to replace a single feed-forward network with multiple parallel feed-forward networks, called experts, and use a gating router to select a small number of experts for each token.

    The mainstream approach is Top-K routing: select the K experts with the highest scores, where K is usually 1 or 2. Gating weights are generally normalized over the selected experts.

    The following code illustrates the computation logic only:

    1. import torch
    2. def moe_forward(x, router, experts, top_k=2):
    3. """Illustrative MoE forward pass: routing decides which experts process each token."""
    4. # router(x) has shape [number of tokens, number of experts], and experts is a ModuleList
    5. logits = router(x)
    6. weights, indices = torch.topk(logits, k=top_k, dim=-1)
    7. weights = torch.softmax(weights, dim=-1) # Normalize only over selected experts
    8. output = torch.zeros_like(x)
    9. for t in range(x.shape[0]):
    10. for w, idx in zip(weights[t], indices[t]):
    11. # Note: real implementations use grouped matrix multiplication to avoid token loops
    12. output[t] += w * experts[int(idx)](x[t : t + 1])
    13. return output

    The advantage of MoE can be summarized as decoupling parameter capacity from computation cost:

    • Total parameters determine the model’s knowledge capacity;
    • Activated parameters determine computation per token;
    • With the same compute budget, MoE can provide much larger capacity than a dense model.

    The costs are also clear:

    • High memory usage: all expert weights must remain resident in memory;
    • Load imbalance: some experts may receive far more tokens than average, affecting inference latency;
    • High communication overhead: in multi-GPU deployment, tokens must be exchanged between devices according to routing results, known as All-to-All communication. This creates a significantly higher dependence on high-speed interconnects than dense models.

    A common industry practice is to combine expert parallelism with tensor parallelism and pipeline parallelism, while using capacity factors, expert dropping, and related strategies to maintain load balance.

    3.2 Million-token context: capability ceiling and cost constraints

    A context length of 1 million tokens means that a single input can contain hundreds of pages of documents, a small-to-medium code repository, or hours of meeting transcripts. However, the cost of length cannot be ignored.

    Consider the KV cache, which is used to reuse Keys and Values for previous tokens during autoregressive generation. For models using grouped-query attention, memory usage can be estimated as follows:

    1. KV cache memory ≈ 2 × layers × KV heads × head dimension × tokens × bytes per value

    Assume a model has 60 layers, 8 KV heads, head dimension 128, and FP16 precision, meaning 2 bytes per value. The KV cache for a single sequence is approximately:

    Context length KV cache for one sequence
    10K tokens about 2.3 GiB
    100K tokens about 23 GiB
    1M tokens about 229 GiB

    Even counting only the cache, a single 1-million-token sequence can require several times the memory of a high-end GPU. This is only for one sequence. Concurrent serving requires the memory to scale further.

    Therefore, long-context deployment usually needs a combination of the following techniques:

    • Use grouped-query attention or multi-query attention to compress the KV cache;
    • Use paged KV cache management to reduce fragmentation and improve utilization;
    • Cache and reuse prefixes to avoid repeated computation;
    • Use quantization to reduce cache precision.

    A more subtle issue is the gap between effective context and nominal context. Many studies have shown that models are less effective at using information in the middle of long texts than information at the beginning and end, a phenomenon often called “lost in the middle.” Retrieval-augmented generation and moving key information earlier are still often irreplaceable in many scenarios.

    3.3 Native multimodality: training method determines capability boundaries

    There are two common multimodal approaches in the industry:

    • Bridging approach: first train a text model, then attach a visual encoder and align it through projection layers;
    • Native approach: mix image-text data during the pre-training stage.

    The advantage of native multimodality is that visual information and language representations are deeply fused early in training. This makes the model more natural at understanding charts, interface screenshots, and layout structures. The cost is that training data preparation and compute investment become more complex.

    For enterprise scenarios that need to process reports, receipts, screenshots, and design files, native multimodality can reduce additional image-understanding stages and shorten the processing pipeline.

    4. Post-training deep dive: why it offers the highest marginal return

    4.1 What post-training actually trains

    Post-training usually contains several connected stages with different objectives:

    1. Supervised fine-tuning: use high-quality instruction-response samples to teach the model to follow instructions, produce structured formats, and adopt roles.
    2. Rejection-sampling fine-tuning: sample multiple answers to the same question, filter correct or high-quality answers using rules or models, and feed them back into training to raise the capability ceiling.
    3. Preference optimization: construct preference data through human feedback or AI feedback, and optimize responses for usefulness and harmlessness.
    4. Verifiable reinforcement learning: in domains such as mathematics and code, where answers can be judged automatically, use verifiable rewards to encourage the model to explore better solutions.

    Each stage requires different amounts of compute and data. Additional post-training compute is usually invested mainly in verifiable reinforcement learning and preference optimization, because these methods are most sensitive to compute and most directly affect agentic task performance.

    4.2 Post-training data quality determines the ceiling

    Post-training is not simply “more compute is better.” High-quality instruction data, accurate preference labels, and automatically verifiable task environments are all necessary.

    A common misconception is that capabilities the model “does not know” can be “taught” through post-training. In practice, post-training mostly activates and calibrates capabilities already present in the base model: knowledge and abilities absent from the base model are difficult to create from nothing, but abilities that exist yet are not expressed consistently can be significantly improved.

    4.3 Agent capabilities: the most direct beneficiary of post-training

    Agentic tasks such as terminal operation, code execution, and multi-step tool calling are essentially long-chain decision processes:

    • each step must interpret environment feedback;
    • decide the next action;
    • handle failures and retries.

    These tasks cannot be completed simply by “knowing the answer.” They require large amounts of trajectory data from interacting with environments. Post-training improves this through methods such as:

    • constructing multi-step tool-calling trajectories;
    • using execution feedback as reward signals;
    • training models to recover from errors.

    This explains why a model based on the original base can approach leading closed-source models on terminal-operation benchmarks.

    5. Benchmark deep dive: what does a 0.7-point gap mean?

    5.1 Benchmark scores are sample averages with inherent variance

    On the terminal-operation benchmark, the product scored 65.7%, the leading closed-source flagship scored 66.4%, and another closed-source model scored 58.2%. A lead of 0.7 percentage points seems small, but several points matter when interpreting it:

    • Sample size and variance: benchmarks have limited tasks, and variation on a single task can cause several percentage points of difference. A rigorous approach is to run evaluations multiple times, calculate confidence intervals, and then determine whether the gap is significant.
    • Scoring criteria: whether the result is first-attempt success rate or success rate after multiple attempts. Different criteria cannot be directly compared.
    • Environment and prompts: differences in tool versions, system environments, and prompt templates can all affect results.
    • Version alignment: both compared models should be checked to confirm they are the latest versions.

    One way to understand this is: a 0.7-point gap means the two models are already in the same tier for agentic terminal tasks. The real differences will appear in specific task distributions. Some scenarios may be more stable with closed-source models, while others may favor open-source models. Enterprises should run tests based on their own tasks rather than relying only on total scores.

    If enterprises want to reproduce the evaluation, they should at least record:

    • benchmark version;
    • prompt template;
    • tool version;
    • number of runs;
    • retry strategy;
    • environment image;
    • confidence intervals.

    5.2 Self-published leaderboards need cross-validation

    On another browsing and retrieval benchmark, the product scored 92.1% and ranked first in the table published by the releasing party. Such self-published leaderboards should be treated cautiously:

    • Benchmark selection: publishers tend to choose benchmarks that align with their own strengths;
    • Evaluation settings: prompts, tool permissions, and retry strategies can affect results;
    • Compared models: whether the latest competitor versions are included.

    After the weights are opened, the community can reproduce the evaluation itself. This is a structural advantage of open source over closed source. Enterprises should use reproduced results as the basis for decisions rather than staying with numbers from the publisher.

    5.3 Basic principles for horizontal comparison

    Common trap Manifestation Countermeasure
    Inconsistent criteria One uses first-attempt success, another uses best-of-multiple success Unify evaluation scripts and scoring rules
    Version mismatch Comparing against an older competitor version Check release dates and version numbers
    Data contamination Benchmark tasks enter training data Use private holdout sets for validation
    Single aggregate score Looking only at one overall score Break down evaluation by task type
    Environment differences Tools, dependencies, and prompts differ Fix environment images and prompt templates

    6. Reliability deep dive: why hallucination rate matters more than benchmark score

    6.1 Hallucination rate: from 23.4% to 8.3%

    The product’s hallucination rate dropped from 23.4% in the previous generation to 8.3%, a reduction of more than 60%. This is one of the most practically valuable changes in this upgrade.

    Benchmark scores measure “can it do the task?” Hallucination rate measures “is what it says trustworthy?” In scenarios such as intelligent customer service, financial research, medical assistance, and code generation, a fluent but factually incorrect answer can cost far more than one failed benchmark task.

    The reduction in hallucination rate usually comes from three post-training approaches:

    • Faithfulness optimization: in preference optimization, give negative signals for unsupported statements and overly confident expressions;
    • Refusal and clarification ability: when information is insufficient or the question is ambiguous, the model learns to ask clarifying questions or explicitly express uncertainty;
    • Tool use and retrieval augmentation: train the model to proactively call search, databases, or document retrieval when factual support is needed, instead of answering from memory.

    6.2 Three approaches to evaluating hallucination

    Method Approach Advantages Limitations
    Human annotation Experts judge factual correctness item by item Accurate and interpretable High cost and difficult to scale
    Model review A strong model scores responses according to criteria Low cost and scalable Review bias exists and requires calibration
    Tool verification The model outputs citations, then they are automatically checked Objective and automatable Depends on verifiable data sources

    In engineering practice, a combination is recommended: use a small-scale human-labeled golden set for calibration, use model review for large-scale screening, and use tool verification as a fallback in production.

    6.3 Hallucination rate also depends on the definition

    Like benchmark scores, hallucination rate depends on the evaluation set and scoring criteria:

    • Different coverage of knowledge domains leads to different results;
    • Factual hallucination and faithfulness hallucination, where the response is inconsistent with given materials, are counted differently;
    • Whether refusals are included and whether multi-turn dialogue is included also affect the number.

    Therefore, enterprises should not directly treat 8.3% as the hallucination rate for their own scenario. They should build their own evaluation set and track it continuously.

    7. Deployment guide: how enterprises can evaluate, deploy, and govern

    7.1 Step 1: build a proprietary evaluation set

    Public leaderboards cannot replace business-specific evaluation. Enterprises can build their own evaluation set through the following steps:

    1. Scenario sampling: sample from real historical requests, covering both high-frequency and long-tail scenarios;
    2. Golden annotation: have business experts label standard answers and scoring points;
    3. Adversarial samples: construct leading questions, ambiguous questions, and questions beyond the knowledge boundary to test hallucination and refusal behavior;
    4. Regression baseline: after each model upgrade, run the fixed evaluation set to form comparable curves.

    The following code illustrates the process. judge, check_faithfulness, and is_refusal are replaceable stubs:

    1. from dataclasses import dataclass
    2. from typing import Optional, Protocol
    3. class GenerativeModel(Protocol):
    4. def generate(self, prompt: str, context: Optional[str] = None) -> str:
    5. ...
    6. @dataclass
    7. class EvalCase:
    8. id: str
    9. prompt: str
    10. golden: str
    11. context: Optional[str] = None
    12. should_refuse: bool = False
    13. def judge(answer: str, golden: str) -> bool:
    14. # Stub: in production, replace with rules, human review, or a strong model review
    15. return answer.strip().lower() == golden.strip().lower()
    16. def check_faithfulness(answer: str, context: Optional[str]) -> bool:
    17. """Stub: replace with citation checks, model review, or human review in production."""
    18. if not answer.strip():
    19. return False
    20. # This only treats empty answers as failures; real faithfulness requires stronger fact checking.
    21. return True
    22. def is_refusal(answer: str) -> bool:
    23. markers = ("cannot answer", "insufficient information", "cannot confirm")
    24. return any(m in answer for m in markers)
    25. def evaluate_model(model: GenerativeModel, dataset: list[EvalCase]) -> dict:
    26. results = []
    27. for case in dataset:
    28. answer = model.generate(case.prompt, context=case.context)
    29. if case.should_refuse:
    30. results.append({
    31. "id": case.id,
    32. "kind": "should_refuse",
    33. "refusal_ok": is_refusal(answer),
    34. })
    35. else:
    36. results.append({
    37. "id": case.id,
    38. "kind": "answerable",
    39. "passed": judge(answer, case.golden),
    40. "faithful": check_faithfulness(answer, case.context),
    41. })
    42. answerable = [r for r in results if r["kind"] == "answerable"]
    43. should_refuse = [r for r in results if r["kind"] == "should_refuse"]
    44. # The hallucination rate denominator counts only answerable samples; should-refuse samples are measured separately.
    45. hallucination_rate = None if not answerable else sum(
    46. not r["faithful"] for r in answerable
    47. ) / len(answerable)
    48. refusal_accuracy = None if not should_refuse else sum(
    49. r["refusal_ok"] for r in should_refuse
    50. ) / len(should_refuse)
    51. accuracy = None if not answerable else sum(
    52. r["passed"] for r in answerable
    53. ) / len(answerable)
    54. return {
    55. "accuracy": accuracy,
    56. "hallucination_rate": hallucination_rate,
    57. "refusal_accuracy": refusal_accuracy,
    58. "answerable_count": len(answerable),
    59. "should_refuse_count": len(should_refuse),
    60. }

    7.2 Step 2: layered routing to control inference cost

    Trillion-parameter MoE models have high memory costs and are not suitable for serving all requests. A common approach is to build layered routing:

    • Lightweight layer: intent classification, sensitive-word filtering, and simple Q&A are handled by small models or rules;
    • Standard layer: routine business Q&A is handled by medium-sized models;
    • Flagship layer: complex reasoning, long-document analysis, and agentic tasks are sent to the trillion-parameter MoE model.

    The following code is a minimal runnable example, and all components can be replaced with real implementations:

    1. from typing import Optional, Protocol
    2. class ModelService(Protocol):
    3. def generate(self, query: str, context: Optional[str] = None) -> str:
    4. ...
    5. class StubModel:
    6. def __init__(self, name: str):
    7. self.name = name
    8. def generate(self, query: str, context: Optional[str] = None) -> str:
    9. return f"[{self.name}] query={query[:20]}, context={len(context or '')}"
    10. faq_model = StubModel("faq")
    11. standard_model = StubModel("standard")
    12. flagship_model = StubModel("flagship")
    13. class IntentClassifier:
    14. def predict(self, query: str) -> str:
    15. # Stub rules: replace with a real intent recognition model
    16. if query in ("hello", "bye") or "chat" in query:
    17. return "simple_faq"
    18. if "long document" in query:
    19. return "long_doc"
    20. if "agent" in query or "tool call" in query:
    21. return "agent"
    22. if "reason" in query or "math problem" in query:
    23. return "reasoning"
    24. return "standard"
    25. classifier = IntentClassifier()
    26. def needs_retrieval(query: str) -> bool:
    27. return "retrieve" in query or "materials" in query or "latest" in query
    28. class Retriever:
    29. def search(self, query: str, top_k: int = 5) -> list[str]:
    30. # Stub: replace with a vector database or search API
    31. return [f"doc-{i}:{query}" for i in range(top_k)]
    32. retriever = Retriever()
    33. def rerank(query: str, docs: list[str]) -> str:
    34. # Stub: replace with a reranking model
    35. return "\n".join(docs)
    36. def route_and_answer(query: str, context: Optional[str] = None) -> str:
    37. intent = classifier.predict(query)
    38. retrieval_needed = needs_retrieval(query)
    39. # simple_faq is handled by the lightweight model only when retrieval is not needed
    40. if intent == "simple_faq" and not retrieval_needed:
    41. return faq_model.generate(query)
    42. # When materials are needed, retrieve and assemble context first
    43. if retrieval_needed:
    44. docs = retriever.search(query, top_k=5)
    45. context = rerank(query, docs)
    46. # Complex reasoning, agentic, or long-document tasks go to the flagship model
    47. if intent in ("reasoning", "agent", "long_doc"):
    48. return flagship_model.generate(query, context=context)
    49. return standard_model.generate(query, context=context)
    50. if __name__ == "__main__":
    51. print(route_and_answer("hello"))
    52. print(route_and_answer("retrieve latest earnings report and analyze"))
    53. print(route_and_answer("help me reason a math problem"))

    7.3 Step 3: memory estimation for inference deployment

    Before deploying a trillion-parameter MoE model, it is recommended to estimate memory usage first and then decide the parallelism strategy.

    Item Estimation method Notes
    Model weights total parameters × bytes per precision 2.8 trillion parameters in BF16 require about 5.6 TB, or about 5.1 TiB
    KV cache 2 × layers × KV heads × head dimension × tokens × bytes Cache overhead is significant for 1M-token context
    Activations and temporary buffers reserve based on experience Usually reserve 10% to 20% of weight memory
    Expert load imbalance reserve for peak load Popular experts can amplify memory demand
    Communication buffers reserve according to parallelism strategy Multi-machine All-to-All communication requires extra bandwidth and buffers

    Unit conversion needs care: 1 TB = 10^12 bytes, and 1 TiB = 1024 GiB. With 2.8 trillion parameters at 2 bytes per parameter in BF16, the weights are about 5.6 TB, or about 5.1 TiB. Multi-machine and multi-GPU deployment is required. If lower-precision quantization is used, memory pressure decreases, but accuracy loss must be evaluated.

    7.4 Step 4: cloud architecture and operations

    In real deployment, enterprises can combine general cloud capabilities:

    • Container platform: support elastic scaling of inference services;
    • Object storage: store model weights, evaluation datasets, and log archives;
    • Message queue: decouple asynchronous tasks such as long-document processing from online services;
    • Logging service: record requests, responses, and tool-call trajectories for traceability;
    • Monitoring and alerting: track latency, throughput, error rate, and hallucination metrics, and set threshold alerts.

    In a general cloud environment, enterprises can choose model inference services, containers, storage, logging, and monitoring based on business load, and build a complete pipeline from evaluation to deployment. Specific available capabilities, specifications, and pricing should be based on the official documentation of the corresponding cloud platform.

    Data security also deserves special attention:

    • mask and desensitize request logs, response logs, and tool-call trajectories;
    • apply least-privilege access control to model access;
    • isolate permissions for evaluation data, training data, and online data;
    • retain audit records for scenarios involving personal, financial, or medical information.

    7.5 Step 5: engineering practices for hallucination governance

    Beyond the model itself, hallucination governance also requires engineering safeguards:

    • Retrieval augmentation: for factual Q&A, require retrieval before answering and ask for citations;
    • Structured output: use JSON Schema to constrain output format and reduce free-form behavior;
    • Confidence threshold: when model confidence is below a threshold, route to humans or trigger clarification;
    • Post-checks: validate critical fields such as amounts, dates, and IDs with rules, and regenerate if validation fails;
    • Human review: retain human review in high-risk scenarios.

    8. Risks and selection recommendations

    8.1 Three main risks

    1. Data contamination risk: after weights are opened, benchmark tasks may enter later training data. Enterprises should maintain private holdout sets and re-test periodically.
    2. Version drift risk: open-source models iterate quickly, and the same model name may correspond to different weights. Enterprises should fix version numbers and establish regression baselines.
    3. License compliance risk: some open-source model licenses impose additional requirements on commercial use and redistribution. Before deployment, legal and security teams should review them together.

    Before introducing an open-source trillion-parameter MoE model, enterprises are advised to establish an executable checklist:

    Check item Recommended action
    License terms Have legal counsel confirm whether commercial use, redistribution, and derivative models are allowed
    Weight hash or version number Fix the weight hash to avoid drift among models with the same name
    Private holdout set Cover real enterprise scenarios and adversarial samples
    Evaluation script Fix prompt templates, tool versions, and number of runs
    Deployment specifications Plan capacity based on memory, bandwidth, concurrency, and latency

    8.2 Selection recommendations

    • Task fit: if the scenario is mainly agentic tasks and long-document analysis, trillion-parameter MoE models deserve serious evaluation. If the scenario is mainly high-concurrency short Q&A, medium-sized models may offer better cost-effectiveness.
    • Hallucination trend: when benchmark scores are close, a reduction in hallucination rate often better reflects real usability improvements.
    • Total cost of ownership: include memory, bandwidth, operations, and evaluation labor, not only single-inference cost.
    • Reproducibility: prefer models with open weights and community reproducibility to reduce black-box risk.

    9. Conclusion

    This open-source flagship upgrade sends a clear signal: post-training is becoming the main battleground of large model competition. Keeping the original base and adding post-training compute can bring a model close to leading closed-source models on agentic tasks while reducing the hallucination rate by more than 60%.

    For technical teams, the real focus is not the 0.7-point gap, but:

    • how to build an evaluation system of their own;
    • how to bring trillion-parameter models under controllable cost;
    • how to turn hallucination governance into an engineering closed loop.

    Doing these things well will allow enterprises to retain initiative in technology selection no matter how the underlying models evolve.