Trillion-Parameter MoE Flagships: Post-Training Gains, Deployment Costs, and Production Practices
This article examines the rise of open-source trillion-parameter Mixture-of-Experts (MoE) models, using a 2.8-trillion-parameter flagship with 104 billion active parameters and a 1-million-token context as a reference case. It explains how post-training can improve agentic behavior without changing the base model, breaks down VRAM, KV cache, and throughput considerations for deployment, and outlines practical approaches for long-context use, native multimodality, hallucination control, and upgrade evaluation.
1. A trillion-parameter MoE flagship is now available
Recently, an open-source trillion-parameter Mixture-of-Experts (MoE) flagship model was released. The weights became available for download within days, and interest continued to rise. The model’s parameter count and benchmark results are already enough to generate discussion. For developers, however, the more important signal is the trend: the gap between open-source and closed-source top models is narrowing, often measured in fractions of a point. At the same time, model selection and deployment are shifting away from “how large is the parameter count?” toward “how well can the model be used in production?”
Key specifications
- Parameter scale: 2.8 trillion total parameters, built on a Mixture-of-Experts (MoE) architecture. About 104 billion parameters are activated per token during inference.
- Context length: Supports up to 1 million tokens.
- Modality: Native text and image multimodal. The model was trained on joint text and image data from the start; images do not require a separate visual encoder as an external “translator.”
- Capability source: It is a post-trained version of an open model from the same generation. The base model was not replaced. Instead, a substantial amount of additional post-training compute was applied to the same base.
- Benchmark performance: It scored 65.7% on a terminal command-line agent benchmark. By public data, a closed-source flagship model scored 66.4%, only 0.7 points higher, while the relevant closed-source flagship series scored 58.2%. On an autonomous browser retrieval benchmark, it scored 92.1%, ranking first in the model provider’s published table.
- Hallucination rate: Dropped from 23.4% in the previous generation to 8.3%, a relative reduction of nearly two-thirds.
What these numbers mean for developers
Taken together, these results support three conclusions that are directly relevant to developers.
- Capability is shifting toward post-training. As pre-training scale approaches the frontier, teams are repeatedly adding post-training compute on the same base model, using finer data and training strategies to extract more capability. The base model sets the knowledge ceiling; post-training determines whether that knowledge can be used reliably.
- Architecture competition is focused on compute efficiency. Trillion-scale total parameters provide capacity, while hundred-billion-scale active parameters control the cost of a single forward pass. In addition, “long context plus native multimodality” is becoming standard for flagship models, creating new memory-budget requirements.
- Evaluation metrics are changing. Terminal and browser agent benchmarks measure multi-step task completion, while hallucination rate measures answer reliability. These metrics are closer to real production scenarios than traditional knowledge-based QA scores.
The remainder of this article follows the path “capability source → architecture and cost → benchmark interpretation → engineering deployment,” with estimation scripts and example code where useful.
2. Capability analysis: how post-training can create a jump without changing the base
What “post-training intensification” means
Training a large model can be roughly divided into two stages:
- Pre-training: Learning language and world knowledge from massive corpora. This determines what the model knows.
- Post-training: Shaping behavior through instruction tuning, preference alignment, reinforcement learning, and similar methods. This determines whether the model can perform tasks and whether its behavior is reliable.
“Post-training intensification” means investing more high-quality data and longer post-training time while keeping the base model unchanged. Common approaches include:
- Expanding instruction and reasoning data: Covering code modification, terminal operation, multi-turn tool calling, and other long-horizon tasks.
- Preference optimization: Making outputs closer to human and rule-based preferences in correctness, concision, and safety.
- Reinforcement learning from verifiable rewards: Using objective signals such as unit tests, compilation results, and task completion as rewards, allowing the model to learn strategies from large numbers of attempts.
Why can post-training alone produce a major score improvement? This does not violate the principle that capabilities are constrained by the base model. Most agentic tasks do not have a high bar for new knowledge. The difficulty lies in planning, tool selection, and error recovery, which are essentially “behavior and strategy” problems. Post-training acts precisely on this layer, so completion rates can improve significantly without replacing the base model.
Practical implications for developers
| Dimension | Change from replacing the base model | Change from post-training intensification | Impact on developers |
|---|---|---|---|
| Knowledge coverage | May change significantly | Mostly unchanged | New-knowledge questions still require retrieval augmentation |
| Instruction following and format stability | Uncertain | Usually improved | Structured output and tool calls become more reliable |
| Multi-step tasks and error recovery | Uncertain | Clearly improved | Better suited for automated workflows |
| Upgrade compatibility | Often accompanied by tokenizer or interface changes | Mainly behavioral changes with the same base | Regression testing is still needed, but migration cost is relatively manageable |
Therefore, when facing a version upgrade that “only changes post-training and does not change the base,” the right attitude is neither to blindly reuse old conclusions nor to rebuild everything from scratch. Instead, teams should:
- Compare versions using their own evaluation set.
- Focus regression testing on tool-call formats, long-context retrieval, and refusal behavior.
- Verify whether structured output remains stable before and after the upgrade.
3. Architecture analysis: the engineering ledger for trillion parameters, hundred-billion active parameters, and million-token context
Total parameters vs. active parameters: “fits” vs. “runs”
MoE splits the feed-forward network into multiple “experts.” For each token, a router activates only a small number of experts. This turns the dense model’s “large and complete” computation into “on-demand invocation.” It creates two independent cost dimensions:
- Total parameters determine memory usage and deployment threshold. They answer: “Does it fit?”
- Active parameters determine the computation of a single forward pass. They answer: “Does it run?”
For this round, the ratio of 2.8 trillion to 104 billion is about 27:1. This means memory pressure and compute pressure are decoupled: memory requirements are high, but the computation per inference is similar to that of a dense model with about 104 billion parameters.
It is worth noting that dense models also need to read all parameters. Weight-reading volume is proportional to parameter count. Therefore, the real advantage of MoE is most visible in high-concurrency throughput scenarios. For single-request, low-concurrency cases, the difference depends on memory bandwidth, parallel sharding strategy, and communication overhead.
Memory and throughput budget: four steps to estimate your deployment threshold
Step 1: Estimate weight memory. It can be calculated as follows:
Weight memory (GB) ≈ total parameter count × bytes per parameter ÷ 1024³
- BF16/FP16: 2 bytes per parameter; 2.8 trillion parameters require about 5.2 TB of memory.
- FP8: 1 byte per parameter; about 2.6 TB of memory.
Both far exceed the memory of a common 80 GB accelerator card. Deployment therefore requires combinations of tensor parallelism, pipeline parallelism, expert parallelism, and other multidimensional parallelism strategies.
Step 2: Estimate KV cache. The number of bytes per token in the KV cache can be calculated as follows. Use the settings of your implementation.
KV bytes/token = 2 × layers × KV heads × head dim × KV element bytes(In GQA architectures, the number of KV heads is usually much smaller than the number of attention heads.)
For example, with 80 layers, 8 KV heads, head dimension 128, and FP16:
- Each token requires about 320 KB.
- 1 million tokens require about 320 GB, almost the same scale as one copy of FP8 weights.
This leads to an important conclusion: in long-context scenarios, the real bottleneck is often not weights, but KV cache and memory bandwidth. Prefix caching, KV quantization, paged KV management, and similar general optimizations become essential.
Step 3: Estimate throughput. Approximate forward computation as 2 × active parameters. With 104 billion active parameters, that is about 2.08 × 10¹¹ forward operations per token, excluding attention overhead.
A rough formula for estimating sustainable throughput per card is:
Tokens per second ≈ memory bandwidth ÷ bytes that must be read per token
If every generated token required a full scan of the weights:
- BF16 would require reading about 5.6 TB.
- FP8 would require reading about 2.8 TB.
With about 8 TB/s memory bandwidth, the theoretical upper bound would be only about 1.4 to 2.9 tokens per second.
This is exactly the value of batching inference frameworks that load weights once and serve many requests: they spread the cost of weight loading across the entire batch, making actual throughput much higher than this theoretical single-request limit. It also shows that MoE models can be more cost-effective in high-concurrency scenarios. Teams should run load tests against their own business concurrency curve rather than relying only on theoretical values.
Step 4: Turn the calculation into a reusable script. The first two steps can be fixed into a budgeting script so that every model-selection exercise can start with a quick estimate:
# Example model deployment memory budget (adjust to actual settings)def estimate_vram(total_params, # Total number of parametersactive_params, # Number of active parameterslayers, # Number of layerskv_heads, # Number of KV headshead_dim, # Head dimensionmax_seq_len, # Context lengthweight_bytes, # Bytes per parameter: BF16=2, FP8=1kv_bytes=2, # Bytes per KV elementbatch_size=1,):gb = 1024 ** 3weight = total_params * weight_bytes / gbkv = (2 * layers * kv_heads * head_dim* kv_bytes * max_seq_len * batch_size / gb)flops_per_token = 2 * active_paramsreturn {"weight_vram_gb": round(weight, 1),"kv_cache_gb": round(kv, 1),"total_vram_gb": round(weight + kv, 1),"forward_tflops_per_token": round(flops_per_token / 1e12, 1),}print(estimate_vram(total_params=2.8e12, active_params=104e9,layers=80, kv_heads=8, head_dim=128,max_seq_len=1024, weight_bytes=1,))print(estimate_vram(total_params=2.8e12, active_params=104e9,layers=80, kv_heads=8, head_dim=128,max_seq_len=1_000_000, weight_bytes=1,))
Example output:
{'weight_vram_gb': 2607.1, 'kv_cache_gb': 0.3, 'total_vram_gb': 2607.4, 'forward_tflops_per_token': 0.2}{'weight_vram_gb': 2607.1, 'kv_cache_gb': 305.2, 'total_vram_gb': 2912.3, 'forward_tflops_per_token': 0.2}
The weight and KV numbers in the script are estimates, used only to judge orders of magnitude and plan resources. Actual memory requirements are also affected by framework implementation, parallel sharding, and memory fragmentation. Reserve about 20% headroom and verify with actual loading tests.
Million-token context: capability vs. effective use
“Supporting 1 million tokens” and “making good use of 1 million tokens” are two different things.
- Attention complexity: Standard self-attention computation grows quadratically with sequence length. Time to first token (TTFT) for million-token inputs can increase significantly. This is usually mitigated with chunked prefilling, I/O-aware attention implementations, and sparse attention variants.
- Information “lost in the middle”: Models often use information at the beginning and end of a long input better than information in the middle, and this effect becomes more pronounced as input length grows. Important information should be placed early or repeated for emphasis.
- Cost grows with length: KV cache and request latency increase together. Reserve ultra-long context only for tasks that truly need it; for other cases, use “retrieval plus summarization” to compress the input.
- Reasonable routing:
- For single-turn tasks with relatively stable knowledge, use retrieval augmentation.
- For codebase understanding or long-document comparison that requires global analysis across many files, use long context.
- For multi-turn dialogue, prioritize server-side session-state continuity and avoid resending the entire history every turn.
Native multimodality: advantages, limits, and task division
Native multimodality means text and images are jointly modeled during pre-training, making cross-modal relationships more natural. Compared with “external visual encoder” approaches, it is usually more stable on tasks such as chart reading, layout understanding, and image-text instruction following. However, it also has clear boundaries.
- Computation overhead: High-resolution images can produce many visual tokens after visual encoding, consuming context budget.
- Precision limits: Small text, complex tables, and dense charts remain difficult. In some cases, zooming, cropping, or using a specialized text recognition service is necessary.
- Task division:
- Native multimodal models: suitable for layout analysis, cross-image-text reasoning, and screenshot understanding.
- Specialized text recognition services: suitable for high-accuracy text extraction.
- Structured extraction: whether the source is text or image, use output schema constraints so downstream systems can consume the result directly.
4. Benchmark interpretation: the 0.7-point gap, top hallucination rate, and trust
Agent benchmarks test “doing” rather than “answering”
For example:
- Terminal command-line benchmarks require the model to complete tasks such as compile fixes, script writing, and environment configuration in a realistic command environment. Scoring is based on whether the task is ultimately completed, testing multi-step planning and tool use.
- Browser retrieval benchmarks require the model to search autonomously in a browser and find answers that are not directly retrievable. They test search strategy, page parsing, and multi-step verification.
On these benchmarks, the open-source model scored 65.7%, while a top closed-source model scored 66.4%. The difference is only 0.7 points, while the relevant closed-source flagship series scored 58.2%. The gap from 58.2% to 66.4% is measured in points; the gap from 66.4% to 65.7% is measured in fractions of a point. Compared with the gap across generations of closed-source flagships, 0.7 points looks more like a minor fluctuation within the same tier than a clear generational capability gap.
However, model selection should not rely on this single number alone.
- Small overall gap: Run repeated evaluations on your own task set and use paired comparisons to determine whether the difference is stable.
- Large gaps in subcategories: For example, tool-call format stability, multi-turn error recovery, and long-context recall may matter more than the total score.
- Evaluation bias: Self-reported leaderboards may differ in task setup, prompt templates, and tool environments. Retest on your real business distribution before launch.
Hallucination rate: why 23.4% to 8.3% matters
Hallucination refers to content that appears reasonable but is inconsistent with facts or lacks support. In developer scenarios, it often appears as:
- Fabricated APIs, parameters, or configuration items.
- Commands that cannot run.
- Citations to nonexistent materials or sources.
- Failure to recognize the model’s own knowledge limits, treating uncertainty as certainty.
A drop from 23.4% to 8.3% means that roughly one in eight answers may still be wrong, which is already close to the upper limit of acceptability for some business scenarios. For high-accuracy QA, customer service, and operations assistance, hallucination rate may be more decisive than a few extra points on a benchmark.
Common post-training methods for reducing hallucination include:
- Reinforcing the behavior of refusing or explaining uncertainty when the model is unsure.
- Correcting errors with objective feedback in verifiable environments.
- Reducing unsupported assertions through preference alignment.
However, the returns from pushing hallucination below 8.3% are diminishing. Further reduction usually depends on engineering methods:
- Use retrieval augmentation to provide factual grounding.
- Require answers to include citations or sources.
- Validate tool-call parameters with schemas.
Three trust checks for benchmark results
| Check item | What to confirm | Risk if not confirmed |
|---|---|---|
| Evaluation protocol | Whether prompt templates, tool permissions, retry counts, and environment versions are consistent | Good-looking numbers may not match production behavior |
| Result source | Whether results come from the model provider, third-party reproduction, or a public leaderboard | Self-reported results may be selectively presented |
| Capability discrimination | Whether the score reflects genuine ability or prior exposure to the tasks | Performance may degrade significantly on real tasks after launch |
5. Engineering deployment: six steps from selection to launch
Step 1: Estimate before selecting
Before selecting a model, use the script in the previous section to estimate memory and throughput, and convert the estimates into required cards and concurrency based on business volume. At the same time, clarify three boundaries:
- Whether ultra-long context is needed.
- Whether image input is needed.
- What latency is acceptable.
Only when the budget is clear can you make informed decisions about parallel strategy, context strategy, and cache strategy.
Step 2: Use long context where it matters
The right way to use million-token context is to layer it:
# Example of tiered long-context usagedef answer_with_long_context(question,corpus, # Long documents or codebase already loaded in contextmodel,rerank_top_k=5,retrieve_top_n=50,chunk_size=1000, # Approximate chunk lengthoverlap=100,max_context_tokens=200_000, # Actual token budget, not necessarily the maximum):# 1. Coarse retrieval: split into chunks and retrieve candidateschunks = []for i in range(0, len(corpus), chunk_size - overlap):chunks.append(corpus[i:i + chunk_size])candidates = search(question, chunks, top_n=retrieve_top_n)# 2. Reranking: keep only the most relevant snippets in the promptselected = rerank(question, candidates, top_k=rerank_top_k)# 3. Generation: require citations to each source for traceabilityprompt = f"Question: {question}\nReference material:\n" + "\n\n".join(f"[{i + 1}] {doc}" for i, doc in enumerate(selected))prompt += "\nAnswer with numbered citations. If the material is insufficient, state 'insufficient information'."return model.chat(prompt, max_context_tokens=max_context_tokens)
The key point is that even if the model supports million-token context, not every request should be filled to the limit. Combining retrieval compression, citation constraints, and refusal fallback is the only way to keep cost controllable while maintaining trust.
Step 3: Build a hallucination-control loop with retrieval augmentation and validation
Hallucination control cannot rely only on the model being “more honest.” It needs a closed-loop mechanism.
- Retrieval augmentation: Split knowledge-base documents and build indexes. Retrieve before answering, require the model to answer only based on retrieved results, and attach source numbers.
- Schema validation: Check whether tool-call parameters match the schema, whether retrieval results are empty, and whether cited numbers actually exist.
- Trust validation: For tasks that depend on external information, ask the model to answer independently with different wording and compare conclusions. If the answers agree, confidence increases. If they conflict, escalate to human review or perform additional retrieval.
- Fallback and monitoring: Make “clearly state when confirmation is impossible” the default behavior. Continuously collect refusal rate, citation hit rate, and human correction rate in production to build a quantifiable quality dashboard.
The core idea is to turn hallucination from a “probability problem” into a “verifiable problem.” Whenever rules, retrieval, or cross-validation can determine correctness, do not rely on model self-assessment.
Step 4: Use multimodality and agent capabilities, but manage boundaries
Native multimodality and strong tool-calling capabilities make models suitable for cross-modal tasks such as document review, report verification, and screenshot-based troubleshooting. When deploying, pay attention to three points:
- Context budget: High-definition images may consume many visual tokens. Limit the number of images per request and crop or compress them when necessary.
- Format constraints: Multimodal output should also be constrained by schemas to avoid free-form text that breaks downstream parsing.
- Operation boundaries: For automation tasks involving terminals and browsers, set permission whitelists, operation logs, and human confirmation steps.
Step 5: Protect each version upgrade with your own evaluation set
Public leaderboards are only useful for initial screening. The real guardrail is your own evaluation set. It is recommended to sample cases from production logs and cover high-frequency, low-frequency, long-tail, and boundary cases. In paired comparisons, run the same cases on both the candidate model and the current model, and record both score differences and per-case results:
import randomimport statisticsdef paired_compare(candidate_fn, baseline_fn, cases, n=200, seed=42):sample = random.Random(seed).sample(cases, min(n, len(cases)))diffs = []for case in sample:a = 1 if candidate_fn(case) else 0b = 1 if baseline_fn(case) else 0diffs.append(a - b)# Paired bootstrap: resample the differences with replacement to estimate# a 95% confidence interval for the mean gain.rng = random.Random(seed + 1)boots = [statistics.mean(rng.choices(diffs, k=len(diffs))) for _ in range(1000)]boots.sort()return {"sample_size": len(sample),"mean_gain": statistics.mean(diffs),"95%_ci": (boots[24], boots[974]),}result = paired_compare(candidate_fn=lambda c: judge(c["answer_new"]) == "correct",baseline_fn=lambda c: judge(c["answer_base"]) == "correct",cases=[{"id": i} for i in range(1, 101)], # Replace with real evaluation cases)print(result)
Interpretation principles:
- If the confidence interval crosses zero, do not conclude that the candidate version is better.
- Even if the total score is flat, significant regression on individual cases may still indicate risk.
- Running the same evaluation on every upgrade is the only way to determine whether a change is “progress” or “regression.”
Step 6: Turn selection, deployment, and evaluation into repeatable capabilities
To turn these steps into organizational capability, three things are needed:
- Budget tooling: Include memory and throughput estimation scripts in the model-selection process.
- Evaluation as a routine: Keep updating the evaluation set as the business evolves, and include it in the release process.
- Monitoring loops: Connect production quality with evaluation results so that regressions can trigger fast rollback.
For infrastructure such as compute resources, storage, and logs, cloud platforms such as Baidu AI Cloud and other providers may offer general categories such as GPU compute instances, container platforms, object storage, logging services, and monitoring alerts. Enterprises can choose based on their own needs, but specific specifications, pricing, and availability should be confirmed against official documentation. Model support should also be verified through official release information rather than rumors.
6. Conclusion: the engineering meaning behind benchmark scores
The release of a trillion-parameter open-source MoE flagship is not just a benchmark refresh. It reflects three trends:
- The source of capability has changed: from “stacking pre-training parameters” to “base model plus post-training intensification.”
- The cost structure has changed: total parameters determine memory, active parameters determine compute, and long context determines KV cache.
- Evaluation metrics have changed: from knowledge QA to agent task completion and hallucination control.
For developers, the three most valuable takeaways are:
- Estimate first: Use memory and throughput estimates to decide deployment shape.
- Layer appropriately: Use long context, retrieval augmentation, and multimodality where each fits best.
- Build guardrails: Use retrieval, validation, and self-built evaluations to keep hallucinations out of production.
When the gap between open-source and closed-source flagships narrows to 0.7 points, product experience is determined not by whose score is higher, but by who can turn model capabilities into business value in a stable, controllable, and cost-effective way.
