Analyzing Post-Training Capabilities and Evaluation Methods for Large-Scale Open Source Models
The focus of large language model competition has shifted from base parameter size to post-training quality. This article analyzes the post-training capabilities and evaluation methods of a recent large-scale open source model, exploring the shift from retraining base models to enhancing post-training. It delves into the implications of Mixture of Experts (MoE) architectures, the practical value of agentic task evaluation, Retrieval-Augmented Generation (RAG) implementation, hallucination reduction, long context limits, and how to establish a comprehensive enterprise evaluation system.
In recent years, the focus of large language model (LLM) competition has gradually shifted from “how large the base parameters are” to “how high the post-training quality is.” A recently released open-source flagship model provides a representative answer: it did not redesign the base model but invested significant post-training compute on the existing foundation. Through reinforcement learning, preference optimization, tool-use training, and multimodal data alignment, it improved the model’s stability and executability in complex tasks.
According to public information, this model adopts a Mixture of Experts (MoE) architecture with a total parameter scale of approximately 2.8 trillion, but the parameters actually activated during a single inference are around 104 billion. Its context window reaches 1 million tokens, and it natively supports text and image processing. In complex agentic task evaluations, it achieved a score of approximately 65.7%, only 0.7 percentage points behind a leading international closed-source model’s 66.4%. In information retrieval and comprehensive browsing tests, it scored 92.1%. More notably, its hallucination rate dropped from 23.4% in the previous generation to 8.3%.
These figures indicate that evaluating a large model cannot rely solely on total parameters; it must also comprehensively consider inference efficiency, task completion rate, factual reliability, and actual deployment costs.
1. From “Changing the Base” to “Enhancing Post-Training”
The capabilities of a large model are typically determined by three stages: pre-training, supervised fine-tuning (SFT), and post-training.
Pre-training is mainly responsible for learning language patterns, knowledge structures, and multimodal representation capabilities. It determines the model’s base ceiling, but training costs are extremely high, often requiring massive data and compute resources. SFT teaches the model to output according to instructions through high-quality Q&A, code, reasoning processes, and task examples. Post-training further introduces preference feedback, automated verification, tool environment interaction, and task result rewards, shifting the model from “being able to generate an answer” to “being able to complete a goal.”
Increasing post-training compute on the same base model offers several distinct advantages:
First, it reduces the uncertainty brought by architectural changes. Redesigning the model structure requires re-verifying training stability, inference compatibility, and ecosystem adaptation. Post-training based on a mature base allows resources to be concentrated faster on capability enhancement.
Second, post-training is closer to real-world usage scenarios. Pre-training excels at learning “what exists in the world,” but not necessarily at handling “what the user wants to accomplish.” For example, when a user asks the model to analyze logs, modify configurations, or execute multi-step operations, the model must not only understand the text but also plan steps, call tools, judge results, and correct errors in time. These abilities rely heavily on post-training.
Third, post-training makes it easier to target specific deficiencies. If the previous generation model was prone to fabrication in factual Q&A or interruptions in long-process tasks, targeted improvements can be made through adversarial samples, result verification, and failure case replay, without completely redoing pre-training.
2. Understanding Trillion-Level Parameters and the MoE Architecture
Total parameters and activated parameters per inference are not the same concept.
Traditional dense models typically use all parameters in a single inference. Mixture of Experts (MoE) models divide the network into multiple expert modules, where a routing mechanism selects a few experts to participate in the computation based on the input. This allows the model to have a very large total capacity while controlling the computation volume of a single inference.
A simplified process can be represented as follows:
Input Token↓Shared Representation Layer↓Routing Module Selects Experts↓Parallel Computation by Experts↓Result Fusion↓Output Next Token
The advantages of this architecture include:
- Larger total capacity to accommodate more domain knowledge and task patterns;
- Single activation scale lower than total parameters, making inference costs easier to control;
- Different experts can form a certain degree of capability division;
- Improved resource utilization across tasks like code, math, long text, and visual understanding.
However, the MoE architecture also brings new engineering challenges. Routing imbalance may cause some experts to be overloaded, affecting throughput; communication between experts increases the complexity of distributed inference; and in small batch request scenarios, hardware utilization may be less than ideal. Therefore, a larger parameter scale does not mean easier deployment. Actual costs also depend on quantization methods, parallel strategies, VRAM capacity, network bandwidth, and request distribution.
3. Why Agentic Task Scores Are a Valuable Reference
Agentic task evaluations typically require the model to complete a series of continuous operations, such as reading project files, analyzing error messages, modifying code, running tests, and adjusting based on results. Compared to single-turn Q&A, these tasks are closer to developer and enterprise automation scenarios.
The difficulties of agentic tasks are mainly reflected in four aspects:
1. Task Planning
The model needs to break down natural language goals into multiple executable steps. For example, confirming the directory structure first, then locating the configuration file, modifying the content, and finally running the verification command. Any mistake in the order can cause subsequent operations to fail.
2. Environment Understanding
The model must dynamically adjust its strategy based on the current working directory, file contents, and execution results, rather than mechanically applying fixed templates from training data.
3. Error Recovery
Real environments often experience missing dependencies, insufficient permissions, test failures, or inconsistent input formats. A high-quality model should be able to identify the cause of failure and choose an alternative solution instead of repeating the same operation.
4. Result Verification
Completing a modification does not equal task success. The model must also confirm that the result actually meets requirements through tests, static checks, output comparison, or other verification methods.
Therefore, an agentic task score close to leading models indicates that the model has strong capabilities in planning, tool use, and closed-loop execution. However, a single score still cannot represent all engineering scenarios. Evaluation environments usually have clear goals and standard answers, whereas actual enterprise projects may involve permission management, legacy code, implicit business rules, and incomplete documentation, making them more difficult.
4. High Information Retrieval Scores Do Not Mean Infallibility
Information retrieval tasks mainly examine whether the model can locate information from multiple sources, cross-verify facts, and form structured conclusions. A score of approximately 92.1% indicates that the model performs strongly in searching, reading, summarizing, and synthesizing.
However, implementing these capabilities still requires establishing a reliable Retrieval-Augmented Generation (RAG) process:
User Question↓Question Decomposition & Keyword Generation↓Recall Candidate Materials↓Relevance Ranking↓Content Extraction & Evidence Retention↓Multi-Source Cross-Verification↓Generate Answer with Evidence
Among these, evidence retention is particularly important. The model should not only provide a conclusion but also explain which materials the conclusion comes from, whether there are conflicts between materials, and which content is inferred rather than direct fact.
Enterprises can adopt the following measures when using RAG:
- Perform version management on documents to prevent old materials from overriding new specifications;
- Retain source, update time, and permission tags for paragraphs;
- Set up manual review for high-risk conclusions;
- Conduct secondary verification on numbers, dates, policies, and configuration items;
- Record the retrieval process for easy traceability.
5. The Practical Value of Hallucination Rate Reduction Over Benchmark Scores
If the previous generation model’s hallucination rate was 23.4% and the new version dropped to 8.3%, this means the proportion of error generation has significantly decreased. For enterprise applications, this change is often more valuable than a few percentage points increase on a comprehensive leaderboard.
The reason is that errors in enterprise scenarios are not just “the answer isn’t good enough”; they can cause actual damage. For example:
- Automatically generating incorrect configurations, causing service startup failures;
- Fabricating non-existent interfaces, wasting developers’ troubleshooting time;
- Incorrectly summarizing contracts or regulations, triggering compliance risks;
- Executing unsafe operations in automated workflows;
- Packaging uncertain information as definite conclusions, misleading decision-making.
Reducing hallucinations cannot rely solely on scaling up the model; it requires systematic design. Effective methods include:
- Uncertainty Expression: Clearly state when evidence is insufficient rather than forcing an answer.
- Retrieval Constraints: Require the model to prioritize generating answers based on authorized materials.
- Structured Output: Reduce the room for free play through fixed fields.
- Programmatic Verification: Automatically check code, calculation results, and configuration files.
- Manual Approval: Set review nodes for high-risk tasks in finance, healthcare, and production operations.
- Log Auditing: Save inputs, retrieved content, model outputs, and final actions for accountability and improvement.
6. The Practical Significance and Limitations of Million-Token Context
A million-token context can accommodate large-scale codebases, design documents, meeting minutes, or historical tickets, theoretically reducing information loss caused by frequent context switching.
However, long context does not mean the model can utilize all content equally efficiently. Common issues include:
- Key information buried in a large amount of irrelevant text;
- Conflicts between documents, making it difficult for the model to judge priority;
- Increased retrieval and reasoning costs due to excessive context length;
- The model may ignore important content located in the middle;
- Sensitive information inadvertently brought into the generation process.
Therefore, engineering should not simply stuff all materials into the model at once, but adopt a layered strategy:
- First establish document indices and directories;
- Recall relevant fragments based on the question;
- Summarize and compress core materials;
- Precisely concatenate final cited content;
- Process ultra-long tasks in stages and save intermediate states.
Long context is suitable for cross-file analysis, long-term project memory, and complex material comparison, but it still requires good data governance and context orchestration.
7. Evaluation System to Focus on for Enterprise Deployment
Before using a large model, it is recommended to establish an evaluation set that fits your own business rather than relying entirely on public leaderboards. The evaluation set should cover at least the following dimensions:
- Task completion rate;
- Output accuracy rate;
- Proportion of hallucinations and unsupported conclusions;
- Tool calling success rate;
- Average response latency;
- Resource consumption per request;
- Long task interruption rate;
- Sensitive information leakage risk;
- Manual modification cost.
Evaluation can be divided into offline and online parts. Offline evaluation is used to compare different model versions, while online evaluation focuses on real user feedback, failure cases, and business metric changes.
For development scenarios, a distinction should be made between “code generation accuracy” and “actual code usability.” The former only looks at whether the text meets expectations, while the latter must pass compilation, testing, static checks, and security scans. Only through complete verification can it be determined whether the model truly improves R&D efficiency.
Conclusion
The significance of this trillion-level open-source model lies not just in its massive parameter scale or single scores approaching leading systems, but in demonstrating an important direction for large model development: based on a mature foundation, continuously improving usability in real tasks through high-intensity post-training, tool interaction, multimodal alignment, and factual optimization.
For developers, the focus should be on whether the model can stably complete multi-step tasks, correct errors based on feedback, and provide verifiable results. For enterprises, model capabilities should be combined with permission control, data governance, log auditing, and manual review.
Ultimately, a model’s value does not depend on a single number in its marketing, but on whether it can work stably within clear boundaries and continuously generate actual returns at a controllable cost.
