A New Evaluation Cycle for Chinese LLMs: Open Weights and Agent Capabilities Advance in Parallel

1. Evaluation Systems Are Moving from Leaderboard Scores to Production Readiness

For the past two years, LLM evaluation has faced a familiar problem: high leaderboard scores alongside poor real-world performance. The reason is straightforward. Public test sets are easily covered by training data, and models can be optimized toward specific tasks to produce inflated scores that degrade sharply in long-chain, multi-turn, cross-modal production scenarios.

A recent update to a major evaluation platform’s intelligence index now covers nine core dimensions, including logical reasoning, code generation, long-text processing, and multimodal understanding. Notably, more than 60% of the test set consists of proprietary, private datasets that are not publicly available. This mechanism substantially reduces the room for leaderboard gaming. The design principle deserves close attention from developers: private evaluation sets mean scores correlate much more strongly with real production performance.

For technical teams, the practical implication is clear. When selecting models internally, public rankings should not be the only reference point. Teams should build private evaluation sets that match their own business distribution. One practical engineering approach can be structured like this:

# Framework for building a business-specific private evaluation set
eval_suite = {
    "reasoning": load_internal_cases("logic_cases.jsonl"),      # Logical reasoning
    "codegen": load_internal_cases("repo_tasks.jsonl"),          # Code generation
    "long_context": load_internal_cases("long_docs.jsonl"),      # Long-text processing
    "agent_task": load_internal_cases("workflow_sessions.jsonl") # Agent tasks
}
def evaluate(model, suite):
    scores = {}
    for dim, cases in suite.items():
        scores[dim] = run_batch(model, cases, metrics=dim_metric[dim])
    return scores

The value of a private evaluation set is that it reflects your environment, not an average environment.

2. Open Weights: From API Calls to Deployment Control

A landmark change in this evaluation cycle is the appearance of the first non-US model in the global top five. It achieved a composite score of 55, with standout performance in long-text processing and multi-turn dialogue consistency. More significant from an engineering perspective, the model has recently released its full weights. It has a total parameter count of 2.8 trillion, 104 billion active parameters, and supports a 1 million token context window.

For developers, open weights matter far more than ranking position:

• Deployment autonomy: Models can run on your own GPU cluster or dedicated cloud environment, keeping data within your infrastructure and meeting compliance requirements in finance, government, healthcare, and other regulated industries.
• Predictable cost structure: After the initial compute investment, marginal inference cost is significantly lower than token-based closed APIs. For high-frequency workloads, total cost of ownership advantages are substantial.
• Customizability: Open weights enable domain fine-tuning, distillation, and quantization, allowing teams to turn model capabilities into lasting technical assets.

The 1 million token context window is equally important. Traditional approaches to long documents often rely on retrieval-augmented generation with chunking and stitching, which can lose information during segmentation and reassembly. Ultra-long context makes it possible to input an entire report at once or analyze a full code repository directly, reducing dependence on complex retrieval pipelines. That said, engineering teams should keep several points in mind:

1. Memory consumption for ultra-long context grows with sequence length, requiring sound batching and KV cache management strategies.
2. Being able to fit content into the context window does not mean the model can use it effectively. Recall quality for key information in very long inputs still needs to be verified through evaluation.
3. For critical tasks, keep retrieval augmentation as a supplementary mechanism, creating a defense-in-depth approach that combines long context with search.

3. Agent Capabilities: From Dialogue to Real Task Execution

If the intelligence index measures general model capability, agent-oriented benchmarks measure the ability to get things done. This cycle includes a leaderboard based on more than 17,500 real user sessions in long-duration agent interactions, with a focus on real agent task performance. One Chinese model ranked 13th globally and first among open-source models worldwide. It achieved first place among open-source models in three core capabilities: workplace automation, autonomous code development, and multi-task orchestration, with a net improvement rate of 5.1% over the previous generation.

These three capabilities map directly to the scenarios enterprises care most about:

Workplace automation: Delegating repetitive report generation, data processing, and email handling to agents. The key technical requirement is reliable function calling and error recovery.

Autonomous code development: Agents that can understand requirements, locate relevant code, implement changes, and run tests. This requires long-context understanding, code generation, and a closed feedback loop with execution results.

Multi-task orchestration: Breaking a complex goal into subtasks and coordinating multiple tools and data sources. This fundamentally combines planning capability with state management.

A simplified agent orchestration framework can be abstracted as:

class AgentOrchestrator:
    def __init__(self, planner, tools, memory):
        self.planner = planner      # Task planning
        self.tools = tools          # Available tool set
        self.memory = memory        # Long-duration state memory
    def run(self, goal):
        plan = self.planner.decompose(goal)
        for step in plan:
            tool = self.tools.select(step)
            result = tool.execute(step, context=self.memory.recall())
            self.memory.store(step, result)
            if not self.verify(step, result):
                plan = self.planner.replan(goal, self.memory)
        return self.memory.summarize()

A net improvement rate of 5.1% may seem modest, but in agent tasks, every incremental gain directly translates into higher task success rates. Industry average iteration improvements often fall in the 1% to 2% range. A 5.1% gain suggests the generational gap is widening.

4. Practical Recommendations for Enterprises and Developers

Based on these trends, here are several actionable recommendations:

First, build a two-track evaluation mechanism. Use public leaderboards for rapid screening of candidate models, and private evaluation sets for final decisions. Evaluation dimensions should cover the core workflow of your business rather than copying generic categories.

Second, prioritize open-weight options. For scenarios with high call volume, sensitive data, or deep customization requirements, open-weight models offer structural advantages in cost and compliance. During evaluation, focus on license terms, inference framework compatibility, and accuracy loss after quantization.

Third, work backward from agent capabilities when selecting models. If your business goal is process automation, tool-calling accuracy, multi-turn state retention, and error recovery matter more than general QA scores. Build end-to-end agent tasks and test candidates directly.

Fourth, pay attention to the engineering requirements of long context. Ultra-long context windows need supporting memory optimization, caching strategies, and retrieval fallback mechanisms. Without these, teams can end up with advertised parameter support that is not practically runnable.

Fifth, act within the deployment cost window. As leading models open their weights, unit inference costs continue to decline, and the price-performance ratio of local technology stacks is improving rapidly. For latency-sensitive and data-compliance-sensitive downstream scenarios such as autonomous driving, industrial manufacturing, and enterprise services, the current period is a good time for evaluation and pilot projects.

5. Conclusion

The Chinese LLM industry is moving from catching up with global benchmarks toward leading in specific dimensions. The shift to private evaluation sets, the gradual release of open weights, and rapid iteration in agent capabilities all point to one conclusion: the accessibility and controllability of the underlying technology stack are improving simultaneously. For developers, the important task is not remembering a particular leaderboard ranking, but building an engineering methodology that enables continuous tracking, rapid validation, and on-demand integration. Models will iterate and rankings will change, but solid evaluation and deployment capabilities remain the foundation of long-term competitiveness.