Benchmarking Open-Source LLMs: A Technical Analysis of Chinese Models Closing the Gap
This article presents a systematic framework for evaluating large language models across code generation, mathematical reasoning, agentic tool use, and instruction following. It shares key findings from hands-on testing of new-generation open-source models—showing that Chinese open-source models are approaching top-tier closed-source performance—and offers practical guidance on model selection, from public benchmarks to business-data replay, gray-rollout validation, and continuous regression testing.
1. Why Systematic Model Benchmarking Matters
As large language models iterate faster than ever, judging model capability based solely on vendor-published scores or anecdotal experience no longer meets the needs of engineering decision-making. Before adopting a new model, a responsible technical team typically needs to answer three questions:
- Where are the capability boundaries? How does the model perform on code, mathematics, reasoning, and long-context tasks?
- Will its weaknesses break the business floor? For example, is the tool-call failure rate in agentic scenarios acceptable?
- Is the cost-performance ratio reasonable? At a given capability level, do inference cost and response latency remain competitive?
Building a reproducible, horizontally comparable benchmarking system has therefore become standard practice in model evaluation. Common approaches in the industry include public academic benchmarks (covering code, math, and general knowledge leaderboards), replaying real business datasets, and multi-dimensional hands-on testing organized by labs. Combining all three is the only way to avoid the mismatch of “high leaderboard scores, failed production rollouts.”
2. Designing the Evaluation: Task Dimensions and Metrics
A serious hands-on evaluation typically covers the following core dimensions:
1. Coding Capability
Code generation is the most competitive track among open-source models. Evaluation generally happens at three levels:
- Algorithmic problem solving: Given a problem statement and input/output constraints, measure the compile pass rate and test case pass rate of generated code. A sample prompt looks like this:
Implement a Python function solve(nums: List[int]) -> intthat returns the length of the longest strictly increasingsubsequence in the array, with time complexity no worsethan O(n log n).
- Engineering-level bug fixing: Provide real repository snippets containing defects and ask the model to locate and fix the issue, testing cross-file understanding and context retrieval.
- Frontend reproduction: Convert a screenshot or design mockup into a runnable page, evaluating visual fidelity and interaction completeness.
2. Reasoning and Mathematics
Mathematical reasoning reflects a model’s “deep reasoning ability” rather than surface-level imitation. When evaluating, it is advisable to disable distracting long-chain-of-thought options and measure:
- Final answer accuracy;
- Logical consistency of intermediate reasoning steps;
- Error propagation in multi-step problems (whether one early mistake causes the entire solution to collapse).
3. Agentic Behavior and Tool Calling
Agentic capability is becoming the dividing line for new-generation models. Key metrics include:
| Metric | Description |
|---|---|
| Tool selection accuracy | Whether the correct function and parameters are invoked |
| Multi-turn task completion rate | Proportion of complex instructions completed end to end |
| Error recovery | Whether the model can self-correct and retry after a failed call |
| Context management | Whether key constraints are forgotten over long sessions |
4. Instruction Following and Safety
This includes format constraints (such as “output JSON only”), precision in executing compound multi-constraint instructions, and whether refusal policies under adversarial prompts are reasonable.
3. Typical Findings from Hands-On Testing
Hands-on testing of new-generation open-source models reveals several broadly applicable trends:
First, the gap between open-source and closed-source models is narrowing fast. Some Chinese open-source models now score close to top international closed-source models on coding and reasoning benchmarks, and even hold natural advantages in Chinese-language scenarios thanks to their training corpora and comprehension. This means that in cost-sensitive scenarios requiring private deployment, open-source solutions have become a first-choice option.
Second, agentic capability is the new generational divide. Older models often perform adequately in single-turn Q&A, but once they enter the “plan—call tool—observe result—continue” loop, they are prone to parameter hallucination, infinite loops, or giving up prematurely. New-generation models are noticeably more robust in error recovery and multi-step planning, and this is the dimension where evaluations show the widest score gaps.
Third, a long context window does not equal usable long context. Many models claim support for extremely long windows, but in needle-in-a-haystack tests, recall decays with positional depth. Evaluation should examine how well the model uses information in the middle and at the end of documents, rather than relying on the nominal window size.
Fourth, output stability affects engineering usability. When the same prompt is sampled multiple times, the format consistency and semantic variance of model outputs directly determine the reliability of downstream parsing and automation pipelines—a factor that leaderboard scores often ignore.
4. From Evaluation to Deployment: Selection Recommendations
Based on the framework above, teams can approach model selection along the following path:
- Run public benchmarks first to establish a baseline, filtering out clearly underqualified candidates;
- Replay your own business data to quantify gaps on real tasks, paying special attention to the distribution of failure case types;
- Validate with a small-traffic gray rollout, monitoring production latency, error rates, and the cost of manual correction;
- Establish a continuous evaluation mechanism that automatically regresses core metrics after model version updates, preventing silent capability regressions from being introduced.
When deploying in the cloud, managed LLM services and general-purpose inference platforms can be used to quickly switch between models for A/B comparison. Combined with logging services and monitoring/alerting systems to accumulate evaluation data, this forms a closed loop of “evaluate—deploy—observe—iterate.”
5. Conclusion
The essence of model evaluation is turning “it feels good to use” into quantifiable, reproducible engineering decisions. Chinese open-source models have now approached world-class levels across multiple core dimensions, but the true basis for selection should still come from systematic, hands-on testing matched to your own business. Building your own benchmark lab is the key capability for navigating a rapidly evolving model ecosystem.
