How to Evaluate and Select Chinese LLMs: A Practical Framework
Selecting a Chinese LLM is less about finding the highest-scoring model than about matching model capabilities to your actual workload. This article presents a reusable evaluation framework covering leaderboard screening, custom business-aligned test sets, engineering readiness, total cost per task, and a practical decision checklist.
1. Why “Which Model Should We Choose?” Is the Wrong Question
The release cadence of Chinese LLMs has accelerated noticeably in recent years, with new flagship versions appearing almost every month. Open any technical community and you will see questions like: “Should we use Model A or Model B?” But when you ask, “What is your use case?”, the person asking often cannot give a clear answer.
That is where the problem starts. LLM selection is not fundamentally about “finding the strongest model”; it is about “finding the best fit.” Differences in training data mixtures, post-training strategies, and alignment approaches cause models to perform unevenly across tasks. Some models are strong at code but average at long-document processing. Others write fluently but are less stable at tool calling. Discussing model quality without a business scenario is like choosing a database without knowing the workload: the conclusion has little meaning.
This article therefore does not aim to conclude that “a certain model is stronger.” Instead, it provides a general evaluation framework. Whenever a new model from any vendor appears on the market, this method can be reused.
2. Step 1: Read Leaderboards, but Do Not Worship Them
Today’s mainstream model capability leaderboards fall roughly into three categories, and their signal-to-noise ratios vary widely.
1. Human blind-test leaderboards
Real users anonymously compare answers from two models and vote, and the votes are converted into Elo scores. These leaderboards reflect “human subjective preference.” Their advantage is that they are close to real user experience. Their disadvantage is that scores can be inflated by particular styles—such as longer answers, more emoji, or a more agreeable tone—and are not fully equivalent to task accuracy.
2. Benchmark leaderboards
These score models on fixed datasets, such as mathematical reasoning, code generation, and multilingual understanding. Their advantage is that they are objective and reproducible. Their disadvantage is the risk of data contamination: if test-set content has leaked into training data, scores can be artificially high. In addition, many benchmark task types are quite different from real business tasks.
3. Vertical-scenario leaderboards
Examples include arena-style leaderboards for front-end page generation, where multiple models complete the same page requirement and the results are compared. These leaderboards are most relevant to developers’ actual work and tend to have higher reference value, but they cover a limited range of task types.
Practical advice: Treat leaderboards as a “filter,” not a “verdict.” Use them to narrow the candidate pool from a dozen models to three or four, and leave the rest to hands-on testing.
3. Step 2: Design Your Own Evaluation Set
This is the most valuable step in the entire selection process. No matter how authoritative a general leaderboard is, it is less reliable than a test set based on your own business. Consider building one as follows:
1. Sample from business logs
Draw 50–200 representative tasks from real sources such as customer support records, code repositories, and document-writing tasks. Make sure to cover three types of samples: high-frequency simple tasks (to ensure no regression), low-frequency critical tasks (to ensure fallback coverage), and boundary hard tasks (to understand the capability ceiling).
2. Define verifiable evaluation criteria
Use different evaluation methods for different task types instead of relying entirely on “gut-feel scoring”:
- Structured-output tasks: directly validate JSON validity and field completeness, and write automated assertions where possible;
- Code-generation tasks: run unit tests and use the pass rate as the core metric;
- Knowledge question-answering tasks: have business experts write key points for the reference answer, then score by key-point hit rate;
- Open-ended creative tasks: use multi-model peer review or human blind review, but keep the sample size large enough to reduce randomness.
3. Turn the evaluation into a repeatable script
Evaluation is not a one-time activity. Model versions keep updating, and a fixed evaluation script lets you quickly run regression validation after a new version is released. Below is a minimal usable evaluation skeleton:
import jsoncases = json.load(open("eval_cases.json")) # Business test setdef evaluate(model_client, cases):passed = 0results = []for case in cases:resp = model_client.chat(messages=[{"role": "user", "content": case["prompt"]}])ok = check_structured_output(resp, case["expected_schema"]) \if case["type"] == "structured" \else run_unit_tests(resp, case["test_code"]) \if case["type"] == "code" \else manual_review(resp, case)passed += okresults.append({"id": case["id"], "pass": ok, "output": resp})print(f"Pass rate: {passed / len(cases):.1%}")return results
During evaluation, also control variables: fix the temperature parameter, fix the system prompt, and take multiple samples and average them (at least three runs per test case is recommended). Otherwise, result variance can mask real differences.
4. Step 3: Evaluate Engineering Fit
Beyond “question-answering ability,” engineering-level performance is often what determines whether a launch succeeds. Focus on the following areas:
1. Instruction following and format stability
In production environments, model output usually needs to be parsed by downstream programs. Test whether the model can stably output valid JSON and strictly follow length and format constraints under long conversations and complex system prompts. Format drift is one of the most common pitfalls in engineering integration.
2. Real usability of context length
The advertised context window length and the “effective context” are two different things. You can run a needle-in-a-haystack-style test: bury key information in a long document and check whether the model can retrieve and cite it accurately. For long-document tasks, also pay attention to whether first-token latency and overall response time are within an acceptable range.
3. Tool-calling capability
If the business involves Agent or RAG architectures, focus on testing the accuracy of function-call argument extraction, the execution stability of multi-step tool chains, and the model’s ability to self-correct when exceptions occur.
4. Integration cost and ecosystem compatibility
Prefer services with common API protocols, streaming output support, and complete SDKs. Many cloud platforms also provide prompt template management, content safety moderation, call monitoring and alerting, and logging. These capabilities can save significant build-it-yourself effort when constructing production-grade applications. Some platforms also offer hosted inference services for open-source models; if the team has private deployment requirements, include them in the evaluation as well.
5. Step 4: Calculate Total Cost, Not Just Unit Price
Many teams compare only the price per million tokens during selection. That is a typical mistake. The correct calculation is cost per task:
Cost per task = average tokens consumed per task × unit price ÷ task success rate
A model with a low unit price but only 70% success rate may, after adding retries and human fallback costs, end up much more expensive than a model with a higher unit price that succeeds on the first attempt. Also account for: the labor cost of prompt debugging, the migration risk of moving from a cheap model to a stronger one, and the hidden impact of rate limits and concurrency quotas on peak-period business.
A common optimization pattern is tiered routing: route simple tasks to lightweight models and complex tasks to flagship models. Use a classifier or rules to perform a first-pass rough filter; overall cost often drops by more than half without sacrificing the quality of critical tasks.
6. Model Selection Decision Checklist
Finally, here is the entire method compressed into a checklist for hands-on use:
- Use leaderboards to narrow the candidate pool to no more than three models, and clarify each candidate’s “expected strengths”;
- Build 50–200 evaluation cases based on real business tasks, and define verifiable pass criteria;
- Run controlled hands-on tests, focusing on three engineering metrics: format stability, effective context, and tool calling;
- Calculate costs by “cost per task,” not “token unit price”;
- Keep the evaluation scripts and include model upgrades in the regular regression process;
- When producing a selection report, include evaluation data and failure cases so decisions are traceable.
The LLM market is still evolving rapidly, and today’s optimal choice may not be the optimal choice six months from now. The real competitive advantage is not “betting on the right model,” but building a low-cost, sustainable capability to evaluate and switch models. That is the fundamental way to cope with the uncertainty of technological iteration.
