Chinese LLM Selection: Evaluation Dimensions and Benchmarking Practices
Choosing a Chinese LLM requires more than scanning leaderboard rankings. This article presents a reproducible selection method covering quality, output stability, latency, total cost, compliance, and engineering integration, with practical test-set design, automated scoring, decision matrices, canary releases, deployment choices, and post-launch governance.
The question of “which model should we choose?” is one of the most underestimated parts of technical selection. Many teams glance at a leaderboard, pick a top-ranked model, put its model ID into a configuration file, and go live. Only after users report poor answers, frequent timeouts, or end-of-month bills exceeding budget do they realize that the evaluation they should have done was never performed.
Chinese large language models are being released with increasing frequency, and the number of options continues to grow. That is good for business, but it also makes selection harder. This article does not compare specific products. Instead, it addresses a more fundamental question: how can teams build a scientific, reproducible model-selection method?
1. Leaderboard Rankings Are a Reference, Not a Conclusion
Public leaderboards are valuable. They can roughly quantify model capabilities and place models side by side. But treating them as the final basis for selection can easily lead to mistakes.
1.1 Data Contamination: The Test Set May Have Been “Seen” During Training
When an evaluation set is published online, model training data may include those questions, intentionally or unintentionally. In that case, the model has already “seen the answers” during training. A high score produced this way does not mean the model will perform equally well on entirely new tasks. The faster evaluation sets are iterated and the more they are reused, the more serious this problem becomes.
1.2 Task Mismatch: The Leaderboard May Not Measure What You Need
A leaderboard focused on web code generation measures a model’s ability to generate complete, runnable page code. But your business may need something different:
- structured data output under strict schemas;
- key-field extraction from long documents;
- understanding of industry terminology and Chinese-language expression habits.
These needs emphasize different capabilities. A model that ranks high on code generation and also performs reasonably well on another benchmark may still be a poor fit for your scenario.
1.3 Inconsistent Scoring Criteria: There Is No Universal Ruler
Different leaderboards may use different scoring methods, sampling parameters, prompt templates, and even inference configurations at model deployment. Looking only at the total score can obscure how the dimensions are composed, and comparisons across leaderboards are even less comparable.
1.4 Version Drift: Models Keep Changing
Vendors continuously upgrade their models. A conclusion reached today may not hold next month. In addition, when calling a model through a hosted platform, the model catalog and available versions may change. Model identifiers should be abstracted into configuration items instead of being scattered through code as strings.
2. A Six-Dimension Evaluation Framework
It is recommended to build an evaluation framework around the following six dimensions. The first four determine whether a model is usable and how well it works. The last two determine whether it can be used reliably over the long term.
| Dimension | Key metrics | Evaluation method | Weight (example) |
|---|---|---|---|
| Output quality | Task-set pass rate, human blind-review preference rate | Regression testing on business data sets, blind review | 35% |
| Output stability | JSON validity rate, field completeness rate, timeout rate, refusal rate | Batch stress testing | 20% |
| Performance and latency | Time to first token, output speed, behavior under high concurrency | Real-world concurrency stress testing | 15% |
| Total cost | Cost per request, monthly bill, hidden costs | TCO calculation based on call volume | 15% |
| Compliance and security | Data residency, content safety, auditing capabilities | Specialized assessment by the security team | Threshold item (not weighted) |
| Engineering integration | API compatibility, streaming output, concurrency and rate limiting, observability | Technical proof of concept (PoC) | 15% |
Weights should be adjusted to the actual business context. For a customer-service Q&A scenario with extremely high quality requirements, the weight for output quality can be increased. For a consumer-facing scenario with massive traffic and high latency sensitivity, the weight for performance should be increased.
2.1 Output Quality: Let Business Data Speak
Split the test set into three layers:
- Core questions (about 20%): high-frequency, typical requests that represent the main scenarios in daily user usage;
- Boundary questions (about 30%): scenarios that are prone to problems, such as extremely long inputs, ambiguous expressions, and multi-turn follow-up questions;
- Randomly sampled questions (about 50%): questions randomly selected from real logs to reflect the real distribution.
Prioritize rule-based validation for core and boundary questions. For example:
- Code tasks: does the output compile or pass unit tests?
- Extraction tasks: is the JSON valid, are all fields present, and are key values correct?
For open-ended questions that are difficult to judge automatically, use human blind review or a “judge model” to assist with scoring.
2.2 Output Stability: The Hidden Barrier to Production Deployment
Many models “perform well occasionally” but are difficult to deploy reliably. The problem often lies in unstable output. Key observations include:
- when JSON output is required, are there many invalid formats?
- in multi-turn conversations, does the model frequently fail to answer the question?
- with long inputs, are there truncations or missed key information?
- under high load, what are the timeout rate and the distribution of error codes?
Use fixed sampling parameters and a sufficient sample size during evaluation, and repeat tests across different time periods and concurrency levels to avoid being misled by random fluctuations.
2.3 Performance and Latency: Focus on What Users Actually Perceive
The feeling of “fast” in user experience is determined by two metrics:
- Time to first token (TTFT): the time from sending a request to receiving the first content chunk, which determines how waiting feels;
- Output speed: the number of tokens generated per unit of time, which determines how long it takes to generate the complete answer.
A service that performs well under low load may degrade noticeably under the concurrency pressure of peak business hours. During stress testing, cover scenarios with several times the peak traffic, observe whether latency grows linearly, and confirm whether the platform provides concurrency quotas, rate-limiting policies, and queuing mechanisms.
2.4 Total Cost: Do Not Focus Only on Unit Price
When comparing prices, unit price is only one item. Complete cost includes at least:
- Input and output billing: in long-document scenarios, input tokens often account for the majority, and the price difference between input and output can have a clear impact;
- Retries and redundancy: failed retries and duplicate generation requests also incur costs;
- Hidden costs: multiple calls caused by cache hits, output length, and context-length limits;
- Engineering cost: development effort spent adapting different APIs and writing compatibility layers.
2.5 Compliance and Security: Non-Negotiable Thresholds
The following items must be confirmed in advance:
- whether data will be used for training and whether this can be disabled;
- whether data is processed within the required jurisdiction and whether it meets industry regulatory requirements;
- whether content safety policies and call auditing are supported.
If any item fails to meet the bottom line, the model should be excluded regardless of how good its output quality is.
2.6 Engineering Integration: Reduce Switching Costs
API consistency, streaming output, concurrency and rate-limiting policies, and observability (logs, metrics, and tracing) all affect the efficiency of subsequent maintenance and incident diagnosis. It is recommended to isolate the specific model behind an API adapter layer so that business code does not directly depend on any one vendor’s service.
3. Automated Evaluation: From Real Problems to Data-Driven Conclusions
Comparing a few models by feel is slow and easily influenced by subjective preferences. The approach below turns evaluation into a repeatable command.
3.1 Test-Set Structure
[{"id": "c001","category": "code","input": "Implement a Python function that determines whether a given string is a palindrome, ignoring case and punctuation.","rules": {"type": "code","assert": "func('A man, a plan, a canal: Panama') == True"}},{"id": "e001","category": "extract","input": "Extract the name, price, and warranty period from the following product description...","rules": {"type": "json","required_fields": ["name", "price", "warranty"]}},{"id": "o001","category": "open","input": "How should I design a reasonable logging specification for my team?","rules": {"type": "manual"}}]
3.2 Evaluation Script
"""Generic LLM evaluation script (illustrative): supports concurrent requests, rule checks, and metric aggregation."""import asyncioimport jsonimport statisticsimport timeimport httpxAPI_BASE = "https://example.invalid/v1" # Replace with the selected model service endpointMODEL = "MODEL_ID" # Replace with the selected model identifierKEY = "YOUR_API_KEY"CASES = json.load(open("cases.json", encoding="utf-8"))SEMAPHORE = asyncio.Semaphore(8) # Concurrency limit: adjust to the service quotadef check_rule(rule, text, usage):"""Rule checks for objective criteria such as executable code or valid JSON structure."""t = rule["type"]if t == "code":scope = {}try:exec(text, scope) # For controlled evaluation environments only; do not execute untrusted code directlyreturn scope["func"]("A man, a plan, a canal: Panama") is Trueexcept Exception:return Falseif t == "json":try:data = json.loads(text)except json.JSONDecodeError:return Falsereturn all(k in data for k in rule["required_fields"])return None # Open-ended questions should be routed to human review or a judge modelasync def run_case(client, case):async with SEMAPHORE:started = time.perf_counter()try:resp = await client.post(f"{API_BASE}/chat/completions",headers={"Authorization": f"Bearer {KEY}"},json={"model": MODEL,"messages": [{"role": "user", "content": case["input"]}],"temperature": 0, # Fix sampling parameters during evaluation"max_tokens": 1024,},)resp.raise_for_status()except httpx.HTTPError:return {"id": case["id"], "passed": None, "ttft": None, "error": True}finally:passlatency = (time.perf_counter() - started) * 1000text = resp.json()["choices"][0]["message"]["content"]usage = resp.json().get("usage", {})return {"id": case["id"],"passed": check_rule(case["rules"], text, usage),"latency": latency,"tokens": usage.get("completion_tokens", 0),}async def main():async with httpx.AsyncClient(timeout=120.0) as client:results = await asyncio.gather(*(run_case(client, c) for c in CASES))valid = [r for r in results if r["passed"] is not None]passed = sum(r["passed"] for r in valid)latencies = [r["latency"] for r in valid if r["latency"]]print(f"Rule-based pass rate: {passed}/{len(valid)}")print(f"Average latency: {statistics.mean(latencies):.0f}ms, "f"P95: {statistics.quantiles(latencies, n=20)[18]:.0f}ms")print(f"Failed requests: {sum(1 for r in results if r.get('error'))}")asyncio.run(main())
3.3 Usage Notes
- Run a small batch first: confirm that the API and response structure are correct before running the full set.
- Handle open-ended questions separately: introduce a judge model, anonymize and mix the original question, reference answer, and candidate answers, then score them. To prevent position bias, randomize the order of candidate answers. The judge model must not be the only criterion; key conclusions still require human review.
- Compare multiple models serially: use the same inputs, the same parameters, and the same concurrency.
- Output a structured report: record each model’s average score, pass rate, latency distribution, and representative failure cases.
4. From Scores to Decisions
4.1 Decision Matrix: Check Thresholds First, Then Compare Totals
Before calculating a total score, check one-vote veto items such as compliance and data security. Then calculate a weighted score according to the weights, and rank the options together with business priorities.
| Dimension | Model A | Model B | Model C |
|---|---|---|---|
| Output quality (35%) | 85 | 90 | 75 |
| Output stability (20%) | 80 | 85 | 70 |
| Performance and latency (15%) | 90 | 80 | 85 |
| Total cost (15%) | 88 | 78 | 90 |
| Engineering integration (15%) | 80 | 85 | 85 |
| Weighted total score | 84.45 | 84.95 | 79.25 |
For Model A: 85×0.35 + 80×0.20 + 90×0.15 + 88×0.15 + 80×0.15 = 84.45.
In this example, Model B has a slightly higher total score, mainly because of its advantages in quality and stability. Model A performs better on latency and cost, making it more attractive for latency-sensitive or cost-sensitive businesses. Model C trails the other two because of a clear weakness in quality. This shows that there is no single right answer in model selection—only an answer that fits the business.
4.2 Small-Traffic Canary Release
After selecting a model, do not switch all traffic at once. First send a small portion of traffic (for example, 5%) to the new model, monitor success rate, latency, and user feedback, and then gradually increase the percentage while retaining the ability to roll back immediately.
4.3 Tiered Routing: Balance Cost and Quality
Not every request needs the “strongest model.” Requests can be routed by task difficulty:
- simple queries, intent recognition, and formatting tasks can go to a lightweight model or a smaller-parameter model;
- complex reasoning, code generation, and multi-document analysis can go to a more capable model.
This strategy can usually reduce overall cost significantly while preserving the experience for complex tasks.
5. Cloud Service or Private Deployment?
After evaluation, teams still need to decide on a deployment model.
Managed cloud API
- Advantages: ready to use, elastic, and no need to manage GPU operations;
- Considerations: billing is based on call volume, so costs grow linearly with heavy traffic; the business also depends on the provider’s availability and quotas.
Private deployment of open-source models
- Advantages: full data control, no per-call billing, and deep customization through fine-tuning, quantization, and parallel strategies;
- Considerations: hardware investment and operational costs are required, capacity planning is complex, and peak idle capacity can cause resource waste.
A common simplification is: stable, predictable call volumes are better suited to private deployment, while highly variable businesses that need fast time to market are better suited to cloud services. In the end, however, the decision should be based on calculations using real traffic data.
Whichever option is chosen, retain the architectural ability to replace the model:
- uniformly encapsulate model calls through a gateway or adapter layer;
- store model identifiers in a configuration center rather than scattering them through business code;
- establish unified routing, retry, timeout, and fallback policies.
Major cloud providers such as Baidu AI Cloud often offer a broad portfolio of LLM products and development platforms that can be included in the candidate pool for evaluation. Specific capabilities and service terms should be verified against the official documentation.
6. After Launch: Continuous Governance
The end point of model selection is not launch; it is continuous governance.
- Establish a regression baseline: after every model upgrade or application change, rerun the evaluation with the same test set to ensure that quality has not degraded.
- Monitor online metrics: track request success rate, P95 latency, output-format validity rate, negative user feedback rate, token consumption, and costs.
- Pin model versions: if the service supports version pinning, production should lock to a verified version. Validate new versions in staging first, and then decide whether to upgrade.
- Re-evaluate periodically: reassess candidate models quarterly or semiannually so that selection decisions keep pace with technical evolution.
7. Common Pitfalls and a Self-Check List
| Pitfall | Problem | Correct approach |
|---|---|---|
| Leaderboard-only selection | Data contamination and task mismatch | Evaluate with your own business data |
| Looking only at unit price | Ignores retries, caching, and engineering costs | Calculate total cost based on full call volume |
| One-time full switch | Lacks rollback and canary capabilities | Use small-traffic canary releases and keep a switch |
| Hardcoded model identifiers | Version changes can cause failures | Use configuration and adapter-layer isolation |
| Missing continuous evaluation | Quality may silently degrade after model upgrades | Maintain a regular regression baseline |
Model-selection self-check list:
- Has a business test set covering core, boundary, and random-sampled cases been assembled?
- Have rule-based validation and human blind review been completed?
- Have latency and stability been stress-tested under high concurrency?
- Has total cost been calculated based on full call volume?
- Has compliance and security assessment been completed?
- Are canary release and rapid rollback capabilities in place?
- Have post-launch monitoring and periodic re-evaluation mechanisms been established?
Conclusion
The range of Chinese large language models available is now broad enough that “which one to choose” is shifting from a contest of capabilities to a contest of fit. Rather than chasing temporary leaderboard positions, it is more worthwhile to invest in building your own evaluation system:
- Clarify the task objectives;
- Establish multidimensional criteria;
- Validate with real data;
- Roll out through canary releases;
- Ensure long-term quality through continuous governance.
This is the most confident and durable answer to the question of “which model should we choose?”
