How to Choose the Right Chinese LLM: A Practical Evaluation Framework
Selecting the right Chinese large language model (LLM) requires more than just glancing at public benchmark rankings. This article presents a scientific evaluation framework based on three core dimensions: capability, engineering, and cost. It explains why public leaderboards may not reflect real business needs, provides a step-by-step method for building your own business-specific evaluation dataset, outlines key technical performance metrics for inference services and private deployment, and introduces a clear checklist for making your final decision. The article also emphasizes the value of keeping model selection flexible to adapt to the fast-paced evolution of LLMs.
Why Choosing a Model by Public Rankings Can Be Misleading
The pace of Chinese LLM releases has accelerated significantly, with new models appearing almost every month. The most common mistake developers make is relying on well-known public rankings: they simply pick the highest-scoring model.
This approach suffers from a few critical flaws:
- Benchmark performance ≠ business performance. Mainstream leaderboards (e.g., general chat arenas, code evaluation suites) measure general conversational and coding ability. Your business, however, may require domain-specific strengths like long-document extraction, structured output, or multi-turn tool calling. A model ranked fifth overall could easily outperform the first-ranked model on your particular tasks.
- Benchmark scores are gameable. Some models are trained with data aligned to common public evaluation sets. This leads to models that score high on exams but underperform in real-world use cases — a phenomenon often called evaluation set contamination.
- Snapshot evaluations become stale quickly. With models iterating so quickly, an evaluation conclusion from just three months ago may already be outdated.
The key is to shift your mindset: instead of searching for the “most capable model,” your goal should be to find the most suitable model under your specific constraints. The evaluation process can be broken down into three main tracks: capability, engineering, and cost.
Capability: Decomposing “Good” into Measurable Metrics
2.1 Interpreting Existing Benchmarks Correctly
Public benchmarks are a good starting point for initial screening, as long as you follow these principles:
- Focus on capability clusters, not total scores. Look past the aggregate score to see performance breakdowns for areas like code generation, mathematical reasoning, long-context understanding, instruction following, and multilingual ability. Identify which strengths align with your business needs.
- Check the evaluation protocol. Even on the same benchmark dataset, differences in sampling temperature, prompt templates, or “thinking mode” settings can lead to huge performance gaps. Prefer leaderboards that publish their complete evaluation configuration.
- Cross-validate with multiple sources. Compare the results of at least two or three independent leaderboards. Do not give more than 30% weight to any single one.
2.2 Build Your Own Business-Specific Evaluation Set
This is the single most valuable step in the entire selection process. Here is how to do it:
- Sample real requests from your business logs. Cover three categories: high-frequency scenarios, edge cases, and previously failed interactions. A good starting size is 100 to 300 cases.
- Define quantitative metrics for each task type. For example:
- For structured output tasks: JSON parsing success rate and field accuracy
- For retrieval-augmented generation (RAG): answer faithfulness (checks for fabricated information) and citation accuracy
- For code generation: the pass rate of compile and test suites
- Write an automated evaluation script. The following is minimal code for parallel multi-model evaluation:
import json, concurrent.futuresCANDIDATES = ["model_a", "model_b", "model_c"] # models to be testeddef run_case(client, model, case):resp = client.chat.completions.create(model=model,messages=[{"role": "user", "content": case["prompt"]}],temperature=0.1,response_format={"type": "json_object"}, # used when structured output is required)return judge(case, resp.choices[0].message.content)def judge(case, output):# Dispatch to different scorers based on task type: rule matching / unit tests / LLM-as-a-judge...with concurrent.futures.ThreadPoolExecutor() as pool:for model in CANDIDATES:scores = list(pool.map(lambda c: run_case(client, model, c), CASES))print(model, aggregate(scores))
- Keep variables tightly controlled. Use the same set of cases, identical prompts, the same temperature and maximum output length across all models. For more reliable results, run multiple iterations and take the average.
- Involve human evaluation for subjective tasks. For tasks that cannot be judged automatically (e.g., writing quality), have a human review approximately 20% of the outputs and check for consistency with automatic scores.
2.3 Capabilities That Are Often Overlooked
The following capabilities appear frequently in daily business use but are often neglected in the model-selection process:
- Instruction-following stability: Can the model consistently respect output length limits, enums, and negative constraints like “do not include certain content”?
- Function-calling quality: Are parameters extracted accurately? Does the model hallucinate parameters that do not actually exist?
- Long-context degradation curve: Most models show a noticeable drop in recall as input approaches the context window limit. We recommend testing at 50%, 80%, and 95% context utilization.
- Time-to-first-token (TTFT) in streaming mode: This directly impacts interactive user experience. We discuss details next.
Engineering: Inference Services and Deployment Options
3.1 Three Primary Paths for Model Access
| Path | Typical Use Case | Key Considerations |
|---|---|---|
| Cloud-hosted API | Quick validation, high traffic fluctuation | Pricing per token, concurrency quotas, SLA, data compliance |
| Self-hosted inference cluster | Data must stay in-house, very high throughput | GPU memory cost, throughput tuning, maintenance overhead |
| On-device / small private models | Offline scenarios, strict privacy constraints | Model size, quantization accuracy loss |
3.2 Evaluating Managed Inference Platforms
The same model is often available from multiple inference platforms. These platforms can differ significantly in two major ways:
- Inference optimization. Most platforms offer a “model-as-a-service” model: you call a unified API format, while the platform handles quantization, batching, and caching. Actual throughput and latency can easily vary by a factor of several times between platforms. We strongly recommend performance testing instead of trusting marketing claims.
- Quotas and rate-limiting policies. The eval phase rarely triggers rate limits, but real production traffic may hit them. Always check the platform’s max concurrency and rate limit constraints before you go live.
When running load tests, we recommend recording at least these metrics:
Time-to-first-token (TTFT) -- determines the streaming user experienceEnd-to-end latency (P50 / P95 / P99) -- determines overall usabilitySingle-concurrency throughput (tokens/s)Maximum sustainable concurrency -- determines capacity planning
3.3 Cost Analysis of Private Deployment
A rough estimate for private deployment is: multiply the model parameter count by your chosen quantization precision (e.g., a 7B FP16 model requires roughly 14GB+ of GPU memory), then add the overhead for the KV cache and the inference framework, and multiply again by your target concurrency level to get the total GPU memory requirement. It is wise to factor in an additional 20% to 30% for framework inefficiency and redundancy. Many teams underestimate the GPU memory explosion of the KV cache specifically in long-context scenarios.
Cost: Beyond the Price per Token
The per-token API price is just the most visible part of the cost. Make sure you account for these three factors:
- Prompt engineering costs. A weaker model may require much longer prompts, more few-shot examples, and more retries to achieve the same level of quality. The actual per-task cost could end up being higher than that of a stronger model. Compare using “cost to complete one task successfully” rather than “price per token”:
Task cost = (input tokens × input price) + (output tokens × output price) + (average retry count × previous sum). - The benefit of a cascade strategy. A common pattern is to use a smaller, cheaper model as a first pass and escalate to a larger model only for low-confidence cases. Before adopting this in production, validate the accuracy loss against your evaluation set to make sure it is acceptable.
- Prompts caching and batch discounts. Many inference providers offer discounted pricing for prefix caching and batch inference. For applications with long system prompts, the cache hit rate can dramatically affect your overall costs and should be measured during evaluation.
A Practical Decision Checklist
The following list summarizes the full evaluation methodology:
- Screen candidates using public leaderboards; shortlist 3 to 5 models.
- Build a business-specific evaluation set with 100+ real cases taken from your workload logs.
- Run automated evaluations with human spot-checks; compare results by capability cluster, not by total score.
- Benchmark each candidate on different inference platforms to measure latency, throughput, and rate-limit quotas.
- Calculate the actual cost per successful task, including retries and prompt caching.
- Check your data compliance requirements to select between managed APIs or private deployment.
- Build a model abstraction layer in your architecture (e.g., a unified adapter for multiple APIs) to allow seamless model switching at any time.
A final but critical piece of engineering advice: do not hard-code model selection in your business logic. Implement a config-driven model router, combined with a canary release process. When a new model is released, direct about 5% of traffic to it, observe the key metrics, and then gradually increase the share. LLMs will keep evolving; the evaluation framework and the abstraction layer you build are the long-term assets you will truly rely on.
