How to Evaluate Chinese LLMs: A Reproducible Multi-Model Comparison Method
This article provides developers with a reproducible workflow for evaluating and selecting Chinese large language models: connect multiple models to the same environment through a unified interface, benchmark them against a fixed task set with clear scoring rules, and carry the findings into production. It does not rank specific models or cite unverified version information—always refer to each platform's current documentation for model IDs, endpoints, and pricing.
1. Understand the Model Landscape Before Deciding What to Test
Chinese LLMs differ mainly in how they are positioned, which helps you define what questions each type should answer during testing:
- Flagship general-purpose models: Typically built for multimodal understanding, complex code generation, and long-context tasks. They suit the primary capabilities of an application. Focus testing on accuracy and quality on complex tasks.
- Cost-efficient models: Emphasize inference speed and per-call cost. They fit high-frequency, latency-sensitive scenarios such as batch classification, summarization, and format conversion. Focus testing on speed, stability, and performance per unit of cost.
- Domain-specific models: Optimized for particular areas such as content creation or customer support Q&A. Test them with real samples from your business, not generic questions.
Classification is only a starting point. Vendors iterate quickly—model rankings, context window sizes, and pricing can all change within weeks. Before selecting a model, verify information against the official model list and current release notes, and record the model IDs and dates you checked as the basis for later comparison.
2. Why Use a Unified Inference Service for Benchmarking
Deploying open-source weights locally is an alternative path, but for most individual developers and small teams, the barrier is high:
- Hardware requirements: Large models have big weight files and demand significant GPU memory and inference hardware. Deployment and maintenance costs may exceed the value of the evaluation itself.
- Environment consistency: When multiple models are accessed through the same inference service, they run on identical hardware and software stacks, making latency and throughput comparisons meaningful. With separate deployments, environment differences will skew the conclusions.
- Version management: Inference platforms typically offer stable model versions with published update records, sparing you from managing dependencies and version compatibility yourself.
Regarding cost: free quotas, quota sizes, and concurrency or rate limits all depend on each platform’s current documentation. Once the free quota is exhausted, most platforms bill by input and output token count. Before testing, check the pricing page to confirm unit prices and billing units, so evaluation costs don’t exceed expectations.
3. The Integration Process in Four Steps
The integration consists of four steps: creating an account, generating a key, configuring a client, and running batch calls with a script. Console menu names and button labels may change between versions—when in doubt, follow the platform documentation.
3.1 Create and Verify an Account
Register an account on the platform’s developer page and complete email or phone verification. Some platforms may also require identity verification for compliance reasons. If a verification code doesn’t arrive, check your spam folder, confirm your phone number format is correct, and then wait a moment before requesting a new code. Most platforms won’t allow API calls until verification is complete.
3.2 Create an API Key and Keep It Safe
Go to the key management page in the console (labeled “API Keys” or “Access Tokens” depending on the platform) and create a new key. Name it after its purpose and environment—for example, eval-local or staging-gateway—so you can troubleshoot and revoke it later.
For expiration, choose from the options available in the console—typically the longest allowed period is fine—but also set up a rotation plan. Keys are usually shown in full only once at creation; once you close the page, they cannot be viewed again, and a lost key must be regenerated.
Security requirements:
- Never commit keys to a code repository or share them through chat tools.
- Read keys from environment variables in your code, such as
LLM_API_KEY, rather than hardcoding them in source files. - Use separate keys for testing and production environments, and grant only the minimum necessary permissions.
- If you suspect a leak, revoke the old key in the console immediately and generate a replacement.
3.3 Configure a Graphical Client
For everyday conversational testing, a graphical client is more intuitive than writing scripts. Take the open-source desktop client Cherry Studio as an example (available on Windows, macOS, and Linux): open the settings page, find “Model Services” or “Model Providers,” and add a provider compatible with the OpenAI API. Fill in three fields:
- API key: Paste the key generated in the previous step.
- API address: Enter the Base URL from the platform documentation. OpenAI-compatible endpoints usually end with
/v1; omitting it will cause requests to fail, so follow the documentation. - Model ID: Copy the full identifier from the platform’s model list—don’t type it from memory.
After saving, click the test button or send a simple message to confirm the response is normal before proceeding with comparisons.
3.4 Batch Calls with a Script
A graphical interface suits manual observation; batch testing requires a script. The example below uses the official OpenAI Python SDK, which works with any platform offering an OpenAI-compatible API. The script does three things: reads configuration from environment variables, retries with exponential backoff on failure, and writes each result with its latency to a JSON file for later analysis.
import jsonimport osimport timefrom openai import OpenAI, APIErrorclient = OpenAI(base_url=os.environ["LLM_BASE_URL"], # OpenAI-compatible endpoint from the platform docsapi_key=os.environ["LLM_API_KEY"],timeout=60,)# Copy real, callable model IDs from the platform's model listMODELS = ["<model-id-a>","<model-id-b>",]PROMPTS = ["Explain dependency injection and provide an example in Go.",]REPEAT = 3 # Repeat each question to observe stabilitydef ask(model, prompt, retries=3):for attempt in range(1, retries + 1):try:start = time.perf_counter()resp = client.chat.completions.create(model=model,messages=[{"role": "user", "content": prompt}],temperature=0.2,)return {"model": model,"prompt": prompt,"answer": resp.choices[0].message.content,"latency_s": round(time.perf_counter() - start, 2),"ok": True,}except APIError as e:if attempt == retries:return {"model": model, "prompt": prompt,"error": str(e), "ok": False}time.sleep(2 ** attempt)results = []for prompt in PROMPTS:for model in MODELS:for i in range(REPEAT):item = ask(model, prompt)item["run"] = i + 1results.append(item)with open("eval_results.json", "w", encoding="utf-8") as f:json.dump(results, f, ensure_ascii=False, indent=2)
Before running, set LLM_BASE_URL and LLM_API_KEY in your environment. If a model ID in MODELS returns an error, it’s usually a typo or the model hasn’t been enabled for your account—check the model list to confirm.
Note that this code measures total response time only, not time to first token. To measure that, switch to streaming (stream=True) and record the time from sending the request to receiving the first chunk.
4. Designing a Reproducible Evaluation
Drawing conclusions from one or two answers leaves you at the mercy of chance. A reproducible evaluation requires deciding the following in advance.
1. A fixed task set. Prepare questions by category, such as code generation, logical reasoning, long-text comprehension, and instruction following. Questions from real business scenarios are best, and keep reference answers or test cases.
2. A clear scoring rubric. The weights below are a starting point and can be adjusted to your business:
| Dimension | Suggested Weight | Judging Method |
|---|---|---|
| Code correctness | 30% | Run the code; passes only if it passes the predefined test cases |
| Reasoning accuracy | 25% | Compare each answer against the reference answer |
| Instruction following | 15% | Check constraints against a checklist: format, length, tone, etc. |
| Logical coherence | 15% | Blind scoring on a 1–5 scale, averaged across two independent raters |
| Speed and stability | 15% | Record median time to first token and total latency, plus failure rate |
3. Consistent measurement. Time to first token runs from sending the request to receiving the first chunk. Total latency runs from sending the request to the end of the response. Call each task at least 3 times and report the median and variance—never a single result.
4. Blind scoring. Hide model names while scoring to avoid bias. Have a second person score independently, then compare the two sets of results for consistency.
5. Predefined pass criteria. For example, code questions must pass all test cases, reasoning answers must be exactly correct, and speed metrics that exceed a threshold are considered failing. Define the criteria before looking at results—only then do the conclusions hold up.
Additionally, the gap between models is shifting from breadth of knowledge to depth of reasoning. Simple factual Q&A no longer differentiates capability. Design more questions requiring multi-step reasoning, implicit-condition inference, or cross-paragraph information synthesis—they better reflect a model’s true ability.
5. Engineering Considerations for Production Selection
After evaluation, you’ll typically be left with one or two candidates. Before production, address the following:
- Pin the version: In production configuration, use the exact version identifier from the platform documentation, not an alias that may change with updates. Subscribe to vendor update announcements and re-run the evaluation set before upgrading.
- Fallback and failover: Don’t rely on a single model. Implement model switching at the gateway layer, with trigger conditions defined in configuration. A sample setup:
primary: <primary-model-id>fallback:- <fallback-model-id-1>- <fallback-model-id-2>switch_when:timeout_s: 30 # per-request timeoutconsecutive_failures: 3 # consecutive failure counthttp_status: [429, 503] # rate-limited or service unavailablerecover_check_interval_s: 300 # how often to probe whether the primary model has recovered
Log switches so you can analyze how often the primary model fails afterward. Fallback models may produce different output styles—confirm with the business side that this is acceptable before enabling switching.
- Cost estimation: Estimate monthly cost from average input tokens, average output tokens, expected daily request volume, and official unit prices. Formula: monthly cost = request volume × (average input tokens × input price + average output tokens × output price). Free quotas are for evaluation only and shouldn’t be treated as a basis for long-term cost.
- Data compliance: If your business involves personal or sensitive data, confirm the platform’s data processing agreement before launch—whether data is used for model training, how long it’s retained, and whether private deployment or in-region processing is supported. These questions should be confirmed by your compliance owner, not just by reading API documentation.
6. Conclusion
Chinese LLMs are converging rapidly in capability, with different models excelling at different tasks. Third-party leaderboards are only a reference in this situation—they reflect performance on specific test sets, and your business data may differ.
A more reliable approach is to put candidate models in the same environment and benchmark them with a fixed task set, a clear scoring rubric, and reproducible measurement. The process itself is reusable long term: re-run it after model updates to see what changed and catch regressions early.
