Chinese LLM Selection Guide: From Benchmarks to Production Integration
This guide provides a practical framework for selecting and integrating Chinese LLMs. It explains why public leaderboards should be used only for initial screening, how to define measurable evaluation dimensions, how to build a small internal evaluation set, how to add a unified model access layer, how to compare models fairly, and which engineering, cost, security, and compliance practices are required before production deployment.
1. First, avoid the “leaderboard decides everything” trap
Chinese LLMs have been iterating quickly over the past two years. New flagship versions appear almost every quarter, and public leaderboards are constantly updated. Many teams still follow a simple selection process: open a leaderboard, look at the rankings, and connect to the top model. In engineering practice, this is a high-risk decision for three reasons:
- Leaderboard tasks are not the same as business tasks. Public evaluations often emphasize general reasoning, code generation, instruction following, and other automatically gradable tasks. Real business workloads often involve domain-specific long-document question answering, structured extraction, or multi-turn support ticket handling. The distributions are very different.
- The same model can behave differently under different sampling settings. Temperature, top-p, maximum output length, and whether a reasoning chain is enabled can significantly change output style and accuracy. A leaderboard reports performance under one fixed configuration, not the performance you will obtain after tuning.
- Leaderboards suffer from version drift. Models are often updated continuously. A version that appears on a leaderboard today may be silently replaced weeks later. If your selection decision is based on a version number rather than your own evaluation set, you lose the ability to run regression validation.
The conclusion is simple: use leaderboards for initial screening, and use your own evaluation set for decision-making.
2. Break model selection into six measurable dimensions
Selection is not about choosing the single “strongest” model. It is about choosing the most suitable model for a specific set of constraints. The evaluation can be broken into six dimensions, each with measurable indicators.
1. Task quality. Measure accuracy, recall, structured-field completeness, and human ratings (1–5) using your own business data, not generic leaderboard scores. The quality dimension should be layered by task. For example, intent recognition can be measured by accuracy, summarization by coverage of key points, and code generation by the pass rate of runnable tests.
2. Context window and long-context effectiveness. Context length should not be judged only by the advertised maximum. More importantly, test “effective utilization of long context.” In practice, you should check whether the model can retrieve critical information placed in the middle of an 80,000-character document; when a document exceeds the window, whether it truncates, automatically summarizes, or retrieves segments. For long-document scenarios, retrieval augmentation is often more cost-effective than simply pursuing an ultra-long context window.
3. Latency and throughput. Distinguish time to first token (TTFT) from generation speed (tokens/s). Conversational applications are usually sensitive to first-token latency and often require sub-second responses. Offline batch tasks care more about overall throughput and cost per unit of work. Load testing is mandatory before launch. Observe the inflection point in the latency curve as concurrency increases.
4. Tool calling and structured output. If your system needs agent capabilities, validate the following: parameter extraction accuracy for function calls, stability across multiple tool calls, JSON validity rate, and the ability to continue output after very long outputs are truncated. In engineering practice, JSON output should be validated against a schema with one automatic retry on failure.
5. Cost. Calculate total cost: input tokens + output tokens + cache-hit discounts + additional retrieval and vector-database costs + amplification from retries. Many teams underestimate retry amplification. It is common for the actual bill to be 20%–30% higher than the estimate.
6. Compliance and data boundaries. Clarify whether data can be transferred across borders, whether it can be used for training, log retention periods, and content-safety filtering requirements. In scenarios involving personal information or sensitive business data, compliance is often a veto item and should be evaluated before model quality.
3. Build your own evaluation set: the real starting point
A usable evaluation set does not need to be large. 80–200 high-quality samples are often enough to reveal meaningful differences between models. The construction method:
- Collect real requests. Sample from production logs, support tickets, and user feedback. Cover both high-frequency and long-tail cases.
- Desensitize and annotate. Remove personal information and business identifiers. Annotate reference answers and scoring points.
- Segment. Tag samples by difficulty (easy/medium/hard) and scenario (extraction/summarization/multi-turn Q&A/code).
- Define scoring. Use exact match or rule-based validation for objective tasks. For subjective tasks, use key-point coverage plus two-person cross-scoring to reduce single-reviewer bias.
- Use it as a regression suite. Run the evaluation every time you change models, prompts, or parameters. Record scores together with latency and cost.
A simplified evaluation report structure is shown below:
# eval/report.yamlmodel_id: "<candidate-model-id>"prompt_version: "v3"sampling: { temperature: 0.2, max_tokens: 2048 }metrics:exact_match: 0.87key_point_coverage: 0.91json_valid_rate: 0.99ttft_ms_p95: 780tokens_per_second: 42cost_per_1k_requests: "<measured value>"
Once this table exists, selection changes from “listening to what others say” to “looking at what your own data says.”
4. Unified access layer: make models replaceable
Model selection should not be a one-time action. Chinese models iterate quickly, and pricing changes frequently. The architecture must assume that models will be replaced. The recommended approach is to add a thin unified gateway between the application and the model services.
Responsibilities of the unified gateway:
- Protocol normalization: Translate a unified internal request format into the actual protocols used by different model services. Many cloud model services provide compatible REST interfaces, which keeps integration costs relatively low.
- Routing and gradual rollout: Route traffic to different models based on business labels. Support percentage-based rollouts and one-click rollback.
- Capability abstraction: Abstract capabilities such as “supports tool calling,” “maximum context,” and “supports streaming” as capability labels. Business code only needs to check the labels.
- Observability: Record request IDs, model identifiers, token usage, latency, and retry counts in a unified way.
Example integration shape (generic REST style):
curl -X POST "https://<your-gateway>/v1/chat/completions" \-H "Authorization: Bearer ${MODEL_SERVICE_API_KEY}" \-H "Content-Type: application/json" \-d '{"model": "${MODEL_ID}","messages": [{"role": "user", "content": "Extract the breach clauses from the contract."}],"temperature": 0.2,"stream": false}'
Externalize configuration: Model identifiers, endpoint addresses, and keys should all come from environment variables or a configuration center. No hard-coded values should appear in code.
# config.pyimport osMODEL_CONFIG = {"endpoint": os.environ["MODEL_ENDPOINT"],"api_key": os.environ["MODEL_API_KEY"], # Never commit this to a code repository"model_id": os.environ.get("MODEL_ID", ""),"timeout": 30,"max_retries": 2,}
With this design, switching models only requires changing configuration. It does not require changing business logic or redeploying the application.
5. Side-by-side model comparison: comparison is a means, not the goal
Sending the same request to multiple models and displaying their outputs side by side is an efficient qualitative comparison method. However, methodological correctness matters:
- Control variables. Except for the model itself, prompts, temperature, maximum length, and system instructions must be identical. Otherwise, you are comparing configurations rather than models.
- Blind evaluation. Hide model identities before asking reviewers to score. This can effectively reduce bias caused by brand expectations.
- Quantitative accumulation. Conclusions from side-by-side comparisons should flow back into the scoring table in Section 3 to avoid “choosing by intuition.”
- Focus on failure modes. More decision value often comes from how models fail on difficult cases—hallucination, missing fields, or excessive refusal—than from cases where all models perform well.
6. Production engineering checklist: turn demos into reliable services
Model capability is only the foundation. Production stability depends on engineering safeguards. Before launch, confirm each item below:
| Item | Recommended approach |
|---|---|
| Timeout | Set connection timeout and total timeout separately. For streaming scenarios, set a time-to-first-token timeout. |
| Retries | Retry only for 429/5xx/network errors. Use exponential backoff plus jitter to avoid amplifying failures. |
| Rate limiting | Use a token bucket on the client side and configure quotas on the server side to prevent sudden traffic from exhausting limits. |
| Fallback | Switch to a backup model or rule-based fallback replies when the primary model is unavailable. |
| Caching | Use semantic caching for high-frequency similar requests. This can significantly reduce token consumption. |
| Observability | Record token usage, latency percentiles, and error-code distributions. Configure monitoring and alerts. |
| Version pinning | Record the actual model identifier and version used to prevent silent upgrades from causing quality regressions. |
Regarding free quotas: platform-level free or trial quotas usually come with rate limits, concurrency limits, and availability fluctuations. They are suitable for prototyping and evaluation, but are not recommended for production traffic. Production environments should use formal services with explicit SLAs and quotas.
7. Cost control and capacity planning
Cost optimization usually has the highest return in this order:
- Prompt slimming. Remove redundant examples and repeated constraints. This directly reduces input tokens.
- Semantic caching. High-frequency repeated questions often have a meaningful hit rate.
- Model-tier routing. Send simple tasks to small models or lightweight versions, and escalate complex tasks to flagship models. In practice, this can balance quality and cost.
- Context trimming. Limit the number of retrieved items during retrieval instead of stuffing an entire knowledge base into the prompt.
- Output-length constraints. Explicitly require “output JSON only” or “no more than 200 words” to reduce unnecessary generation.
Capacity planning should be based on peak traffic rather than average traffic, with headroom for bursts. In addition, track “cost per 1,000 requests” as a core operational metric.
8. Security and compliance baseline
- Input side: Desensitize or mask sensitive information before sending it to the model. Add truncation protection for overly long inputs.
- Output side: Connect content-safety filtering. Validate structured output against a schema before storing it.
- Prompt-injection protection: Mark external retrieved content and system instructions as separate sections to prevent document content from being executed as instructions.
- Credential management: Store keys in a secret-management service or environment variables. Rotate them regularly. Assign permissions according to least privilege. Never commit them to code repositories.
- Data retention: Define how long request content in logs should be retained and who can access it, in order to meet audit requirements.
9. Conclusion
Selecting a Chinese LLM is not a single-choice question of “which one to pick.” It is a reusable methodology:
First define constraints (compliance, budget, latency), then build a small evaluation set to measure quality, then use a unified access layer to make models pluggable components, and finally use engineering measures—retries, rate limiting, caching, routing, and observability—to turn them into stable services. Even if models change every six months and prices change monthly, your system can remain stable, portable, and regression-testable.
If you want to reduce the effort of building and operating your own stack, you can also consider managed options, such as model services and supporting capabilities offered by Baidu AI Cloud—for example, model inference, logging, monitoring, and alerting under a unified interface. This lets teams focus on business effectiveness rather than infrastructure.
