Choosing Chinese LLMs: Evaluation Criteria and Production Deployment Practices
Selecting a Chinese large language model requires more than comparing benchmark scores. This guide explains how to define requirements, build an internal evaluation set, design a unified integration layer, route tasks across models, manage cost and latency, and validate security and deployment readiness. It also outlines how to keep model selection decisions current as models and business needs evolve.
1. Why Model Selection Is Getting Harder
Chinese large language models (LLMs) are numerous and evolve rapidly. Even versions from the same provider can differ significantly in code generation, long-context understanding, Chinese-language output, and inference cost.
Developers commonly face three problems:
- A model scores well on public benchmarks but performs inconsistently on their own workloads.
- The same model excels at some tasks and struggles with others.
- Rapid release cycles make selection decisions outdated quickly.
The goal is therefore not to identify a single, universally strongest model. It is to find a sufficiently capable, affordable, and replaceable option for each task within the relevant constraints.
2. Define Requirements Before Comparing Models
Clarify these five categories of constraints before building a shortlist:
- Task type: Question answering, summarization, information extraction, code generation, and agent tool use require different capabilities.
- Context length: Document size and conversation history determine the minimum context window.
- Latency requirements: Real-time, user-facing interactions have very different latency tolerances from offline batch processing.
- Cost budget: For token-based pricing, estimate monthly spending from input and output tokens per request and average daily request volume.
- Compliance requirements: Determine whether data may cross borders, whether private deployment is required, and how long logs may be retained.
Document these constraints in a table to eliminate unsuitable candidates quickly. For example, if data must remain within an internal network, externally hosted services should be excluded.
3. Evaluation: How to Use Public Benchmarks
Public leaderboards provide a useful comparison point, but they have clear limitations. Their task distribution may not match your workload, some test sets have been widely discussed, and small score differences may not be statistically meaningful.
Use benchmarks for initial screening, then make the final decision using an internal evaluation set.
Build an Internal Evaluation Set
- Sample 200–500 real production requests. Include both common scenarios and edge cases.
- Define reference answers or scoring rules for every sample. Compare objective answers directly and use scoring rubrics for open-ended tasks.
- Measure multiple dimensions, not just accuracy:
- Task accuracy: Does the output meet business acceptance criteria?
- Format compliance: Does structured output strictly follow the required field definitions?
- Tool-call success rate: Are arguments complete, and is the call sequence appropriate?
- Refusal and false-refusal rates: How often does the model refuse, and does it incorrectly reject answerable requests?
- Time to first token and total response time.
- Cost per task: Average token consumption and monetary cost for completing a task.
- Run the same samples against every candidate. Record results and conduct a manual review.
- Estimate confidence intervals for key metrics. When differences are not significant, favor the lower-cost, lower-latency option.
Keep prompts fixed during evaluation runs. Because prompts can substantially affect results, give each model a modest amount of prompt tuning before freezing its evaluation configuration. Otherwise, the comparison may be unfair.
4. Integration Architecture: Standardize Interfaces and Isolate Change
Model versions change frequently. Business code should not depend directly on model-specific parameters.
Introduce a unified integration layer between application logic and model services. Its responsibilities should include:
- Protocol adaptation: Convert application requests into the formats required by each model service.
- Model routing: Select a model based on task type.
- Authentication and rate limiting: Manage credentials centrally and assign quotas by caller.
- Timeouts and retries: Set appropriate timeouts and use exponential backoff for retryable errors.
- Degradation and fallback: Switch to a backup model when the primary is unavailable.
- Logging and auditing: Record request metadata and redact sensitive content according to policy.
The following pseudocode illustrates the structure; it is not a complete, runnable implementation:
function call_model(task_type, messages):config = read_routing_config(task_type) # Read from configuration management; do not hard-codefor candidate in [config.primary_model, config.backup_model]:try:result = send_request(endpoint=candidate.endpoint,api_key=get_secret(candidate.secret_name),messages=messages,timeout_seconds=config.timeout,)if validate(result):record_metrics(candidate.name, elapsed_time, usage)return resultcatch RetryableError:record_alert(candidate.name)continuereturn fallback_response
Retrieve credentials from a secrets management service or environment variables. Never include them in source code or logs.
5. Multi-Model Routing Strategies
In practice, a single model rarely serves every task equally well. A common approach is to route tasks by complexity and importance:
- Use low-cost models for simple classification and format conversion, prioritizing throughput.
- Use more capable models for complex reasoning and long-document analysis, accepting higher costs.
- Use thoroughly validated models for critical user-facing responses. Introduce new models through a small-percentage canary rollout.
Define stop conditions before a canary release. For example, automatically revert if task accuracy falls beyond a specified threshold or if error and timeout rates increase substantially.
Manage routing rules separately from application releases so that changes can be rolled back without redeploying the application.
6. Managing Cost and Performance
With token-based billing, prompt length and output length are major cost drivers. The following measures can help:
- Streamline system prompts. Remove repeated instructions and organize fixed rules into reusable templates.
- Cache results for repeated requests. Build cache keys from the task type, normalized input, and model version.
- Limit maximum output length. Also state the expected response length in the prompt.
- Batch non-real-time tasks. This can reduce request frequency and queuing overhead.
- Set a daily budget for each caller. Trigger an alert or switch to a lower-cost model when the budget is exceeded.
For performance monitoring, track time to first token separately from generation speed. Streaming responses can substantially improve perceived latency, but the frontend must handle interruptions and reconnections correctly.
7. Security and Compliance
Security risks in LLM applications are concentrated in three areas:
- Data leakage: User input may contain personal information or business secrets. Redact sensitive data before sending it, and verify the provider’s policies on data retention and use for training.
- Prompt injection: Retrieved web pages, documents, and tool responses may contain text intended to redirect the model’s behavior. Treat this content as data rather than instructions, and restrict the tools the model can access.
- Output risks: Models may generate incorrect facts or inappropriate content. Add rule-based validation for critical use cases and human review where necessary.
Apply the principle of least privilege. Models should only access tools required for the task, and calls that perform write operations should require additional confirmation.
8. Pre-Launch Checklist
Verify each item before deployment:
- The internal evaluation set covers common scenarios and edge cases, with explicit acceptance criteria.
- Business code does not directly depend on model-specific parameters; all calls pass through the integration layer.
- Primary and backup models have completed the same evaluation, and backup availability has been verified.
- Storage and access controls for credentials, logs, and caches meet compliance requirements.
- Timeout, retry, degradation, and rollback behavior have been tested through fault injection.
- Monitoring includes sampled accuracy checks, latency, error rate, and cost per task.
- The release documentation includes a canary rollout plan and explicit stop conditions.
9. Keep Selection Decisions Current
Model selection is not a one-time decision. Rerun candidate models against the same evaluation set after major model releases or on a regular monthly schedule, and update the selection record.
Refresh the evaluation set as business needs change. Without new samples, it will gradually stop representing the workload it is meant to measure.
