Chinese LLM Selection: Evaluation, Integration, and Production Deployment

    Many Chinese large language models have recently become available through managed hosted inference services. The developer question has changed from “can we use them?” to “which one should we choose, how should we integrate it, and how do we keep it under control?” Public leaderboards and one-time demos are not enough. This article presents a reusable framework for selecting Chinese LLMs by evaluating model capability, inference service quality, and engineering infrastructure together. It covers task profiling, acceptance criteria, business-specific evaluation sets, access modes, unified inference gateway design, cost and latency optimization, observability, version governance, security/compliance, and a practical proof-of-concept flow.

    Several Chinese LLMs have been released in a short period, and many are now entering managed hosted inference services. For developers, the real question is no longer whether these models are usable. It is how to select them, how to integrate them, and how to govern them continuously.

    A public leaderboard or a single hands-on demo can easily lead to a one-sided conclusion. Benchmark tasks often do not match real business tasks. Model version updates can change output distributions. Integration choices affect latency, cost, and stability. Therefore, model selection should be treated as a combined decision across model capability, inference service quality, and engineering infrastructure, rather than a point-by-point comparison of individual model names.

    Why Selection Is a Systems Engineering Problem

    A common mistake is to treat “model capability” as equivalent to “business usability.” A model that performs well in general Q&A may not be stable for structured business output, long-document retrieval-augmented generation, tool calling, code generation, or multi-turn task planning. Even if the model capability is sufficient, a tightly rate-limited inference service, unstable time-to-first-token, limited context length, or uncontrollable version upgrades can directly affect the online experience.

    A more practical approach is to build a three-layer evaluation:

    1. Model layer: task accuracy, instruction following, robustness, long-context understanding, structured output, tool calling, multilingual ability, and safety boundaries.
    2. Service layer: availability, rate limits and quotas, concurrency, streaming output, error-code semantics, version management, data retention policy, and billing transparency.
    3. Engineering layer: unified access, fallback routing, caching, monitoring and alerting, evaluation regression, canary release, access auditing, and cost governance.

    Only when all three layers meet business acceptance criteria should a model enter production. Otherwise, even a strong model can be dragged down by latency, cost, or compliance issues during engineering deployment.

    Start by Defining the Task and Acceptance Criteria

    The first step is not to collect a list of candidate models. It is to decompose business tasks into evaluable inputs and outputs. A task profile should include these dimensions:

    • Task type: general conversation, knowledge Q&A, summarization, classification and labeling, code generation, long-document analysis, multimodal understanding, agent tool calling, and so on.
    • Input characteristics: average and maximum context length, whether the input contains tables or code, whether it is multilingual, whether it contains sensitive information, and whether multi-turn state needs to be maintained.
    • Output constraints: free-form text, fixed fields, structured data, cited sources, explanatory reasoning, function-call parameters, and so on.
    • Quality requirements: accuracy, recall, format validity, acceptable hallucination rate, refusal behavior, and human review ratio.
    • Service requirements: time-to-first-token, end-to-end latency, concurrency, peak traffic, availability target, and failure degradation strategy.
    • Compliance requirements: data classification, encryption in transit and at rest, access control, audit trails, data retention and deletion, and geographic restrictions.

    Acceptance criteria should be as quantifiable as possible. For structured output tasks, define field completeness, type validity, and business validation pass rate. For knowledge Q&A, define answer accuracy, citation traceability, and refusal accuracy. For code tasks, define compilation pass rate, unit test pass rate, and human readability score. These indicators are model-agnostic, so they support horizontal comparison and long-term regression.

    Capability Evaluation: From Public Benchmarks to Business Evaluation Sets

    Public leaderboards are useful for initial screening, but they should not be the final decision basis. Benchmark tasks differ from real business distributions. Prompts and decoding parameters significantly affect results. Model scores may drift after version updates. Some benchmark data may also carry contamination risk.

    A recommended process is:

    1. Use public leaderboards for initial screening.
    2. Re-test candidates with business-specific evaluation sets.
    3. Run blind human evaluation to reduce brand bias.

    A business evaluation set should cover real distributions and boundary scenarios:

    • Real sampling set: stratify historical business requests and cover both high-frequency and long-tail tasks.
    • Adversarial and boundary set: ambiguous input, overly long input, missing fields, malformed formats, mixed languages, and sensitive requests.
    • Regression set: fix a representative sample set and run it after every model or prompt change.
    • Blind human review set: hide model identity and let domain experts score outputs using a unified rubric.

    Metrics should be task-specific, not a single total score. Record accuracy, format validity, citation consistency, refusal accuracy, average output length, and failure-type distribution. For tool-calling scenarios, also evaluate parameter generation correctness, call-order plausibility, and exception recovery. Keep the original input, output, parameters, model version, and scoring records for later review and traceability.

    Access Modes: Managed APIs, Dedicated Instances, and Private Deployment

    Chinese LLMs are typically accessed in three forms, and each form fits a different stage:

    • Managed hosted inference APIs: low integration cost, suitable for rapid validation, low-frequency workloads, and elastic traffic. Pay special attention to rate limits, quotas, error semantics, version policy, and data retention terms.
    • Dedicated instances or provisioned throughput: suitable for stable traffic and workloads that require isolation and predictable performance. Evaluate scaling cycle, concurrency ceiling, cold start, monitoring indicators, and operational boundaries.
    • Private or hybrid deployment: suitable for data-sensitive, network-isolated, or strongly regulated scenarios. Evaluate hardware resources, deployment complexity, upgrade and maintenance, model licensing, and security hardening costs.

    Regardless of the access mode, add a unified inference gateway between the business code and the model service. The gateway handles authentication, rate limiting, parameter normalization, model routing, fallback, log desensitization, and metric collection. This reduces vendor lock-in risk.

    On the infrastructure layer, general cloud engineering capabilities can be used to build this unified access and observability layer. For example, Baidu AI Cloud provides managed inference hosting, container platform, logging, and monitoring/alerting capabilities. These are generic cloud capabilities; the specific model and service choices should still be evaluated independently according to business requirements and compliance rules.

    Unified Inference Gateway Design Example

    The value of a unified gateway is not simply “connecting several models.” It is to converge model differences into one manageable interface. The pseudocode below is illustrative and is not tied to any specific API or product.

    1. function handle_request(task, tenant, prompt, context):
    2. route = select_route(
    3. task_type = task.type,
    4. tenant_policy = tenant.policy,
    5. health = health_registry.snapshot(),
    6. cost_budget = tenant.cost_budget,
    7. latency_slo = task.latency_slo
    8. )
    9. request = {
    10. model: route.primary_model,
    11. prompt: render_prompt(task, context),
    12. params: {
    13. temperature: task.temperature,
    14. max_tokens: task.max_tokens,
    15. stream: task.stream_enabled
    16. },
    17. timeout_ms: task.timeout_ms,
    18. trace_id: new_trace_id()
    19. }
    20. response = inference_gateway.invoke(request)
    21. record_metrics(request, response)
    22. if response.ok and quality_guard(task, response):
    23. return response
    24. fallback = select_fallback(route, reason = response.error_type)
    25. if fallback.exists:
    26. fallback_request = build_request(fallback, task, context)
    27. fallback_response = inference_gateway.invoke(fallback_request)
    28. record_metrics(fallback_request, fallback_response)
    29. if fallback_response.ok and quality_guard(task, fallback_response):
    30. return fallback_response
    31. return safe_degrade(task, reason = "inference_unavailable")

    This pseudocode illustrates several engineering principles:

    • Routing considers task type, tenant policy, health status, cost budget, and latency SLO at the same time.
    • Every request carries a trace identifier.
    • Quality guarding checks not only whether the API call succeeded, but also format validity, business validation, and security policy.
    • Fallback strategies are defined in advance, rather than assembled during an incident.

    Cost and Performance: The Full Chain, Not Just Unit Price

    Model unit price is only one part of cost. A production environment also needs to account for input/output length distribution, retry rate, cache hit rate, human review cost, evaluation regression cost, and business loss caused by degradation.

    Performance should not be measured by average latency alone. Also observe time-to-first-token, P95/P99 end-to-end latency, throughput, error rate, and timeout rate.

    Optimization directions include:

    • Long/short context routing: use lighter routes for short tasks, and only enable long-context capability for complex tasks.
    • Caching and reuse: cache results or intermediate representations for stable knowledge Q&A and templated summarization to reduce repeated calls.
    • Batch and asynchronous processing: use batch processing for offline tasks, and keep streaming output for online tasks to improve perceived latency.
    • Small-model routing and degradation: route simple classification or extraction tasks to smaller models, and reserve stronger models for complex reasoning.
    • Retry and circuit breaking: distinguish retryable errors from non-retryable errors to avoid amplifying failures.
    • Budget and tenant isolation: set budgets and quotas by tenant or business line at the gateway layer to prevent one workload from consuming all resources.

    Observability, Quality Loop, and Version Governance

    After launch, model output can drift as input distribution, prompts, parameters, and model versions change. A quality loop is therefore necessary:

    1. Logging and tracing: record request identifier, model version, parameters, latency, error type, and desensitized input/output summaries.
    2. Metric system: API success rate, time-to-first-token, P95/P99 latency, format validity rate, refusal rate, human feedback rate, and fallback rate.
    3. Evaluation regression: run a fixed regression set and compare against baseline after every model upgrade, prompt change, or parameter change.
    4. Canary release: validate a new model or prompt on a small traffic slice first, observe quality and performance metrics, then gradually expand.
    5. Version management: version the model, prompt, routing policy, and parameter templates so they can be rolled back quickly.
    6. Human feedback loop: feed expert annotations, user feedback, and failure cases back into the evaluation set to continuously cover long-tail scenarios.

    Security, Compliance, and Data Governance

    Security and compliance should be considered before selection. At minimum, check the following:

    • Data classification covers inputs and outputs.
    • Encryption is used for data in transit and at rest.
    • Access control supports least privilege.
    • Audit logs are traceable.
    • Sensitive information is desensitized.
    • Data retention and deletion policies are clear.
    • Content safety policies can be configured.
    • Vendor terms state whether business data can be used for training or secondary use.

    For scenarios involving personal privacy, trade secrets, or strongly regulated data, prioritize isolated deployment and data-staying-within-boundary solutions.

    Decision Matrix and PoC Process

    Use a decision matrix to turn subjective judgment into a discussable weighted score. Example dimensions include:

    • Task fit
    • Output stability
    • Service availability
    • Integration compatibility
    • Performance and cost
    • Observability capability
    • Compliance and security
    • Version governance
    • Ecosystem support

    Set weights and acceptance thresholds for each dimension. First filter by thresholds, then compare by composite score.

    A proof of concept can proceed as follows:

    1. Define two or three core tasks and their acceptance metrics.
    2. Prepare real sampling sets, boundary sets, and regression sets.
    3. Blind-test candidate models using the same prompt template and parameters.
    4. Integrate the unified gateway and stress-test latency, concurrency, rate limiting, and error handling.
    5. Run a small-traffic canary and observe quality, cost, and stability.
    6. Produce a selection report that clearly states the primary route, fallback route, and future governance plan.

    Common Pitfalls and Practical Advice

    Common pitfalls include relying only on public leaderboards, treating one demo as a systematic evaluation, ignoring the impact of prompts and parameters, having no fallback route, failing to record model versions, equating API availability with business availability, and ignoring long-tail inputs and human review cost.

    For implementation, keep three principles:

    1. Build business evaluation sets first.
    2. Use a unified gateway to isolate model differences.
    3. Continuously govern versions and metrics.

    Chinese LLM selection is not a one-time ranking. It is an engineering decision that should be continuously calibrated as business needs evolve.