Chinese LLM Selection as an Engineering Decision: From Evaluation to Unified Access

    As more Chinese LLMs become available, teams need a repeatable way to choose models without coupling business code to any single provider. This article explains how to define scenario constraints, build a layered evaluation process across offline tests, shadow traffic, and canary releases, and isolate model differences behind a unified gateway. It also covers credentials, rate limiting, caching, observability, security and compliance, data governance, and a practical rollout roadmap for a sustainable multi-model governance system.

    1. Selection Is Not a Leaderboard Purchase—It Is an Engineering Decision

    As the number of Chinese LLMs grows, the most common mistake teams make is treating leaderboard rankings as the basis for model selection. Leaderboards are usually designed for general tasks and do not match the data distribution, prompt style, or output-format requirements of real business workloads. The same model can also behave differently across versions, parameter settings, and context lengths. More importantly, production launch requires attention to latency, concurrency, cost, stability, data compliance, and observability—metrics that a single leaderboard can hardly cover.

    Model selection should therefore be treated as a continuous engineering governance problem: first define scenarios and constraints, then build a quantifiable evaluation system, and finally use a unified access layer to isolate model differences from business code. In this way, even when model versions are updated or vendors change, the business system does not need major modifications.

    2. Define Scenarios and Constraints Before Discussing Model Capabilities

    Different business scenarios have very different requirements for models. Customer service Q&A focuses more on accuracy, refusal rate, and response latency; document summarization focuses more on long-context coverage and faithfulness; structured extraction focuses more on format compliance and field completeness; code assistance focuses more on executability and context understanding; and content generation focuses more on style stability and safety filtering. Before selecting a model, teams should break down the scenarios clearly and define constraints for each one.

    It is recommended to build a scenario profile from the following dimensions:

    • Task type: Q&A, summarization, extraction, generation, multi-turn dialogue, tool calling, and so on.
    • Input/output scale: average and peak context length, output length, and whether ultra-long documents are involved.
    • Performance requirements: first-token latency, end-to-end latency, concurrency, and timeout tolerance.
    • Quality requirements: accuracy, recall, format compliance rate, hallucination rate, and refusal policy.
    • Cost constraints: cost per request, monthly budget, and whether batch processing is allowed.
    • Data sensitivity: whether personal information is involved, whether data can leave the region, and whether log retention is allowed.
    • Engineering constraints: deployment form, network isolation, audit requirements, and failure degradation strategy.

    Only after these constraints are clearly documented will subsequent evaluation avoid becoming “scoring by feel.”

    3. Build a Layered Evaluation System: Offline, Shadow, and Canary

    The evaluation system should be divided into three layers. The first layer is offline evaluation: sample from real business traffic to build an evaluation set that covers high-frequency questions, long-tail questions, boundary inputs, and adversarial inputs. Each sample should have a standard answer or scoring rule, and the metrics should include accuracy, field completeness rate, format compliance rate, hallucination rate, refusal appropriateness, and stability. The evaluation set should not be frozen after it is built once; it should be continuously expanded as the business changes.

    The second layer is shadow traffic: copy real online requests to candidate models without directly affecting users, and record only outputs and metrics. Shadow traffic can expose input-distribution issues that offline evaluation struggles to cover, and it can also reveal latency and error rates. The third layer is small-traffic canary release: enable candidate models for a controlled percentage of users and monitor adoption rate, human correction rate, end-to-end latency, error classification, and cost per request. During the canary period, a fast rollback capability should be retained.

    Evaluation should also pay attention to prompt fairness: the same scenario should use the same prompt template and output constraints to avoid distorted conclusions caused by prompt differences. Every time a model version changes, the regression evaluation set should be run again to prevent “version upgrades” from causing “business metric declines.”

    4. Unified Access Layer: Confine Model Differences to Adapters

    The biggest risk in multi-model selection is binding business code directly to a specific model API. It is recommended to add a unified model gateway between business services and model services. The gateway handles authentication, routing, rate limiting, retries, degradation, caching, logging, and metering. Business services depend only on a unified request/response structure, while model differences are absorbed by adapters.

    Pseudocode example:

    1. Interface ChatModel:
    2. chat(messages, options) -> response
    3. Model registry ModelSpec:
    4. name, endpoint, capability, maxContext, costLevel
    5. Route select(scene, inputTokens):
    6. candidates = filter models where maxContext >= inputTokens
    7. return candidate with highest score by scene weights
    8. Gateway handle(request):
    9. spec = select(request.scene, request.inputTokens)
    10. response = call spec.endpoint; on failure retry in fallback order
    11. record trace ID, token count, latency, error type
    12. return uniformly formatted response

    Configuration example:

    1. route:
    2. default: model-a
    3. fallback: [model-b, model-c]
    4. limits:
    5. qps: 20
    6. timeout_ms: 30000
    7. guard:
    8. max_input_tokens: 32000
    9. sensitive_data: mask

    The benefits of a unified access layer are direct: adding or replacing a model only requires updating the registry and adapter; routing can be performed by scenario, for example, high-accuracy scenarios use the primary model while low-latency scenarios use a lightweight model; automatic degradation is possible during failures; and all calls have unified logs and metering, making cost attribution easier.

    5. Engineering Key Points: Credentials, Rate Limiting, Caching, and Observability

    When implementing the unified access layer, focus on the following issues:

    1. Credential management: Access credentials should be centrally managed, assigned with least privilege, rotated regularly, and never hard-coded in frontends or code repositories. Different environments should use different credentials, and call sources should be recorded.
    2. Rate limiting and quotas: Set rate limits and monthly quotas by tenant, scenario, and model. Use token bucket or leaky bucket strategies to prevent burst traffic from overwhelming upstream services.
    3. Retries and circuit breaking: Retry only retryable errors with a limited number of attempts, and set timeout and circuit-breaking thresholds. Retries can amplify costs and must be combined with idempotency design.
    4. Caching strategy: Exact caching can be used for repeated questions; semantic caching should be used cautiously and must handle privacy and answer freshness. Cache hit rate should be included in monitoring.
    5. Observability: Record trace ID, scenario, model ID, input and output token counts, first-token latency, end-to-end latency, error classification, and degradation count. Logs must be masked, and retention policies must comply with regulatory requirements.
    6. Cost governance: Distinguish input and output token costs, prioritize small models for simple tasks, truncate or summarize long documents first, and make batch tasks asynchronous. Perform scenario-level cost attribution monthly.

    6. Security, Compliance, and Data Governance Cannot Be Deferred

    Model selection must consider security and compliance at the same time. First, classify data to clarify which data can enter model calls, which must be masked, and which is prohibited from being sent externally. Second, set up content safety filtering and output validation, and configure whitelists and parameter validation for tool calls to prevent prompt injection from causing unauthorized operations. For auditing, necessary call records and policy change records should be retained, but sensitive information must not be leaked in logs.

    For multi-model access, the data usage boundaries and retention policies for each scenario must also be clarified. Security policies should be implemented centrally at the gateway layer as much as possible to avoid duplicate construction by each business team.

    7. Use General Cloud Capabilities to Support the Selection Platform

    In a general cloud environment such as Baidu AI Cloud, container platforms can be used to deploy the model gateway, object storage can be used to manage evaluation sets and masked logs, and log services, monitoring and alerting, and key management capabilities can form a closed loop for observability and security. In this way, selection, evaluation, routing, and governance can become platform capabilities rather than being scattered across business teams. It is important to emphasize that cloud platforms provide general infrastructure capabilities; specific model choices should still be based on business evaluation data.

    8. Implementation Roadmap

    It is recommended to proceed according to the following cadence:

    1. Week 1: List scenarios and constraints, and define evaluation metrics.
    2. Week 2: Build the first evaluation set, covering high-frequency and long-tail samples.
    3. Weeks 3–4: Build a minimum viable version of the unified gateway and connect two to three candidate models.
    4. Week 5: Complete offline evaluation and shadow traffic comparison, and produce an evaluation report.
    5. Week 6: Select a low-risk scenario for a small-traffic canary release, and observe adoption rate and latency.
    6. Week 7: Determine primary/backup routing and degradation strategies based on data, and establish a regression evaluation mechanism.
    7. Every subsequent quarter: Re-run the evaluation set and update the model registry and routing weights.

    9. Conclusion

    Chinese LLM selection should not stop at the question of “which leaderboard ranks higher.” A more reliable approach is to start from business scenarios, use a self-built evaluation set as the basis, use a unified access layer as the engineering anchor, and use canary releases and observability as safeguards to form a closed loop of “evaluation—access—canary—regression.” What this selects is not an isolated model, but a sustainable and evolving multi-model governance system.