Model Selection Methodology and Engineering Deployment Guide for Chinese LLMs

    This article presents a practical framework for selecting and deploying Chinese large language models in production. It explains why public leaderboards are only an initial filter, how to build reproducible business evaluation sets, how to calculate unit task cost instead of relying on token price alone, and how to assess deployment and compliance constraints early. It also covers multi-model routing, gradual rollout, fallback strategies, monitoring, and a final selection checklist to help teams balance quality, cost, latency, and compliance.

    Model Selection Methodology and Engineering Deployment Guide for Chinese LLMs

    1. The Essence of Selection: Match, Not Maximum Strength

    When faced with multiple Chinese LLMs, many teams first look at public leaderboard rankings, parameter scale, or unit price. In real engineering, however, model selection is not a simple “leaderboard purchase.” It is a systematic decision centered on business goals, engineering constraints, and long-term maintenance cost.

    A model that performs well on general benchmarks is not necessarily suitable for your business corpus, output format, latency requirements, and cost budget. For example:

    • Customer service summarization tasks care more about stability, factual consistency, and low latency.
    • Code assistance tasks care more about context length, structured output, and tool-calling capability.
    • Long-document analysis tasks care more about long-context recall, citation grounding, and segmentation handling.
    • Batch offline processing tasks care more about cost per token and throughput.
    • Scenarios involving sensitive data care more about private deployment, audit logs, permission isolation, and data redaction.

    Therefore, the first step in model selection is not to collect a list of models, but to break business requirements into evaluable metrics. It is recommended to build candidate model profiles across the following dimensions:

    1. Task type: general conversation, information extraction, summarization, code generation, multimodal understanding, tool calling, and so on.
    2. Quality requirements: accuracy, recall, factual consistency, format-following rate, and refusal policy.
    3. Performance constraints: time to first token, output throughput, concurrency capability, and long-context stability.
    4. Cost structure: input/output token consumption, retry rate, cache hit rate, and batch processing capability.
    5. Engineering integration: standardized APIs, streaming output, structured output, function calling, logging, and monitoring.
    6. Deployment and compliance: public APIs, dedicated instances, private deployment, data retention, and audit requirements.
    7. Version stability: model release frequency, behavior drift risk, and rollback mechanisms.

    Only when both “capability” and “constraint” are included in the evaluation does model selection become meaningful for engineering.

    2. Public Leaderboards Are for Initial Screening Only

    The value of public benchmarks is to quickly narrow the candidate pool. They help teams understand the general capability distribution of different models, such as language understanding, code generation, mathematical reasoning, and long-context handling. However, leaderboards have natural limitations:

    • Limited scenario coverage: benchmark questions are usually general and rarely cover private enterprise business corpora.
    • Data contamination risk: public questions may enter training data, inflating scores.
    • Different scoring standards: automatic metrics and human preferences do not always align.
    • Version drift: after model updates, historical rankings may no longer apply.
    • Missing cost and latency data: leaderboards rarely reflect real concurrency, latency, and cost per task.

    A more reasonable approach is to use public leaderboards as a “candidate pool filter,” retaining only a few models that meet basic capability requirements, and then move to business evaluation. Do not integrate a model across the board just because it ranks high on one leaderboard, and do not exclude a model too early just because it is absent from one leaderboard.

    3. Build a Reproducible Business Evaluation Set

    The business evaluation set is a core asset for model selection. It is recommended to sample redacted data from real business logs and build an evaluation set that does not need to be large but has sufficient coverage. It can generally be divided into:

    • Core set: covers high-frequency tasks and critical business paths, used for main model comparison.
    • Boundary set: covers long-tail issues, ambiguous inputs, abnormal formats, and adversarial expressions.
    • Format set: verifies structured output, field completeness, and citation consistency.
    • Safety set: verifies sensitive information handling, refusal policy, and privilege escalation risk.
    • Cost set: measures token consumption and retry counts for completing similar tasks.

    The evaluation process should use a three-layer mechanism: “automatic metrics + human scoring + online canary rollout.” Automatic metrics support fast regression checks, human scoring judges semantic quality and usability, and online canary rollout verifies stability under real traffic.

    The following pseudocode shows a generic evaluation configuration. It focuses on evaluation dimensions and weights rather than binding to a specific API:

    1. evaluation_config = {
    2. "candidates": ["model_a", "model_b", "model_c"],
    3. "datasets": {
    4. "core": 0.40,
    5. "boundary": 0.20,
    6. "format": 0.15,
    7. "safety": 0.15,
    8. "cost": 0.10
    9. },
    10. "metrics": {
    11. "quality": ["accuracy", "consistency", "format_following"],
    12. "performance": ["first_token_latency", "throughput", "timeout_rate"],
    13. "cost": ["input_tokens", "output_tokens", "retry_rate"]
    14. },
    15. "decision_rule": "weighted_score >= threshold and safety == pass"
    16. }

    During evaluation, fix the input samples, prompt templates, and output parsing rules, and record model versions and parameter settings. Otherwise, results from different rounds are not comparable, and model selection can easily become a matter of subjective impression.

    4. Cost Accounting: Do Not Look Only at Unit Price

    Model unit price is only one part of cost. Real cost also includes retries, invalid output, redundant context, human review, cache misses, and operational investment. A model with a slightly higher unit price but a higher first-pass success rate may be more economical than a low-cost model that requires frequent retries.

    It is recommended to compare models using “cost per task” rather than “price per token”:

    1. unit_task_cost =
    2. average_input_tokens * input_price
    3. + average_output_tokens * output_price
    4. + retry_rate * retry_cost
    5. + human_review_rate * review_cost
    6. + infrastructure_cost_per_request

    Capacity planning should also be considered. For online services, pay attention to queueing latency and timeout rate under peak concurrency. For offline batch processing, higher latency may be acceptable, but batch scheduling, caching, and result reuse should be optimized. Common optimization methods include:

    • Cache results or use semantic caching for repeated questions.
    • Summarize long documents in segments and then aggregate, reducing unnecessary context.
    • Apply stricter output constraints for structured tasks.
    • Use smaller or distilled models for low-risk tasks.
    • Keep stronger models as a fallback for high-value tasks.

    5. Evaluate Deployment Models and Compliance Constraints Early

    Model selection cannot wait until launch to consider deployment. Different businesses have different requirements for data retention, network isolation, audit logs, and access control. Common deployment models include standardized API calls, dedicated instances, and private deployment. Teams should confirm in advance:

    1. Whether data is allowed to leave the self-owned environment.
    2. Whether request-level auditing and content retention are required.
    3. Whether quotas and permission isolation are needed by tenant, department, or project.
    4. Whether log redaction, access control, and security auditing are required.
    5. Whether integration is needed with existing monitoring and alerting, object storage, message queues, and other general cloud capabilities.

    If you already have Baidu AI Cloud resources, you can reuse its general compute, object storage, logging, and monitoring capabilities, but model selection should still be based on business evaluation results. The earlier the deployment model is determined, the more accurate the subsequent evaluation metrics and cost model will be.

    6. Implementation Strategy: Multi-Model Routing and Canary Fallback

    In most production systems, a single model can hardly satisfy all scenarios at the same time. A more reliable strategy is “multi-model routing + tiered fallback”:

    • Route simple tasks to low-cost models.
    • Route high-value tasks to high-quality models.
    • Route ultra-long-context tasks to models with more stable long-context capability.
    • Automatically fall back to a backup model when the primary model times out or fails format requirements.
    • Roll out new models with small traffic first, then gradually increase the percentage.

    An illustrative routing logic is as follows:

    1. function route_request(request):
    2. if request.task_type == "simple_extraction":
    3. return call_model("low_cost_model")
    4. if request.task_type == "complex_reasoning":
    5. return call_model("high_quality_model")
    6. if request.context_length > long_context_threshold:
    7. return call_model("long_context_model")
    8. if request.priority == "high":
    9. return call_model("primary_model", fallback="backup_model")
    10. return call_model("default_model")

    During canary rollout, monitor quality, latency, error rate, retry rate, unit task cost, and user feedback. Do not look only at average metrics; also examine percentiles and long-tail failure samples. For format-sensitive tasks, focus on parsing failure rate and field missing rate. For open-ended generation tasks, focus on factual consistency and human adoption rate.

    7. Selection Checklist and Common Pitfalls

    Before implementation, use the following checklist for final review:

    • Has a redacted business evaluation set been established?
    • Have prompt templates and parsing rules been fixed?
    • Has unit task cost been compared instead of token unit price alone?
    • Have peak concurrency and long-tail latency been verified?
    • Have structured output and tool-calling stability been evaluated?
    • Have data compliance and deployment boundaries been clarified?
    • Are model version rollback and canary mechanisms in place?
    • Have continuous evaluation and monitoring alerts been established?

    Common pitfalls include ranking-only decisions, parameter-only decisions, price-only decisions, ignoring version drift, ignoring failure samples, and ignoring long-term maintenance cost. A more reasonable strategy is to treat model selection as continuous engineering: validate on a small scale first, then roll out with canary traffic, and finally iterate routing and evaluation sets based on monitoring data.

    8. Conclusion

    Model selection for Chinese LLMs should not remain a discussion of “which model is stronger,” but should return to “which model fits the current business constraints.” Public leaderboards are for initial screening, business evaluation sets are for decision-making, cost models are for validation, and canary monitoring is for continuous optimization. By establishing reproducible evaluation processes, multi-model routing, and fallback mechanisms, teams can achieve a more robust balance among quality, cost, latency, and compliance. The ultimate goal is not to select an “optimal model” once, but to build a model selection and governance system that can evolve continuously with business changes.