LLM Selection for Chinese Models: A Developer's Evaluation Framework and Practice

    This article explains why "Top 4" leaderboard rankings are not a sound basis for choosing among Chinese LLMs. It shows how to turn business requirements into measurable quality, engineering, cost, and compliance metrics. It then covers five evaluation dimensions (task fit and knowledge coverage, context handling and structured output, reasoning and tool calling, deployment and cost structure, and security, compliance, and observability). Readers also get a four-stage workflow (capability screening, POC comparison, canary release, and continuous regression) with tiered-routing pseudocode, plus guidance on caching, RAG, fine-tuning, and prompt engineering. It closes with common pitfalls to avoid. The goal is a traceable evidence chain built on cost, compliance, and operational constraints, rather than a static ranking.

    LLM Selection for Chinese Models: A Developer’s Evaluation Framework and Practice

    Why Leaderboard Rankings Cannot Be Your Selection Decision

    Asking “which Chinese LLMs are the strongest” is essentially a multi-objective optimization problem: quality, latency, throughput, cost, stability, compliance, and maintainability often constrain one another. From an engineering perspective, a single overall ranking is not a useful answer. You first need to define your constraints and objective functions.

    • Task distribution drives performance. Summarization, code completion, structured extraction, long-document QA, and multi-turn tool calling place very different demands on a model. Strong open-ended QA performance does not mean a model is equally reliable at strict JSON output, function-argument generation, or understanding domain terminology.
    • Benchmarks can be misleading. Benchmark contamination, version drift, and prompt differences can all skew results. A model that leads one evaluation may simply have training data that overlaps with the benchmark, or a prompt template that suits it better.
    • Production adds constraints beyond quality. You also need to account for cost per request, peak concurrency, failover and graceful degradation, log auditing, and data boundaries.

    A more defensible approach replaces the “Top 4” question with a selection process: candidate capability tiers → scenario-based evaluation → canary validation.

    Before You Evaluate: Turn Business Requirements into Measurable Metrics

    The first step is not to gather a list of models but to establish a requirements baseline. Break the workload into tasks that can be evaluated independently, such as information extraction, text classification, question answering, summarization, code generation, and tool calling. Define metrics for each task type:

    • Quality metrics: accuracy, recall, valid structured-output rate, citation coverage, and blind human-evaluation pass rate.
    • Engineering metrics: time to first token, end-to-end latency, throughput, error rate, timeout rate, and behavior under rate limiting.
    • Cost metrics: go beyond price per call. Measure the total cost of ownership of completing one business task, including retries, caching, retrieval, human review, and operations.
    • Compliance metrics: data masking, log retention, access control, content safety, and regional deployment requirements.

    For experiment design:

    • Fix the test set and prompt templates.
    • Keep a baseline model for comparison.
    • Use blind evaluation and statistical confidence intervals.
    • Avoid drawing conclusions from a small number of samples.

    For business-critical use cases, build a regression set. Whenever a model version or prompt changes, run it automatically and record changes in quality, latency, and cost. This creates a traceable evidence chain for selection decisions.

    Core Evaluation Dimensions: From “Capabilities” to Verifiable Items

    Task Fit and Knowledge Coverage

    Evaluate with real business samples, not just public benchmark questions. Public questions work for initial screening, but final validation should use a small-scale test on anonymized business data.

    Look at how well the model handles:

    • Domain terminology, abbreviations, tables, code, and structured fields.
    • Failure modes such as over-generation, missing fields, and format drift.

    Knowledge coverage is not the same as parameter count. Data quality, training data mix, and post-training strategy matter just as much.

    Context Handling and Structured Output

    • Long documents: test cross-passage referencing, information retrieval, robustness to distractors, and summary consistency.
    • Structured output: test JSON validity, field completeness, adherence to enum values, and stability when input is malformed.

    Don’t just ask “Can it read long text?” Ask instead: “At the target length and under noisy conditions, does it reliably recall the key information?”

    Reasoning and Tool Calling

    For tasks that require multi-step reasoning or tool orchestration, test whether the model can:

    • Break the task into steps
    • Select the right tools
    • Generate valid arguments
    • Handle tool responses and retry after failures

    A single successful call is not enough. Also measure argument error rate, repeated or looping calls, timeout handling, and explainability. For high-risk operations, add human confirmation and permission checks. Never execute model output as instructions directly.

    Deployment Model and Cost Structure

    Candidates may include:

    • Hosted model services
    • Private (on-premises or self-hosted) deployment
    • A hybrid of both

    In your evaluation:

    • Separate training cost from inference cost.
    • Separate peak from average load.
    • Calculate the effective cost after accounting for cache hits.

    For high-concurrency scenarios, evaluate:

    • Batching
    • Streaming output
    • Concurrency limits
    • Degradation strategies

    For sensitive-data scenarios, evaluate:

    • Network boundaries
    • Storage encryption
    • Audit capabilities

    You can assess cloud infrastructure by generic categories such as object storage, vector search, message queues, logging, and monitoring and alerting. Avoid binding specific vendor brands into the core of your architecture.

    Security, Compliance, and Observability

    Enterprise applications should log:

    • Request summaries
    • Model versions
    • Prompt template versions
    • Retrieved content
    • Model outputs
    • Human feedback

    They should also enforce sensitive-data filtering, protection against unauthorized access, and content safety policies.

    At minimum, observability should cover:

    • Call volume
    • Latency percentiles
    • Error codes
    • Retry rate
    • Cache hit rate
    • Cost trends
    • Quality spot-check results

    Selection Workflow: Shortlist, POC, Then Canary Release

    A four-stage process works well:

    1. Capability screening. For each task type, shortlist a few candidate models (capability tiers). Validate them quickly using public benchmarks and a small set of business samples.
    2. POC comparison. Use fixed evaluation scripts that cover quality, latency, cost, and stability. Produce reproducible reports.
    3. Canary release. Route a share of users or traffic to the selected model. Put rollback procedures and human review in place.
    4. Continuous regression. Whenever the model version, prompt templates, or retrieval strategy changes, run the regression set automatically. This prevents the common pattern of a model peaking at launch and then drifting out of control.

    You can also use tiered routing in production:

    • Simple tasks go to a low-cost model.
    • Complex tasks go to a high-quality model.
    • Failures fall back to another model or escalate to a human.

    Example pseudocode:

    1. function route(request):
    2. task = classify(request)
    3. if cache.hit(request):
    4. return cache.get(request)
    5. if task.complexity == "low":
    6. candidate = model_pool.low_cost
    7. else if task.complexity == "high":
    8. candidate = model_pool.high_quality
    9. else:
    10. candidate = model_pool.balanced
    11. result = candidate.invoke(request)
    12. if result.invalid or result.confidence < threshold:
    13. result = model_pool.fallback.invoke(request)
    14. if result.needs_review:
    15. enqueue_human_review(result)
    16. cache.set(request, result)
    17. log_metrics(request, result)
    18. return result

    Implementation Patterns: Caching, RAG, Fine-Tuning, and an Evaluation Loop

    • Caching fits high-frequency, repeated questions and stable prompt templates.
    • Retrieval-augmented generation (RAG) fits use cases where knowledge changes frequently and answers need citations. However, you must evaluate chunking, retrieval, reranking, and citation consistency.
    • Fine-tuning fits vertical tasks with a stable style and sufficient labeled data. Guard against overfitting and catastrophic forgetting.
    • Prompt engineering is useful for rapid iteration, but prompts should be version-controlled.

    Whichever approach you choose, build an evaluation loop:

    1. Route business samples back into the pipeline.
    2. Label them.
    3. Run regression tests.
    4. Release gradually through canary deployments.
    5. Monitor cost.

    Only when model selection is part of a continuous engineering system can you truly answer “Which model is right for me?” instead of chasing a static leaderboard.

    Common Pitfalls and Conclusion

    Common pitfalls include:

    • Judging models by parameter count alone, without checking task fit
    • Looking only at price per call, not total cost of ownership
    • Relying on public benchmarks without blind evaluation on your own business data
    • Ignoring version drift and prompt template differences
    • Executing tool-calling results directly without permission checks
    • Missing fallback paths and human review

    There is no one-size-fits-all ranking of Chinese LLMs, and no simple “Top 4” list. Turn the question into measurable metrics. Then use small-sample POCs, canary releases, and regression evaluation to build an evidence chain. Finally, make your decision in light of cost, compliance, and operational constraints. That is how a solution stays controllable and sustainable in a fast-moving technical landscape.