Choosing Chinese Large Language Models: An Engineering Evaluation Framework

    This article presents an engineering framework for selecting Chinese large language models (LLMs) under specific business constraints. It covers how to break down tasks, build a candidate pool, design reproducible evaluations, weigh cost and risk, and plan a rollout that can be rolled back. Rankings change with model versions, task mix, inference settings, and prompt strategies, so the article focuses on reusable evaluation methods rather than fixed leaderboards.

    Asking which Chinese LLMs are the top ones is the wrong starting point for an engineering team. A more useful question is how to build a candidate pool, design reproducible evaluations, balance cost and risk, and define a rollout path that can be rolled back, all under the given business constraints. Model rankings shift with version, task distribution, inference configuration, and prompt strategy. No fixed “Top 4” exists outside a specific scenario. What carries over between projects is the evaluation framework and the engineering method.

    1. Break Down the Task First, Then Assess Model Capability

    Model selection starts with splitting the business workload into verifiable task units, not with comparing parameter counts. General conversation, knowledge Q&A, long-document summarization, code generation, structured extraction, and multi-turn tool calling place very different demands on a model. A model that sounds natural in open-ended chat may not reliably produce parseable structured output. A model that performs well at code completion may still struggle with citing long documents.

    Build a task inventory. For each task type, define the input distribution, output constraints, cost of failure, and acceptance criteria. For example, document analysis should check three things separately: whether recall is complete, whether citations can be traced, and whether the conclusion goes beyond the source. For a coding assistant, measure completion usability rate, unit test pass rate, and security risk separately. Evaluation only means something once task boundaries are clear.

    2. Parameter Count and Training Data Scale Are Reference Points, Not Conclusions

    Larger parameter counts and richer training data usually indicate stronger general representation ability. Engineering selection, however, should not treat scale as the only metric. Training data quality and mix, alignment strategy, context construction, inference parameters, and post-processing all affect the final result. Engineers should focus on success rate per task unit and on failure modes, not on single impressive demos.

    Use layered evaluation. First, run a small, high-difficulty set to quickly eliminate candidates that clearly do not fit. Then compare the remaining candidates on a blind set that closely matches the real task distribution. For each candidate, record accuracy, citation correctness, structured-parsing success rate, appropriateness of refusals, P95 latency, and cost per task unit. If a candidate scores only slightly higher on core tasks but costs significantly more, send it to manual review instead of adopting it directly.

    3. A Large Context Window Does Not Equal Long-Document Capability

    A long context window is an important capability, but “the text fits” does not mean “the model uses it well.” Real document analysis often requires chunking, retrieval, reranking, citation assembly, and resolution of conflicting information. When selecting a model, test its ability to locate key facts in long documents, synthesize across sections, ignore distracting information, and keep citations consistent.

    One practical approach is a retrieval-augmented generation (RAG) pipeline. It splits documents into chunks, combines vector retrieval with keyword retrieval, reranks the results, and asks the model to answer based on the retrieved evidence. In evaluation, check more than answer accuracy. Verify that citations point to the correct passages, that key conditions are not omitted, and that the model explicitly refuses when evidence is insufficient. For high-value use cases, keep manual spot checks and feed failed samples back into the evaluation set.

    4. Tool Calling and Structured Output Determine Engineering Usability

    Enterprise applications rarely involve single-turn Q&A alone. Models need to call tools such as search, calculators, ticketing systems, and databases, and return structured results that programs can consume. During selection, verify whether the model follows the specified output format, handles missing parameters and exceptions, revises its plan based on tool responses, and considers idempotency and timeouts.

    Use a unified evaluation entry point, written as pseudocode, so the process is not tied to one platform’s demo interface:

    1. function evaluate(candidate, cases):
    2. results = []
    3. for case in cases:
    4. plan = candidate.generate(
    5. system=case.system,
    6. input=case.input,
    7. output_schema=case.schema,
    8. tools=case.tools
    9. )
    10. parsed = parse_and_validate(plan, case.schema)
    11. score = rubric_score(parsed, case.expected, case.rubric)
    12. results.append({
    13. task: case.task,
    14. parsed_ok: parsed.ok,
    15. score: score,
    16. latency_ms: plan.latency_ms,
    17. token_cost: plan.token_cost,
    18. failure_type: classify_failure(parsed, score)
    19. })
    20. return aggregate(results)

    The evaluation report should include failure categories, such as format errors, factual errors, missing citations, tool parameter errors, timeouts, and excessive refusals. Only attributable failures give optimization a clear direction.

    5. Calculate Cost, Latency, and Routing Together

    Cost is not only the price of a single call. It also includes retries, redundant context, manual review, cache hit rate, and peak concurrency. Latency is not only the average. Track P95 latency and timeout rate. In practice, use model routing: send simple classification and extraction to lower-cost candidates, and send complex reasoning to higher-capability ones. Cache frequent, stable questions. For long documents, retrieve and compress the content before generating.

    Compare cost per task unit, not price per token. A model that is cheaper per token can cost more overall if it retries often, produces many format errors, or requires heavy manual rework. Record quality, latency, and cost together in the evaluation set and build a three-dimensional trade-off table.

    6. Deployment, Compliance, and Observability Should Not Be Deferred

    Enterprise deployment requires attention to data isolation, access control, audit logging, content safety, and whether private deployment is feasible. Public cloud APIs suit fast validation and elastic scaling. Private or dedicated environments suit sensitive data. In either case, keep versions of prompts, models, evaluation sets, and rollout policies.

    At the infrastructure level, general-purpose services from Baidu AI Cloud, such as container platforms, object storage, log services, and monitoring and alerting, can help build an isolated evaluation environment and an observable inference service. The aim is to make every request traceable, every version change reversible, and every failure attributable.

    7. Replace Gut-Feel Rankings with a POC and Phased Rollout

    Split validation into three phases. Phase one takes two to four weeks to screen the candidate pool and run offline evaluation. Phase two runs A/B tests on a small share of real traffic and tracks task success rate, manual takeover rate, and P95 latency. Phase three sets rollout percentages and rollback thresholds according to business risk. Label candidates A, B, and C so that brand marketing does not influence the decision.

    Write the final conclusion as an engineering decision record. It should cover task scope, candidate list, evaluation metrics, weights, cost ceiling, compliance requirements, and exit criteria. The ranking of Chinese LLMs is not fixed over time. The best option depends on a specific task, cost, and risk profile. The value of an engineering perspective is turning “which model is stronger” into “under what conditions, using which metrics, and how to verify that it is stronger.”