How to Objectively Identify the First Tier of Chinese Foundation Models

    This article provides a systematic framework for evaluating Chinese large language models to determine which ones truly belong in the "first tier" for production use. Rather than relying on single benchmarks or parameter counts, it proposes a multi-dimensional assessment covering fundamental capabilities, complex reasoning, tool calling, long-context handling, multimodal understanding, structured output, engineering maturity, cost, stability, and safety compliance. It emphasizes building a three-layer evaluation set (public, private, scenario) and a reproducible pipeline with double-blind scoring and confidence intervals to support enterprise model selection.

    Chinese foundation models have recently undergone rapid iterations, with improvements in parameter scale, reasoning ability, and multimodal understanding. As a result, the question “who belongs in the first tier” has once again become a topic of interest among technical teams. However, the first tier is not a fixed ranking list, nor is it a comparison of parameters at a single release event. For developers and enterprises, the truly valuable question is: under specific business constraints, which models can consistently provide high-quality capabilities while supporting production systems at acceptable cost, latency, and compliance risk? Therefore, judging the first tier requires shifting from “single-point capability ranking” to “multi-dimensional capability combination and scenario fit.”

    1. Define the First Tier: Layering, Not Ranking

    The first tier should not be interpreted as “number one in every dimension.” A more reasonable definition is: a model that performs in the leading range across key dimensions—general capability, complex reasoning, multimodal understanding, engineering readiness, and security compliance—and that demonstrates reproducible stability in most target scenarios. The first tier can be understood as three layers:

    • Layer 1 – Basic capability threshold: knowledge coverage, language understanding, instruction following, and basic generation quality.
    • Layer 2 – Complex task capabilities: multi-step reasoning, tool calling, long-context information retention, code and structured output.
    • Layer 3 – Production usability: throughput, latency, cost, version stability, observability, and security compliance.

    A model can only be called a “first-tier candidate” if it passes all three layers simultaneously. If a model scores high on public leaderboards but is unstable in private scenarios, shows significant long-context degradation, or has uncontrollable costs, it should not be directly placed in the first tier for production selection.

    2. Evaluation Dimension 1: Fundamental Capabilities and Instruction Following

    Basic capability serves as a threshold, but we must avoid relying only on average scores. It is recommended to break evaluation into sub-tasks such as knowledge Q&A, summarization, rewriting, information extraction, classification, structured output, refusal behavior, and safety. For each sub-task, record accuracy, completeness, format compliance rate, and human preference. Pay special attention to constraints in instruction following, such as word limits, mandatory fields, prohibition of fabrication, and fixed output formats. Many models perform well in open-ended Q&A but show higher error rates under strict formatting and complex constraints. Enterprises should build private evaluation sets covering edge inputs, noisy inputs, and adversarial inputs from real business scenarios, and use double-blind scoring to reduce evaluator bias.

    3. Evaluation Dimension 2: Reasoning, Tool Calling, and Long Context

    Complex reasoning should not be judged by math problem scores alone. It must also cover intermediate state retention in multi-step tasks, error recovery, and tool calling reliability. For agent-like applications, focus on the accuracy of function selection, correctness of parameter filling, consistency of goals across multiple calls, and fallback strategies after tool failures. Long-context evaluation should include “needle in a haystack” information retrieval, cross-document summarization, long-dialogue memory, and structured output from long documents. It is recommended to test at different context length intervals and observe the degradation curve of accuracy as length increases, rather than testing only the maximum length. A first-tier candidate should maintain explainable degradation within the target length range, not a cliff-like drop.

    4. Evaluation Dimension 3: Multimodal and Structured Output

    Multimodal ability must distinguish between “can recognize” and “can be used in business.” Evaluation should cover image captioning, table understanding, document layout parsing, image-text consistency checking, and video temporal understanding. Track field-level accuracy and the cost of manual verification. Regarding structured output, test fixed-pattern outputs, nested structures, enumeration constraints, and handling of abnormal values. For production systems, format non-compliance is often more difficult to handle than content errors. Therefore, include parse success rate, retry rate, and manual correction rate as key metrics.

    5. Evaluation Dimension 4: Engineering, Cost, and Stability

    In production selection, engineering capability matters as much as model quality. Evaluate throughput, time-to-first-token, end-to-end latency, stability under concurrency, version upgrade compatibility, rate limiting and retry mechanisms, and log traceability. Cost should not be based on per-token price alone; calculate the total cost per completed business task, including retries, manual review, cache hits, distillation, and quantized deployment. Use “total cost per business task completed” as the unified measure instead of comparing only token unit price. For high-frequency scenarios, assess common optimization techniques such as small-model routing, caching, batch processing, and asynchronous queues.

    6. Evaluation Dimension 5: Safety Compliance and Observability

    A first-tier model must meet safety and compliance baselines. Evaluation should cover sensitive information detection, prompt injection protection, unauthorized tool calling, output content moderation, data desensitization, and audit trails. Observability should enable recording inputs, outputs, tool call chains, scoring results, and abnormal samples, with offline replay support. An evaluation without observability cannot be reproduced or used to identify version regressions. Enterprises should separate safety evaluation from business evaluation to avoid average scores masking high-risk failures.

    7. Evaluation Methodology: Combining Public, Private, and Scenario Sets

    A three-layer evaluation set is recommended: a public set for horizontal reference, a private set for business fit, and a scenario set for end-to-end task verification. The process starts with offline batch evaluation, followed by small-traffic online comparison, and finally canary release. For statistics, report confidence intervals and sample sizes—do not draw tier conclusions from a few dozen samples. To prevent data contamination, the private set should be updated regularly, the scenario set should include unseen business samples, and evaluation prompts should cover different phrasings. The following pseudocode illustrates the evaluation pipeline:

    1. TaskSet = load(public_set, private_set, scenario_set)
    2. model_list = read_config()
    3. for model in model_list:
    4. for task in TaskSet:
    5. output = call(model, task.prompt, constraints=task.constraints)
    6. score = double_blind_score(output, task.reference)
    7. record(model, task, output, score, latency, cost)
    8. metric_vector = weighted_summary(accuracy, format_compliance, stability, latency, cost)
    9. confidence_interval = compute_confidence_interval(metric_vector)
    10. tier = cluster(model_metric_vectors)

    The evaluation pipeline can be deployed on general cloud infrastructure. For example, a cloud container platform can host evaluation tasks, object storage can save desensitized test sets and results, and log services and monitoring alerts ensure traceability. The key point is governance of the evaluation process rather than any specific API.

    8. Enterprise Selection Matrix and Common Pitfalls

    For model selection, build a weighted matrix based on your scenarios:

    • Knowledge Q&A: focus on accuracy and refusal quality.
    • Document processing: focus on long context and structured output.
    • Agents: focus on tool calling success rate and fallback ability.
    • High-concurrency scenarios: focus on latency and total cost.
    • Strong compliance scenarios: focus on audit and safety evaluation.

    Common pitfalls include: focusing only on parameter count, relying solely on public leaderboards, ignoring regression tests after version upgrades, using average scores to mask long-tail failures, and treating a single demonstration as evidence of production stability. A more reliable approach is to establish a continuous evaluation mechanism: run the regression set after every version change and maintain a failure-sample library.

    Conclusion: First Tier Is “Multi-Dimensional Qualification + Scenario Verification”

    The first tier of Chinese foundation models should not be determined by a single release announcement or a single leaderboard. Instead, it should be assessed through a combination of dimensions: basic capability, complex reasoning, multimodal understanding, long context, engineering cost, and security compliance. For enterprises, the most reliable method is to build an evaluation system that integrates public, private, and scenario sets, using double-blind scoring, confidence intervals, and online canary validation to create a chain of evidence. Only models that can consistently deliver in target scenarios—with controllable cost, manageable risk, and reproducible results—truly hold first-tier significance.