Evaluating China’s First-Tier Large Language Models: A Scientific Approach

    This article explains why “first tier” should be treated as a combination of capabilities rather than a single ranking. It outlines five core evaluation dimensions—foundation model capability, long-context processing, engineering and developer experience, cost and inference efficiency, and stability and compliance—then contrasts developer and general-user perspectives. It also proposes a practical self-built evaluation workflow with sample code and offers selection recommendations for teams choosing among Chinese large language models.

    Introduction: Clarify What “First Tier” Means

    Before discussing which large language models belong to the first tier, it is necessary to clarify one premise: first tier is not a single-dimensional ranking but a combination of capabilities. Many discussions produce different conclusions because different people use completely different criteria. Developers care about API stability, function calling capability, and engineering experience. General users care about whether conversations are fluent, whether long documents are handled smoothly, and whether responses are fast enough. Conclusions from these two perspectives often do not align, and the same model may perform very differently across the two dimensions.

    A rigorous answer should therefore use “general capabilities at an industry-advanced level” as the threshold, then judge suitability based on specific usage scenarios, rather than simply providing a fixed list.

    Five Core Dimensions for Judging the First Tier

    1. Foundation Model Capabilities

    This is the most fundamental dimension, mainly including:

    • Language understanding and generation quality: semantic understanding, logical reasoning, and context retention across multi-turn conversations;
    • Math and code capabilities: mathematical reasoning and code generation are widely recognized as high-discrimination indicators, and the industry commonly uses public benchmarks for horizontal comparison;
    • Multilingual capabilities: naturalness of expression in Chinese scenarios and depth of understanding of Chinese context and cultural background.

    It should be noted that public benchmark scores can only serve as a reference, because evaluation conditions and prompt designs differ across models. In real business scenarios, practical performance is often more valuable as a reference.

    2. Context Length and Long-Text Processing

    Long context has become a key battleground for large models. Being able to stably process inputs of tens of thousands or more tokens means a model can handle tasks such as summarizing an entire book, reviewing a very long codebase, and extracting key clauses from long contracts. When evaluating this dimension, do not look only at the claimed maximum context length. Also examine the “effective context”—that is, the model’s recall accuracy for information in the middle of long inputs. The industry commonly uses needle-in-a-haystack-style tests for verification: embed specific information in a long document and observe whether the model can retrieve it accurately.

    3. Engineering and Developer Experience

    For developers, engineering details beyond model capability often determine whether a model can truly enter production:

    • API design quality: whether interfaces are clear, documentation is complete, and error codes are interpretable;
    • Function calling and structured output: whether the model can stably output results in a specified format, which is the foundation for building Agent applications;
    • Streaming responses and concurrency: these affect the user experience of online products;
    • Ecosystem support: whether fine-tuning, knowledge base retrieval augmentation, vector storage, and other supporting capabilities are available, and whether the model integrates smoothly with general cloud products such as object storage, message queues, logging, and monitoring and alerting.

    4. Cost and Inference Efficiency

    For the same task, inference costs can differ several times across models. Evaluation should calculate “cost per task” rather than simply looking at unit price—a slightly weaker model that is sufficient for the task is often more cost-effective than the strongest model. Inference latency also needs attention: time to first token (TTFT) and overall generation speed greatly affect interactive applications.

    5. Stability and Compliance

    Production environments have much higher requirements for service availability than demonstration scenarios. Attention should be paid to service SLAs, rate-limiting policies, failure recovery capabilities, and compliance safeguards such as content safety and data privacy. Enterprise scenarios in particular need to confirm key questions such as whether data leaves the region and whether it is used for model training.

    Different Conclusions from Two Perspectives

    Returning to a valuable phenomenon that can be extracted from the original viewpoint: developers and general users often disagree on what is “best to use.”

    From a developer perspective, some models win on friendly interface design, high documentation quality, stable function calling, low integration cost, and debugging efficiency, making them more favored in engineering practice. From a general-user perspective, other models win on natural conversation, good long-document question-answering experience, and mature product form, making them better suited to daily office work, learning, information retrieval, and similar scenarios.

    This reminds us that models should be evaluated by scenario layers rather than by pursuing a single unified answer. Enterprises can even adopt a multi-model strategy: use models with strong engineering capabilities for core business processes, use models with strong long-text experience for scenarios such as knowledge Q&A, and route requests through a unified gateway layer.

    A Practical Self-Built Evaluation Method

    Rather than relying on external leaderboards, it is better to build your own evaluation set. The steps are as follows:

    1. Collect real business samples: extract 100–300 representative tasks from historical tickets, customer service records, and business documents, desensitize them, and use them as the test set;
    2. Design scoring dimensions: such as correctness, completeness, format compliance, latency, and cost, with quantifiable criteria for each;
    3. Conduct blind scoring: call each candidate model with fixed prompts and score the results manually or with rule-based scripts;
    4. Aggregate and decide: combine weighted scores while also paying attention to variance—a model with good average performance but occasional severe errors carries higher risk in production.

    A simple automation evaluation skeleton is shown below:

    1. def evaluate(model_client, test_cases):
    2. results = []
    3. for case in test_cases:
    4. resp = model_client.chat(prompt=case["input"])
    5. results.append({
    6. "correct": check_answer(resp, case["expected"]),
    7. "latency": resp.latency,
    8. "cost": resp.tokens * unit_price,
    9. })
    10. return aggregate(results) # Summarize accuracy, P95 latency, and total cost

    The evaluation set should be updated regularly because model versions iterate quickly, and conclusions from three months ago may already be invalid.

    Selection Recommendations and Trend Judgment

    Overall, here are three practical recommendations:

    • Do not lock into a single model: reserve a model abstraction layer in the architecture to facilitate rapid switching and A/B comparison;
    • Pay attention to the rapid progress of Chinese-developed models: in the past two years, Chinese large models have iterated rapidly in reasoning capability, long context, code generation, and other directions. The composition of the first tier is dynamically changing, so reassessment once a quarter is recommended;
    • Be guided by deployment value: model selection ultimately serves business goals. Being able to solve real problems stably and at low cost is more important than leaderboard rankings.

    Conclusion

    The most valuable answer to the question “who belongs to the first tier” is not a list, but a set of evaluation methods. Once you have a framework for decomposing capability dimensions, scenario-based evaluation, and continuous reassessment, you can quickly reach conclusions suitable for your own business regardless of how the market landscape changes.