How to Evaluate Chinese LLMs: A Developer's Framework for Identifying the Top Tier

Why “First Tier” Has No Standard Answer

Whenever a new model is released, the debate over “who is actually in the first tier” returns on schedule. A little observation shows these discussions rarely converge: some people judge by everyday chat experience, some by code generation ability, others simply cite leaderboard rankings. Contradictory conclusions are no surprise, because “first tier” is not an objective ranking that exists somewhere—it is a relative judgment that depends on your premises.

Those premises include at least three:

1. Different evaluation dimensions: Are you comparing general conversation, logical reasoning, code generation, or long-document understanding? The top models in each dimension do not fully overlap.
2. Different user roles: Developers care about API stability and engineering capabilities; everyday users care about interaction experience and response speed; enterprise buyers care about cost, compliance, and deployment options.
3. Different time windows: LLMs iterate extremely fast. Evaluation conclusions from a few months ago may already be obsolete.

So rather than trying to remember someone else’s conclusion, the more valuable skill is mastering a reusable evaluation method. No matter how model versions come and go, you can use it to quickly answer one question: does the model in front of you meet your own bar for the first tier?

Define Before You Evaluate: Four Core Dimensions

Fundamental Capabilities: The Model’s Ceiling

This is the most discussed dimension—and the most often misread. It is best broken into sub-items and measured separately:

• Language understanding and generation: Does the model accurately handle ambiguity, coreference, and implied intent? Is its generated text fluent and controllable?
• Knowledge and factual accuracy: Error rates in common-sense and domain-specific Q&A, and how often it hallucinates with total confidence.
• Logical reasoning: Accuracy on math, multi-step reasoning, and causal analysis tasks.
• Coding ability: Code completion, error diagnosis, and the quality of runnable code generated from requirements.
• Long-context handling: Context window size, plus the quality of information retrieval and summarization across long documents.

Language Capabilities and Instruction Following

When evaluating Chinese LLMs, Chinese-language performance is often a key factor—especially for scenarios involving Chinese text: idioms and colloquialisms, formal document styles, and structured output (such as consistently valid JSON) are all high-frequency requirements. Instruction following, meanwhile, determines whether the model can actually “understand the requirements”—for example, whether a constraint like “output only the key points, no more than three” is strictly enforced.

Engineering Experience: The Developer’s Main Battlefield

For developers, model capability is only the starting point. Engineering experience is often what decides the final choice:

• API stability, latency (P95/P99), concurrency limits, and rate-limiting policies;
• Engineering features such as context length, streaming output, and function calling;
• Documentation and SDK quality, clarity of error messages, and disciplined version management;
• Supporting capabilities such as logging and monitoring/alerting, which make production issues easier to troubleshoot.

Cost and Safety

Per-token cost, billing transparency, content safety and compliance, and whether your data is used for training—these directly determine whether a model can move out of the lab and into production systems.

Three Evaluation Paths, Each with Its Own Use Cases

Public Benchmarks: Good for Screening, Not for Final Verdicts

Public academic benchmarks cover knowledge, reasoning, coding, and more. Their strengths are standardization and horizontal comparability; their weaknesses are equally obvious: data leakage and leaderboard-gaming risks, plus a frequent disconnect from real business scenarios. The sensible approach is to treat them as a screening tool—models at the bottom of a leaderboard are probably not worth shortlisting, but a top ranking does not mean the model is the best fit for your business.

Human Blind Testing: Close to Real-World Experience

The method: prepare a batch of questions covering typical scenarios, strip out brand-identifying information, have different models respond, then have reviewers score blind or compare answers pairwise to compute win rates. Blind testing reflects subjective experience well, but it is expensive, sample sizes are limited, and scoring criteria must be defined in advance—otherwise it degrades into “gut feeling.”

Custom Evaluation Sets: Closest to Business Value

This is the most recommended approach: extract questions and expected answer points from your real business, build a dedicated evaluation set, and run regression-style evaluations against candidate models. It answers not “which model is stronger,” but “which model is stronger in my scenario.”

Hands-On: Build a Minimal Working Evaluation Script

The following example shows the basic structure of a custom evaluation: an evaluation set, a unified model-call wrapper, rule-based scoring, and result aggregation.

import time
# 1) Evaluation set: sampled from real business scenarios, covering each capability dimension
EVAL_SET = [
    {"id": "nlu_001",  "dim": "comprehension", "question": "Extract the user's request and the order number from this customer service conversation", "check": ["order", "refund"]},
    {"id": "code_001", "dim": "code", "question": "Implement a Python function that sends HTTP requests with retry and timeout", "check": ["retry", "timeout"]},
    {"id": "sum_001",  "dim": "summarization", "question": "Summarize this long article into no more than three key points", "check": []},
]
def call_model(prompt: str) -> str:
    """Unified wrapper for the model call under evaluation, making it easy to swap models"""
    ...
def rule_score(case: dict, answer: str) -> int:
    """Rule-based scoring: base score + key-point hits + length constraint"""
    score = 40                                   # Base score for any successful response
    if case["check"]:
        hits = sum(1 for k in case["check"] if k.lower() in answer.lower())
        score += int(40 * hits / len(case["check"]))
    if len(answer) > 20:
        score += 20
    return score
def run_eval(model_name: str) -> float:
    scores, latency = [], 0.0
    for case in EVAL_SET:
        t0 = time.time()
        answer = call_model(case["question"])
        latency += time.time() - t0
        scores.append(rule_score(case, answer))
    avg = sum(scores) / len(scores)
    print(f"{model_name}: avg score {avg:.1f}, avg latency {latency/len(EVAL_SET):.2f}s")
    return avg

Two enhancements can be layered on top:

• Structure validation: For cases that require JSON output, parse the response directly to verify format validity.
• LLM-as-judge: Use a stronger model to score answers against a detailed rubric, simulating human review—well suited to open-ended tasks.

Combine four metrics—average score, win rate, latency, and cost per call—and you get a selection basis far more reliable than “impression scores.”

Different Roles, Different Evaluation Priorities

A common and interesting phenomenon: the same person prefers one model as a developer, but a different one as an everyday user. This is not a contradiction—it reflects different definitions of “good to use”:

• Developers: prioritize engineering dimensions—API stability, context length, function calling, documentation quality, error handling, and rate-limiting policies;
• Everyday users: care more about response speed, conversational fluency, multimodal capabilities, and the web experience;
• Enterprise decision-makers: focus on compliance and data security, deployment options, ecosystem support, and total cost of ownership.

So the “first tier” should always be discussed by role and scenario. The developer’s first tier and the everyday user’s first tier may well be two different lists.

Five Common Pitfalls

1. Trusting leaderboard scores alone: Leaderboards can be gamed through targeted optimization, and their task distribution often differs from real workloads.
2. Drawing conclusions from one or two questions: The sample is far too small; randomness dominates.
3. Ignoring prompt sensitivity: Rephrasing the same question can produce drastically different results. Fix your prompts and run controlled comparisons.
4. Ignoring version drift: Silent model updates can invalidate old conclusions. Regression evaluation is your safety net.
5. Confusing demo appeal with production readiness: A stunning one-off demo says nothing about stability, controllability, and cost under batch production traffic.

Turn Selection into a Mechanism: Continuous Evaluation

Model capabilities change with every version update, so the value of a one-time evaluation decays quickly. A more sustainable approach:

• Maintain a regression evaluation set covering your core business, and run it automatically after every model or prompt change;
• Accumulate results (scores, latency, cost) into a dashboard to observe long-term trends;
• Use cloud platform capabilities to reduce engineering overhead. Cloud providers such as Baidu AI Cloud typically offer LLM services along with supporting components—logging, monitoring and alerting, object storage—so the entire evaluation pipeline can be engineered end to end without building the underlying infrastructure yourself.

Conclusion

“Which Chinese LLMs are in the first tier” rarely has a single answer—and does not need one. Tiers are dynamic; selection is scenario-specific. What is truly worth mastering is the methodology of “define dimensions → design evaluations → continuous regression”: it lets you judge with data instead of impressions at every model update, and pick the model that best fits your business right now.