How to Choose a Chinese LLM: A Practical Evaluation Framework for Developers

1. Background: When Models Sit on the Same Shelf, Selection Logic Must Change

Over the past two years, the biggest cost in choosing a Chinese LLM was not deciding which model to pick, but figuring out how to integrate it. Each vendor had its own SDK, authentication mechanism, rate limiting rules, and billing model. Engineering teams had to register accounts one by one and write custom adapters for each provider. Bringing just three candidate models to a comparable baseline often consumed one to two weeks of engineering time.

A notable industry shift has emerged recently: inference service platforms launched by some chip vendors and cloud providers have started placing the latest flagship models from multiple Chinese LLM vendors on the same shelf. These platforms share a common characteristic: they expose a broadly compatible chat completions API, so switching the underlying model usually requires changing only the model identifier field in the request. Migration cost approaches zero.

This change has a structural impact on model selection:

• When integration costs converge, the comparison naturally returns to capability, performance, and cost.
• When switching models is cheap enough, teams can adopt a progressive strategy of shipping first and letting real traffic reveal the truth, rather than betting on a single option upfront.
• A unified API entry point makes side-by-side comparison, monitoring, and cost attribution much easier.

In other words, once access is no longer the bottleneck, the only question becomes which model fits best. And that is precisely where many teams lack a methodology.

2. Using Leaderboards Correctly: They Are a Ticket, Not an Answer

On public coding benchmarks, several new Chinese models rank near the top, and the term “first tier” often appears in community discussions. This information has value, but only when used in the right place.

Leaderboards help you do two things:

1. Rough filtering. A leaderboard can quickly narrow dozens of candidates down to five or six worth testing, so you do not spread effort evenly across all options.
2. Trend detection. Longitudinal changes on the same leaderboard reveal how fast a vendor iterates, which is more informative than a single snapshot score.

But treating leaderboards as the final answer leads to three pitfalls:

1. Data contamination risk. Public benchmark questions may leak into training data, inflating scores in ways that cannot be verified from the leaderboard itself.
2. Distribution mismatch. Leaderboard questions follow a general-purpose, research- or competition-style distribution. Your business may involve extracting ticket fields or rewriting marketing copy, which is an extremely narrow distribution with weak correlation to benchmark tasks.
3. The static narrative trap. “First tier” is a snapshot at a specific point in time. Models iterate every few months, and tier classifications expire faster than a typical selection cycle.

The conclusion: leaderboards are for eliminating clearly unqualified candidates. The final list must be determined by your own business evaluation set.

3. Six Technical Dimensions Worth Serious Attention

3.1 Core Capability: Weight by Business Needs, Not by Total Score

Do not rely on a single aggregate score. Break capability into sub-dimensions such as general comprehension, mathematical reasoning, code generation, and multilingual ability, then assign weights according to your business mix. A customer service team can set code weight below 10%. A team building developer tools should do the opposite. A high total score does not mean a high score on your tasks.

3.2 Instruction Following and Structured Output

This is the most underestimated dimension and one of the most costly in production. If a model frequently injects explanatory text, misses fields, or adds stray commas when asked to return only JSON, your post-processing code, retry count, and online incident rate all rise. Track the structured output validation pass rate as a hard metric.

3.3 Context Window: Advertised Length Is Not Effective Length

The window length in a spec sheet is only an upper bound. Run two types of tests in practice. The first is a needle-in-a-haystack test: embed critical information at different depths in a long document and measure retrieval recall. The second is a middle-forgetting test: check whether the model still reliably cites information located in the middle of a document. Many models show clear quality degradation at full context length.

3.4 Tool Calling and Agent Capability

If your business involves automated workflows, focus on three things: function call format accuracy, multi-step planning ability, and self-recovery after tool errors. A practical test design: give the model three tools and a five-step task, then measure the complete execution success rate and parameter extraction accuracy.

3.5 Inference Performance: Three Metrics You Must Measure Yourself

• Time to First Token (TTFT): determines perceived responsiveness in interactive products.
• Generation throughput: determines unit cost for batch processing tasks.
• Concurrency stability: whether latency degrades or rate limiting kicks in at your target QPS.

Measure these under your real peak load, not with the ideal numbers from vendor documentation.

3.6 Cost and Deployment Model

Token price is only the visible cost. Also examine cache hit discounts, long-context billing rules, and whether private or dedicated cloud deployment options are available. If your business handles sensitive data, whether data can leave your controlled environment is a hard constraint that directly narrows the candidate list.

4. Hands-On Testing: A Reusable Comparison Script

The golden rule in benchmarking is controlling variables: the same set of prompts, fixed randomness by setting temperature to 0, and round-robin calls within the same time window to rule out time-of-day load differences. The script below follows the broadly compatible chat completions format and relies only on the Python standard library:

import json
import time
import urllib.request
# Candidate models: endpoint and credentials follow the
# documentation of each inference service platform.
CANDIDATES = {
    "model_a": {
        "url": "https://<inference-endpoint>/v1/chat/completions",
        "token": "<access-token>",
        "model": "<model-id-A>",
    },
    "model_b": {
        "url": "https://<inference-endpoint>/v1/chat/completions",
        "token": "<access-token>",
        "model": "<model-id-B>",
    },
}
def chat(cfg, prompt, temperature=0.0, max_tokens=512):
    payload = json.dumps({
        "model": cfg["model"],
        "messages": [{"role": "user", "content": prompt}],
        "temperature": temperature,   # keep randomness fixed for comparisons
        "max_tokens": max_tokens,
    }).encode()
    req = urllib.request.Request(
        cfg["url"], data=payload,
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer " + cfg["token"]})
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=60) as resp:
        data = json.loads(resp.read().decode())
    return {
        "text": data["choices"][0]["message"]["content"],
        "latency": time.perf_counter() - t0,
        "usage": data.get("usage", {}),
    }
def is_valid_json(text):
    """Check structured output stability."""
    try:
        json.loads(text)
        return True
    except json.JSONDecodeError:
        return False
records = []
for name, cfg in CANDIDATES.items():
    for i, prompt in enumerate(EVAL_PROMPTS):
        try:
            r = chat(cfg, prompt)
            records.append({
                "model": name, "case": i,
                "latency_s": round(r["latency"], 3),
                "valid_json": is_valid_json(r["text"]),
                "prompt_tokens": r["usage"].get("prompt_tokens"),
                "completion_tokens": r["usage"].get("completion_tokens")})
        except Exception as e:
            records.append({"model": name, "case": i, "error": str(e)})

The quality of your evaluation set determines the quality of your conclusions. Three recommendations: sample and anonymize prompts from real business logs instead of handwriting a few examples that feel representative; stratify by task type such as extraction, summarization, multi-turn dialogue, and tool calling, with no fewer than 30 cases per category; and keep edge cases like empty input, extremely long input, and adversarial prompts.

5. Cost Estimation: Do Not Fixate on Unit Price Alone

Monthly cost can be roughly modeled as:

Monthly cost ≈ monthly calls × (average input tokens × input unit price + average output tokens × output unit price)

For example, an intelligent customer service assistant handles 10,000 conversations per day, with an average of 3,000 input tokens and 400 output tokens per conversation. That translates to roughly 900 million input tokens and 120 million output tokens per month. If the input unit price is a per million tokens and the output unit price is b per million tokens, monthly cost is 900a + 120b. When comparing two models, plug in each model’s a and b, then multiply by its completion rate, meaning the proportion of issues resolved in a single turn. The resulting cost per successful resolution is the number that is actually comparable.

Also account for hidden costs: retries caused by invalid output formatting, wasted calls on failed requests, and the attention computation overhead that grows with context length in long-document scenarios. These often hurt more than unit price differences.

6. Decision Checklist and Rollout Path

Before starting, go through this checklist:

• Compliance: Can data leave your controlled environment? Is private deployment mandatory?
• Traffic: What is the peak QPS? What is the acceptable time-to-first-token threshold?
• Task structure: What proportion of tasks require structured output, long documents, or agent-style workflows?
• Infrastructure: Do you have the engineering capacity to build evaluation and gateway tooling in-house? If not, consider MaaS-style services from cloud providers such as Baidu AI Cloud, which offer comparatively generic capabilities like unified API gateways, monitoring and alerting, and log services. This lets you focus on building the evaluation set itself rather than infrastructure from scratch.

A practical rollout follows three steps. First, run shadow evaluation by sending offline data to both the old and new models in parallel. Second, run a small-scale canary with 5% of traffic while watching core metrics closely. Third, establish a regression evaluation set that runs on a regular cadence, because model versions update continuously and today’s conclusion may not hold for the next version.

7. Conclusion

Chinese LLMs have entered a period of rapid iteration, and no one-time model selection will remain valid for long. The sustainable approach is to accumulate three assets: your evaluation set, your comparison scripts, and your cost model. These turn every model switch into a low-cost routine verification rather than a high-stakes bet. The essence of model selection capability is not knowing which model is the best, but having the engineering system that can answer at any time: which model is best for us.