Choosing a Chinese LLM: Capability Dimensions and Engineering Practices

1. The Starting Point: When the Model Shelf Gets Crowded

Over the past two years, the number of Chinese large language models available to enterprises has expanded rapidly from a handful to well over a dozen series. The capability gap between flagship models is narrowing, release cycles are accelerating, and pricing strategies change frequently. For developers on the ground, the question of “which one to choose” has not become simpler. With more candidates, the cost of choosing poorly has not gone down either.

A more practical shift has happened on the distribution side. Early on, most Chinese models could only be accessed through each vendor’s own platform. Today, more models are being integrated into unified inference service platforms, allowing developers to access models from multiple sources through a single API. This “model supermarket” approach changes the nature of selection from “can we access it” to “how do we choose correctly.”

This article does not recommend or rank any specific model. Instead, it provides an evaluation framework to help teams make decisions that are explainable and reproducible.

2. Before Selecting a Model: Clarify What You Are Paying For

Many selection discussions start by comparing leaderboard scores, which is often the least efficient approach. Leaderboards measure performance on general benchmarks, while production systems care about combined performance on a specific task, under specific cost constraints, and within specific latency requirements. Before opening any evaluation page, answer the following questions:

1. Task type: Is it single-turn Q&A, multi-turn dialogue, long-document understanding, code generation, structured extraction, or tool calling?
2. Context size: What are the average and P99 input lengths? Do you need to handle entire books or repository-level inputs?
3. Latency budget: Is this offline batch processing where minutes are acceptable, or online interaction where first-token latency needs to be within a few hundred milliseconds?
4. Output determinism: Can you tolerate variation in generated results? Do you need strict structured output such as JSON Schema constraints?
5. Deployment boundary: Is data allowed to leave your own network? Are there compliance requirements mandating private deployment?
6. Cost structure: Which is more economical: token-based billing for hosted calls, or the fixed cost of GPUs for self-hosted inference?

These six questions largely determine the candidate pool. For example, a “data must not leave the network” requirement immediately eliminates all public cloud hosted options and reduces the problem to “which model weights can be deployed privately.”

3. Capability Evaluation: Translating Leaderboard Scores into Engineering Metrics

Public leaderboards have value, but they need to be interpreted correctly. Take coding ability as an example. A high score on a coding benchmark may come from being good at completing short functions, while your scenario requires the model to understand cross-file dependencies before refactoring a module. These two tasks demand different capabilities.

A layered evaluation approach is recommended:

3.1 General Capability: Use for Screening, Not Decision-Making

General benchmarks such as language understanding, commonsense reasoning, and math are useful for quickly filtering out candidates that clearly do not meet the bar. They are not suitable for ranking top-tier models against each other. The gap between leading models on these dimensions is usually within a few percentage points, and that gap may not reproduce on your specific task.

3.2 Domain Capability: Build a Small Custom Evaluation Set

What carries real decision-making value is an evaluation set you build yourself. Here is how:

• Extract 50 to 200 real samples from production logs or business documents, covering both typical and edge-case inputs.
• Define automatically checkable acceptance criteria for each sample, such as exact match, field completeness, unit test pass rate, or a manual scoring rubric.
• Run all candidate models with the same prompt and decoding parameters, recording pass rate, average output length, and average latency.

A sample evaluation script structure:

# Evaluation flow outline (pseudocode; adapt interfaces to your platform)
cases = load_eval_cases("eval_set.jsonl")   # Contains input and expected validation logic
results = {}
for model_id in candidate_models:
    passed, latencies = 0, []
    for case in cases:
        resp = call_model(model_id, case["input"], temperature=0)
        latencies.append(resp.latency_ms)
        if case["validate"](resp.text):
            passed += 1
    results[model_id] = {
        "pass_rate": passed / len(cases),
        "p50_latency": percentile(latencies, 50),
        "p99_latency": percentile(latencies, 99),
    }

The value of this process is that it produces not “which model is better” but “which model is better on my task.” Only the latter is a conclusion that can go into a technical decision document.

3.3 Long Context: Mind the Gap Between Effective Length and Advertised Length

The advertised context window length is not the same as the length the model can effectively use. A common phenomenon is that once input exceeds a certain threshold, the model’s recall of information in the middle sections drops noticeably. A simple verification method is the “needle in a haystack” test: embed factual questions at different positions in a long document and observe the recall curve across positions. If your scenario involves long-document Q&A or repository-level understanding, this test should take priority over any general leaderboard.

3.4 Tool Calling and Structured Output

If the model needs to call external tools or produce output consumable by programs, verify two things in particular: first, the correctness of function-call parameters, especially nested structures and enum values; second, the stability of structured output, meaning the JSON parse rate across multiple samples. These capabilities are poorly reflected in general benchmarks and must be tested directly.

4. Deployment Mode: Hosted API vs. Self-Hosted Inference

The other half of model selection is deployment mode. The two paths differ significantly in cost structure and engineering burden.

Dimension	Hosted API	Self-Hosted Inference
Upfront investment	Nearly zero	GPU purchase or rental, inference framework deployment
Marginal cost	Linear growth per token	Nearly fixed, amortized by utilization
Elasticity	Guaranteed by the platform	Requires self-managed scaling
Data boundary	Data leaves your network	Data stays in your own environment
Operations burden	Low	Requires monitoring VRAM, throughput, and queue backlog
Model switching	Change a model identifier	Redeploy weights and re-run load tests

A useful rule of thumb: when call volume is low, demand is volatile, and the team lacks inference operations experience, prefer hosted APIs. When call volume is large and stable, data boundary requirements are strict, and GPU operations capability exists, consider self-hosting. The middle ground can use a hybrid approach: hosted APIs for online interactive traffic to guarantee elasticity, and self-hosted inference for offline batch processing to amortize costs.

For self-hosted inference, also pay attention to how well the inference framework supports the model: whether quantization options are complete, whether batch scheduling is efficient, and whether prefix caching is supported. Prefix caching has a significant impact on multi-turn dialogue and long system-prompt scenarios. These factors often affect real-world throughput more than differences in model parameter count.

5. Abstraction Layer Design: Making Your Choice Replaceable

Regardless of which model you ultimately choose, the integration layer should be designed to be replaceable. This is not over-engineering; it is a basic requirement for this type of system. The pace of model iteration means any selection decision has an expiration date.

A few concrete suggestions:

1. Unified abstraction layer: Add a thin wrapper between business code and model APIs, unifying message format, timeout, retry, and error codes. Business logic should depend only on this layer.
2. Configurable model identifiers: Model selection should be delivered through configuration rather than hardcoded, making it easy to run canary rollouts and A/B comparisons.
3. Observability from day one: Record the model identifier, input and output token counts, latency, and error type for every call. Without this data, any future optimization or migration is flying blind.
4. Fallback path: Prepare backup models and degradation strategies for critical paths, such as switching to a smaller and faster model on timeout, or returning cached results.

# Abstraction layer outline: configurable model + failure fallback
class ModelGateway:
    def __init__(self, primary, fallback, timeout_ms=8000):
        self.primary, self.fallback = primary, fallback
        self.timeout_ms = timeout_ms
    def complete(self, messages, **kwargs):
        try:
            return self._call(self.primary, messages, **kwargs)
        except (TimeoutError, UpstreamError):
            return self._call(self.fallback, messages, **kwargs)

6. Cost Accounting: Converting Token Prices into Business Costs

Token price is the most intuitive but also the most misleading metric. What you actually need to calculate is the cost per unit of business output:

Cost per task = (average input tokens x input price + average output tokens x output price) / task success rate

The denominator matters significantly. A model that is 30% cheaper but has a 15% lower success rate may end up with a higher effective cost, because failed tasks require retries or manual intervention. These costs do not appear on the invoice, but they show up in labor overhead.

Also note output length differences: under the same prompt, different models may differ by more than a factor of two in verbosity, which directly amplifies output-side costs. The evaluation phase should record average output token count rather than looking only at the price sheet.

7. A Practical Selection Workflow

The above can be compressed into executable steps:

1. Define constraints: Data boundary, latency budget, context size, and structured output requirements. Use these to narrow the candidate pool.
2. Build an evaluation set: 50 to 200 real samples with automated validation logic, covering typical and edge-case inputs.
3. Run batch tests: Fix prompts and decoding parameters. Record pass rate, output length, and P50/P99 latency.
4. Run targeted verification: Long-context recall curves, tool-call parameter accuracy, and structured output parse rates.
5. Calculate effective cost: Use “cost per task = token cost / success rate,” including retry and fallback overhead.
6. Canary rollout: Switch 5% to 10% of production traffic through a configurable flag and observe behavior on real distributions.
7. Solidify the abstraction layer: Complete the abstraction and observability work so the next migration costs as little as possible.

8. Conclusion

Choosing a Chinese large language model is not fundamentally a question of “which one is the strongest.” It is a matching problem: which one fits best under given constraints. Leaderboards provide prior information; self-built evaluation sets provide posterior evidence; and a replaceable engineering design ensures that your conclusion does not become obsolete with the next model release.

For most teams, time invested in evaluation processes and abstraction-layer design pays back far more than repeatedly agonizing over candidate models. Models will keep iterating, but a good evaluation and integration system can be reused for the long term.