A Practical Evaluation Framework for Selecting Chinese LLMs: From Inference Deployment to Production

Why “Which Model Is Best?” Is the Wrong Question

In technical communities, “Which Chinese LLM should I choose?” is a common question. However, this phrasing implies a universal best model exists. In reality, model selection depends heavily on the use case, integration method, latency budget, data compliance requirements, and the team’s ability to maintain the system.

A more useful way to frame the question is:

• How will the model be accessed: online API, self-hosted weights, or edge inference?
• What are the business requirements for time-to-first-token, throughput, and concurrent requests?
• Do you need long context, tool calling, structured output, or multimodal capabilities?
• Will prompt assets need to be reused across different models?
• How will logging, audit trails, content safety, and permission policies be implemented?

Therefore, the core of model selection is not comparing names but establishing a process from requirements to validation.

The Value of a Unified Inference Layer

Many teams begin by directly using a single vendor’s online API. As the business evolves, models are swapped or multiple candidates are introduced. If business code is deeply coupled to one vendor’s SDK, replacement costs rise quickly.

A more robust approach is to introduce a unified inference access layer. This layer is responsible for:

• Normalizing differences in request and response formats across providers;
• Centralizing authentication, retry, timeout, and rate-limiting policies;
• Supporting model routing, canary releases, and fallback;
• Recording standardized inference logs.

Example: define a unified chat request structure within the application:

@dataclass
class ChatRequest:
    model: str
    messages: list[dict]
    temperature: float = 0.7
    max_tokens: int = 1024
    stream: bool = False

Business code depends only on this structure, not on any vendor SDK. An underlying adapter then converts it into the HTTP format required by each platform.

Trade-offs Between Online APIs and Self-hosted Deployment

Chinese LLM vendors generally offer online inference APIs. For prototyping and low-concurrency internal tools, these APIs significantly reduce startup costs. However, production environments require attention to:

• Whether dedicated inference clusters or resource isolation are available;
• Per-account QPS and TPM limits;
• Support for private network access and data staying within a controlled domain;
• Degradation plans during outages or rate limiting.

If the business requires strict data residency, model version locking, or gives predictable inference costs, self-hosting is a more suitable choice. Self-hosting typically involves containerized deployment and GPU resource management. A common workflow:

1. Acquire model weights and an inference runtime;
2. Deploy the inference service on a container platform;
3. Expose a standard HTTP interface through a load balancer;
4. Configure health checks, log collection, and monitoring alerts;
5. Test quantization, concurrency, and GPU memory usage.

Realistic Inference Performance Testing

Benchmark rankings do not equal production performance. Teams should run three types of tests based on their own business data.

1. Quality Testing

Build a test set from real requests, covering at least:

• Information extraction and summarization;
• Context retention in multi-turn conversations;
• Structured output, such as JSON Schema constraints;
• Edge cases and malformed inputs.

Evaluation criteria should be defined in advance, such as factual consistency, format validity, and recall. Where necessary, use human review or a secondary model to cross-check results.

2. Latency and Throughput Testing

Focus on these metrics:

• TTFT: time to first token;
• TPOT: average time per output token;
• Maximum sustained throughput;
• Stability during long outputs.

Example load testing script structure:

import time
import concurrent.futures
def measure_one(prompt):
    start = time.time()
    resp = call_llm(prompt)
    elapsed = time.time() - start
    return elapsed, len(resp.get("text", ""))
with concurrent.futures.ThreadPoolExecutor(max_workers=8) as ex:
    results = list(ex.map(measure_one, prompts))

By adjusting concurrency, you can observe latency inflection points and rate-limit thresholds, which helps determine whether the model meets your SLA.

3. Cost Testing

Online API costs depend on the number of input and output tokens. During testing, separately track:

• Input token count per request;
• Output token count;
• Discounts from cache hits;
• Hidden costs of long-context tasks.

For self-hosted models, account for GPU instance costs, idle time, and operational overhead.

Prompt Portability Is Often Underestimated

Different models show varying sensitivity to system prompts, dialogue roles, function calling, and output constraints. A prompt that performs well with model A may produce format drift when migrated to model B.

Treat prompts as versionable assets. For example:

version: 1.2
role: Technical Support Assistant
constraints:
  - Only answer questions covered by product documentation
  - State explicitly when the answer is unknown
output_format: json

When testing, run the same prompt across multiple models and compare:

• Whether the output structure is respected;
• Whether extra explanatory prefixes or suffixes appear;
• Whether constraints hold across multi-turn conversations.

For scenarios requiring strict structured output, introduce an output parser and validation layer instead of relying on the model’s intrinsic discipline.

Verifying Tool Calling and Agent Capabilities

If the business involves tool calling, you must verify that the model can reliably output parseable tool arguments. Common issues include:

• Inconsistent parameter name spellings;
• Comments or plain text mixed into JSON;
• Incorrect selection among multiple tools;
• Failure to continue reasoning after a tool returns a result.

Key test points:

• Define 10 to 20 tool-calling test cases;
• Verify that each output can be parsed as JSON;
• Validate that arguments conform to the schema;
• Track the single-call success rate and frequent failure patterns.

Attributing failed samples helps determine whether the issue lies in the prompt, the model capability, or the tool description.

Security, Compliance, and Content Governance Cannot Be an Afterthought

In enterprise settings, LLM applications involve data classification, permission isolation, and auditing. What matters is not whether the model itself is “safe,” but whether the deployment chain provides governance capabilities.

Consider adding to the inference pipeline:

• Sensitive information detection on the input side;
• Compliance checks on the output side;
• Full request log retention;
• Usage statistics by application, user, and model;
• Alerts for abnormal calls or unauthorized access.

For self-hosted deployments, ensure that logs do not record plaintext sensitive fields. Use masking or hashing before storage where necessary.

Observability Determines Long-term Maintenance Costs

Model deployment is only the beginning. Long-term operations require attention to:

• Call success rate for each model;
• Changes in latency percentiles;
• Token consumption trends;
• Distribution of error codes;
• Fallback and retry frequency.

A recommended unified log structure:

{
  "trace_id": "abc123",
  "model": "candidate-1",
  "latency_ms": 842,
  "input_tokens": 320,
  "output_tokens": 180,
  "status": "success",
  "error_code": null
}

Based on these logs, you can build comparison dashboards for different models and continuously run A/B tests on low-risk traffic.

Building a Candidate Evaluation Matrix

Summarizing the above dimensions into an evaluation matrix improves decision-making efficiency. Example:

Dimension	Candidate A	Candidate B	Candidate C
Quality score	8.5	8.1	7.6
Median TTFT	380ms	510ms	420ms
Tool-call success rate	91%	84%	88%
Structured output pass rate	96%	89%	93%
Cost per 10k calls	Medium	Low	High
Private deployment support	Yes	No	Yes

Weights should be assigned according to business priorities. For example, an online customer service application may prioritize stability and cost, while a code generation tool may emphasize quality and tool-calling ability.

Conclusion: Replace Conclusions with a Process

Returning to the original question. The most professional answer to “Which model should I choose?” is not a single model name but a repeatable evaluation mechanism.

Teams are advised to take the following steps:

1. Define business priorities for quality, latency, cost, and compliance;
2. Build a realistic test set rather than relying on public benchmarks;
3. Use a unified access layer to isolate model differences;
4. Run parallel tests on candidate models with low-traffic requests;
5. Base the final decision on logs and metrics;
6. Design model selection as a replaceable capability, not an architectural lock-in.

Under this framework, any newly released LLM can be quickly integrated into the validation process without overturning the existing system. That approach offers more long-term value than chasing a single leaderboard.