A Practical Guide to Selecting Chinese LLMs and Deploying Inference Platforms
This guide walks through how to treat LLM selection as an engineering decision rather than a leaderboard ranking exercise. It covers building a requirement profile from concrete task units, creating a reproducible evaluation baseline with business data, measuring cost per successful task, adding a unified inference gateway for routing, fallback, and observability, and closing the loop with a weighted decision matrix and a phased rollout process. Whether you call public cloud APIs, deploy privately, or run a hybrid architecture, the core message is the same: define goals and constraints first, validate with reproducible benchmarks, then operationalize through a well-governed inference platform.
1. Background: Model Selection Is Not “Pick the Top of the Leaderboard”
As Chinese large language models continue to improve, developers and enterprises face an expanding set of choices: general conversation, code generation, long-document understanding, structured extraction, multimodal understanding, and tool calling may each map to different models and inference service options. Teams often fall into two traps early on. The first is relying solely on public leaderboard rankings, which ignores differences between your business data distribution and the benchmark’s evaluation methodology. The second is comparing only per-call pricing, which ignores retries, concurrency, caching, operations overhead, and compliance costs.
A more reliable approach is to treat model selection as an engineering decision: define business goals and constraints first, build a reproducible evaluation baseline, and then use an inference service platform to handle deployment, observability, and iteration.
On the cloud side, more and more inference services expose model capabilities through catalogs that developers can call on demand, without maintaining the underlying inference clusters from scratch. This lowers the cost of experimentation but introduces new governance requirements: version management, rate limiting and quotas, failure fallback, data security, cost visibility, and observability must all be designed in parallel. If you use Baidu AI Cloud, you can combine general-purpose container platforms, object storage, log services, and monitoring/alerting to assemble deployment and operations capabilities. This article does not bind itself to any specific API; it focuses on general methodology.
2. The First Layer of Selection: Decompose Tasks Before Comparing Models
Before selecting a model, break business requirements down into verifiable task units instead of asking vaguely “which model is the strongest.” Build a requirement profile along these dimensions:
- Task type: open-domain Q&A, summarization, translation, information extraction, code completion, code review, table Q&A, multi-turn customer service, tool calling, and so on. Different tasks have different demands for instruction following, long context, structured output, and stability.
- Input and output format: plain text, tables, images, audio, code repository snippets; whether the output is free text, JSON, function-call arguments, or multi-part structured results.
- Context length and document scale: long-document understanding requires attention to the context window, truncation strategy, segmented retrieval, and citation traceability—not just the advertised maximum length.
- Latency and throughput: interactive online scenarios care about time-to-first-token and end-to-end latency; batch processing cares more about throughput, concurrency, and unit cost.
- Controllability requirements: whether you need fixed output formats, cited sources, refusal policies, sensitive-content filtering, audit logs, and human review.
- Deployment and compliance constraints: whether data can leave a given boundary, whether private or hybrid deployment is required, data residency requirements, log retention periods, vendor terms, and security certifications.
With the requirement profile in place, build a candidate model pool. Keep the pool small: organize it into three categories—a general-capability baseline, scenario-specific candidates, and cost-oriented alternatives—with only a few models per category entering hands-on testing, so evaluation costs stay under control.
3. Building a Reproducible Evaluation Baseline
Public leaderboards are useful as leads, but they cannot replace business-specific evaluation. A reproducible baseline should include data, scripts, scoring rules, and version records.
1. Build a business evaluation set
Sample from real business logs after de-identification, covering high-frequency cases, long-tail cases, edge cases, and adversarial samples. Annotate each sample with expected key points, prohibited content, format requirements, and reference outputs. Layer the evaluation set: a core regression set for version comparison, an exploration set for discovering new problems, and a safety set for refusal and privilege-escalation testing.
2. Design scoring dimensions
At minimum, include: task correctness, instruction following, structured-output pass rate, factual consistency, citation traceability, refusal appropriateness, latency, cost per task, and failure rate. For subjective tasks, use double-blind review by two evaluators with consistency checks to avoid single-reviewer bias.
3. Unify invocation and logging
Evaluation scripts should record model version, inference parameters, prompt version, input summary, output, elapsed time, token usage, and status codes. Pseudocode:
function evaluate(endpoint, cases, prompt_version):records = []for case in cases:start = now_ms()resp = call_http(endpoint, {"prompt": render(prompt_version, case.input),"timeout_ms": 30000,"max_tokens": case.max_tokens})records.append({"case_id": case.id,"model_version": resp.model_version,"latency_ms": now_ms() - start,"token_usage": resp.usage,"status": resp.status,"output": resp.text,"prompt_version": prompt_version})return aggregate(records)
4. Run ablation and stability tests
The same model can behave very differently under different prompts, temperatures, max output lengths, and concurrency levels. Test retry rates, timeout rates, rate-limit trigger rates, and output format drift—do not wait until after launch to discover stability problems.
4. Cost, Performance, and Capacity Planning
Cost cannot be measured by unit price alone. Establish a “cost per successful task” metric:
Cost per successful task =(input token cost + output token cost + retry cost + cache and storage cost + amortized operations cost)/ number of successful tasks
For online scenarios, watch time-to-first-token, end-to-end latency, and tail latency under concurrency; for batch scenarios, watch throughput and queue wait time. Capacity planning should reserve headroom for peak load and include rate limiting, queuing, degradation, and caching strategies. For high-frequency repeated questions, introduce semantic caching or result caching; for long-document tasks, use segmented summarization, retrieval-augmented generation, and citation backfilling to reduce wasted context consumption.
5. Inference Service Integration and Engineering Governance
Whether you choose public cloud API calls, private deployment, or a hybrid architecture, place a unified inference gateway between the business layer and the models. The gateway masks model differences and centralizes authentication, quotas, retries, fallback, logging, and auditing. It should provide:
- Routing policies: select candidate models by task type, cost threshold, and latency requirements;
- Fallback policies: switch to a backup model when the primary model times out or is rate-limited, and record the reason for the switch;
- Prompt and version management: prompt templates, model versions, and evaluation set versions must be traceable;
- Observability: request volume, error rate, latency percentiles, token consumption, cache hit rate, and rate-limit counts;
- Security governance: input de-identification, output filtering, secondary confirmation for sensitive operations, and audit log retention.
On the deployment side, use a general-purpose container platform to host the gateway and surrounding services, object storage for evaluation sets and audit archives, and log and monitoring/alerting services to build the observability stack. If you use Baidu AI Cloud, you can combine the general categories above, but specific APIs and quotas should be verified against the actual console and official documentation.
6. Compliance, Security, and Risk Control
Model selection must incorporate data security and compliance assessment: define data classification and grading, de-identification rules, cross-border and residency requirements, log retention periods, vendor security terms, content safety policies, and human review workflows. For sensitive data such as personal information, contracts, financials, or code, prioritize minimal-necessary input, de-identification, access control, and audit trails. Also watch for model version drift, vendor lock-in, and quota risks: keep multiple candidate models behind the unified gateway, configure backup options on critical paths, and run periodic regression evaluations to avoid single points of dependency.
7. Decision Matrix and Rollout Steps
Close out the selection with a weighted decision matrix. Example dimensions and weights (adjust to your business):
| Dimension | Description | Example Weight |
|---|---|---|
| Task correctness | Pass rate on the business evaluation set | 30% |
| Structure and controllability | Format pass rate, tool-call success rate | 15% |
| Latency and stability | Tail latency, timeout rate, rate-limit rate | 15% |
| Cost per successful task | Including retries and amortized operations | 15% |
| Compliance and security | Data residency, auditing, filtering capability | 15% |
| Engineering compatibility | Integration cost, version management, observability | 10% |
The rollout process can be summarized as: requirement profile → candidate pool → business evaluation set → blind testing and stress testing → cost modeling → security and compliance review → small-traffic canary release → monitoring and alerting → periodic regression and replacement. The selection conclusion is not a permanent answer—it is a sustainable, iterative decision mechanism. For most teams, building the unified gateway and evaluation baseline first, then gradually expanding candidate models and scenarios, is far safer than chasing the “strongest model” in one shot.
