Selecting a Domestic Large Language Model: A Developer’s Guide to Evaluation and Practical Adoption
This article offers a practical methodology for evaluating and selecting Chinese large language models (LLMs) from a developer’s perspective. Rather than relying on public leaderboards or price alone, it emphasizes a systematic approach built on task definition, deployment constraints, and a five‑dimension evaluation framework (task capability, performance, cost, engineering friendliness, stability and compliance). The guide walks through a concrete selection workflow, including building a custom evaluation set, running small‑scale pilot tests, load testing, and using an abstraction layer for model‑agnostic integration. It concludes that the best model is not the one with the highest benchmark score, but the one that best aligns with your tasks, cost structure, and engineering infrastructure.
Why “Choosing the Right Model” Matters More Than “Chasing the Newest”
In the past year, the pace of releases for Chinese large language models (LLMs) has accelerated dramatically. New models appear almost every month, and public leaderboards reshuffle constantly. When selecting a model, many teams fall into two extremes: “use whichever model ranks highest” or “use whichever one dropped its price first.”
Both approaches ignore the same fundamental fact: model selection is an engineering problem, not a leaderboard contest. Leaderboards reflect performance on certain public benchmarks, but your business scenario, calling patterns, cost constraints, and compliance requirements may differ greatly from how those benchmarks are set up. A model that tops general evaluation suites may not be the best for your specific vertical task; a cheaper model may end up costing more overall if it requires many retries or heavy post‑processing.
Therefore, this article does not aim to declare “which model is the best.” Instead, it provides a practical evaluation methodology that your team can execute on its own.
Before You Start: Answer Three Questions
Before diving into evaluation, clarify the following three questions — otherwise your evaluation will lack direction.
1. What is the core task?
Different tasks demand different model capabilities:
- Code generation and assisted programming: focus on code correctness, multi‑language support, context understanding, and compatibility with AI programming tools.
- Long‑context processing: focus on effective context length, information extraction from long documents, and consistency.
- Conversation and customer service: focus on instruction following, multi‑turn consistency, and response stability.
- Agent and tool calling: focus on function‑call accuracy, multi‑step planning, and error recovery.
The more specific your task definition, the more targeted your evaluation set – and the more trustworthy your conclusions.
2. What is the deployment model?
There are roughly three paths:
- Direct API calls: low startup cost and fast iteration; suitable for validation phases and small‑to‑medium traffic.
- Managed inference services via cloud platforms: for example, model services provided by major cloud vendors. These balance operational overhead with elastic scaling and often integrate well with object storage, message queues, monitoring, and alerting.
- Private deployment of open‑weight models: maximum control, but requires building your own inference cluster, managing GPU memory optimization, concurrency scheduling, and more. This suits scenarios with strict data privacy requirements.
3. What are the constraints?
These include budget ceilings, latency requirements (e.g., online customer service often demands first‑token latency below 1 second), peak concurrency, and data compliance rules (whether data is allowed to leave your environment). These constraints will immediately filter out some candidate models.
Five Core Evaluation Dimensions
1. Task capability: build your own evaluation set instead of relying on public leaderboards
Public leaderboards work as a preliminary filter – they can narrow dozens of candidates to three to five. But they are not the final criterion. The correct approach is to construct your own evaluation set:
- Extract 100–500 representative requests from real business traffic, covering typical scenarios, edge cases, and error‑prone cases.
- Decide how to grade outputs: for objective tasks (e.g., classification, extraction), compare directly to ground truth; for open‑ended tasks (e.g., copywriting), use human scoring or a “strong model cross‑review” approach.
- Ensure reproducibility: fix sampling parameters such as temperature and record model version numbers to avoid result drift caused by silent model updates.
A minimal batch evaluation script might look like this:
import json, timedef run_eval(model_client, cases, output_path):results = []for case in cases:start = time.time()resp = model_client.chat(messages=[{"role": "user", "content": case["prompt"]}],temperature=0.1,)results.append({"id": case["id"],"prompt": case["prompt"],"output": resp.text,"latency_s": round(time.time() - start, 2),"tokens": resp.usage,})with open(output_path, "w", encoding="utf-8") as f:json.dump(results, f, ensure_ascii=False, indent=2)
Once the evaluation results are saved, perform unified human scoring and statistical analysis.
2. Performance metrics: latency, throughput, and stability
- Time to first token (TTFT): critical for streaming output scenarios.
- Generation speed (tokens/s): determines how long users wait for long responses.
- Performance degradation under concurrency: a single request being fast does not guarantee stability under high concurrency. Use load‑testing tools to simulate realistic traffic patterns.
- Error rate and timeout rate: especially during peak hours – directly affects retry strategy design.
3. Cost model: look at total cost, not unit price
Estimate monthly cost based on real traffic. A simplified formula is:
Monthly cost ≈ daily requests × average input tokens × input price+ daily requests × average output tokens × output price+ extra cost from retries and failed requests+ (if private deployment) GPU depreciation + operation personnel cost
Also, leave room for cache optimization: using prompt caching for repeated prefixes, and routing frequent Q&A to an external knowledge base instead of stuffing all context into the prompt, can significantly reduce actual expenses.
4. Engineering friendliness
These points are often overlooked but heavily impact development experience:
- Does the API follow mainstream calling conventions? What is the migration cost?
- Does it support streaming output, function calling, or structured output constraints?
- Quality of SDKs and documentation; are error codes clearly defined?
- Observability on the cloud side: comprehensive call statistics, logging, monitoring, and alerting to facilitate troubleshooting in production.
Cloud platforms like Baidu AI Cloud typically offer model capabilities together with storage, messaging, monitoring, and other basic components within the same ecosystem. For teams that need to quickly build a full application, this can lower integration costs. Of course, evaluate this dimension against your team’s existing technology stack.
5. Stability and compliance
- Is the model version update policy transparent? Can you pin a specific version?
- Do content safety and data compliance meet the regulatory requirements of your industry?
- What are the service availability commitments and rate‑limiting policies?
A Recommended Selection Workflow
Combining the above dimensions, here is a practical selection process:
- Preliminary filtering: based on public leaderboards and community reputation, narrow candidates down to 3–5.
- Custom evaluation: build an evaluation set from real business data and run all candidates uniformly.
- Small‑scale pilot: integrate 1–2 models into a gray‑release environment to observe real latency, cost, and user feedback.
- Load testing and failure drills: simulate peak traffic to validate rate limiting, retry, and fallback strategies.
- Abstract integration layer: in code, create a model‑agnostic wrapper. For example:
class LLMGateway:def __init__(self, primary, fallback):self.primary = primaryself.fallback = fallbackdef chat(self, messages, **kwargs):try:return self.primary.chat(messages, **kwargs)except Exception:return self.fallback.chat(messages, **kwargs)
By isolating the specific model behind a gateway, future swaps or side‑by‑side comparisons only require configuration changes – not business‑code changes.
- Continuous evaluation: set up online performance monitoring and periodically re‑test with new samples to avoid silent degradation of model capabilities going unnoticed.
Conclusion
The overall capability of domestic Chinese LLMs has reached a highly usable stage, and the gap between top models is narrowing. This means the key to selection is no longer “which model is strongest” but “which model best fits your tasks, cost structure, and engineering ecosystem.” Use your own evaluation set instead of blindly trusting leaderboards, compare total cost instead of unit price, and maintain switching flexibility with an abstraction layer. Doing these three things will keep you proactive even as models continue to evolve rapidly.
