Domestic LLM Selection Guide: From Capability Evaluation to Engineering Deployment

1. Why “Choosing a Model” Has Become a New Challenge

Over the past two years, the iteration speed of domestic large language models has visibly accelerated. Top-tier models are now quite close in general capability, but their strengths differ significantly: some lead in code generation, some excel at long-context understanding, and others are more stable in tool-calling (Agent) scenarios. What makes this even more difficult is the release cadence — new models or new versions appear almost every month. A conclusion drawn last month may need to be revalidated this month.

At the same time, the channels for accessing models have also changed. In the early days, developers had to apply for and adapt to each vendor’s official API one by one. Now, “model marketplace” style inference service platforms have emerged: models are packaged together with their inference runtimes into standardized service units. The models on the shelf are continuously updated, while the API protocol remains stable. This is a real benefit for developers — the engineering cost of switching models has dropped significantly, and “parallel multi-model deployment with scenario-based routing” has moved from an architectural fantasy to routine practice.

The richer the tooling, the more you need a systematic selection methodology. Otherwise, it is easy to fall into the passive mode of “following the leaderboards.” Below is a reusable methodology organized around five dimensions.

2. Step 1: Break Down Your Scenarios Clearly

The starting point for model selection is not leaderboards — it is breaking your business down into a set of describable, measurable tasks. If you skip this step, all subsequent evaluations will be distorted.

Key elements to clarify include:

• Task type: text generation, code generation and completion, structured extraction, multi-turn dialogue, tool calling, long-document understanding, and multimodal processing. Different tasks place very different demands on model capabilities.
• Quality tolerance: generating marketing copy has high error tolerance, while generating SQL that will be executed has close to zero tolerance — for the latter, accuracy should carry far more weight than fluency.
• Latency requirements: online conversations care about time-to-first-token latency, while offline batch processing cares more about throughput and concurrency.
• Cost budget: under high concurrency, small differences in per-token pricing get amplified into significant cost gaps.
• Compliance constraints: whether sensitive data is involved, whether private deployment is required, and whether data can leave the local environment.

It is recommended to organize these elements into a decision matrix:

Dimension	Weight	Evaluation Method
Task accuracy	40%	Scoring on a self-built evaluation set
Response latency	20%	P95 time-to-first-token
Unit cost	20%	Estimated from real token distribution
Integration cost	10%	Migration development effort
Stability	10%	Rate-limiting policy and availability performance

Weights can be adjusted according to your business. The key is to transform the vague question of “which model is the best” into the quantifiable question of “which model achieves the highest composite score on my task distribution.”

3. Step 2: Use Evaluation Signals Correctly

3.1 Public Leaderboards: For Initial Screening, Not Final Decisions

The value of public evaluation leaderboards (general capability, coding capability, long-text capability, and so on) lies in quickly narrowing down the first tier. However, they have three inherent limitations. First, there is data contamination risk: training corpora may include evaluation questions, inflating scores. Second, there is distribution mismatch: the question distribution on a leaderboard differs from your business distribution, so a high leaderboard score does not mean the model works well for you. Third, there is timeliness lag: leaderboards often update more slowly than new version releases. The correct approach is to use leaderboards to narrow candidates from dozens down to three to five, and leave the final decision to a self-built evaluation.

3.2 Self-Built Evaluation Sets: The Highest-ROI Investment

Sample 100–500 items from real business logs and manually annotate ideal answers or acceptable answer sets to form a “golden set.” The goal is not a large scale; the key is covering typical scenarios and edge cases. A minimal viable evaluation framework looks like this:

import time
class EvalHarness:
    def __init__(self, model_client, judge_fn):
        self.model = model_client
        self.judge = judge_fn
    def run(self, dataset):
        results = []
        for case in dataset:
            t0 = time.time()
            answer = self.model.chat(case["prompt"])
            results.append({
                "case_id": case["id"],
                "score": self.judge(answer, case["reference"]),
                "latency": time.time() - t0,
            })
        avg = sum(r["score"] for r in results) / len(results)
        p95 = sorted(r["latency"] for r in results)[int(len(results) * 0.95)]
        return {"avg_score": round(avg, 4), "p95_latency": round(p95, 2)}

The judge_fn can be exact matching, rule-based validation (whether JSON parses, whether SQL passes syntax checks, whether the answer contains key information), or a “model-as-judge” approach. If you use a model as the judge, first calibrate its preferences against human annotations to avoid the judge’s own bias contaminating your conclusions.

3.3 Online A/B Testing: Let Real Traffic Speak

After offline evaluation, it is advisable to add a layer of online validation: route different models to different traffic buckets and observe real business metrics — adoption rate, manual edit rate, complaint rate, and task completion rate. Some capabilities (such as the “feel” of instruction following or whether the response style fits the product tone) are hard to quantify offline; online metrics are more honest.

4. Step 3: Understand Model Marketplace-Style Inference Services

Model inference service platforms that have emerged in recent years essentially package “model weights + inference runtime + standardized API” into ready-to-use service units. Their value is not in any single model, but in two things:

1. Unified protocol: multiple models are exposed through the same API protocol, so business code does not need to be aware of underlying model differences. Switching costs drop from “rewrite” to “change configuration.”
2. Flexible deployment: cloud-hosted APIs are suitable for quick starts and elastic scaling; containerized inference services can be deployed in private environments to meet data compliance requirements.

Major cloud providers, including Baidu AI Cloud, generally offer such model inference and hosting capabilities, often accompanied by tooling for fine-tuning, evaluation, and prompt management. When evaluating platforms, focus on these points: whether they cover your candidate models, context length limits, concurrency and rate-limiting policies, billing granularity, data compliance commitments, and availability metrics. These “platform-side constraints” are often overlooked, yet they tend to surface only after traffic scales up.

5. Step 4: Get the Cost Calculation Right

Token-based billing is far more nuanced than “looking at the unit price.” Several factors are easy to underestimate:

• Input and output prices differ: in most businesses, input tokens far exceed output tokens (long prompts, short answers). Estimate based on your real token distribution, not the 1:1 assumption in official examples.
• Context length is positively correlated with cost: packing your entire knowledge base into the prompt is usually a poor cost strategy; retrieval-augmented approaches are often more economical.
• Caching mechanisms: many services offer common optimizations such as prefix caching, which can yield significant benefits for highly repetitive calls.
• Hidden overhead: failed retries, timeouts, and rate-limit queueing all drive up actual costs. These need to be identified during load testing.

A straightforward estimation function:

def estimate_cost(avg_in, avg_out, calls, in_price, out_price):
    """in_price / out_price are per-thousand-token prices"""
    per_call = avg_in * in_price / 1000 + avg_out * out_price / 1000
    return per_call * calls

Put the estimates for all candidate models into the same table and compare them against evaluation scores to see “how much more it costs per point of accuracy gained.” The decision becomes much clearer.

6. Step 5: Architect with the Assumption That You Will Switch Models Within Six Months

No matter which model you choose today, you will most likely switch within six months. Therefore, the architecture should treat “switching models” as a normal event rather than an incident:

• Unified abstraction layer: business code should depend only on an internally defined ModelClient interface. Direct calls to vendor SDKs scattered throughout the codebase should be prohibited.
• Configuration-based routing: model selection goes into a configuration center, supporting scenario-based and percentage-based canary switches.
• Failure fallback: automatically fall back to a backup model when the primary model fails, ensuring availability.

Example implementation:

import logging
log = logging.getLogger(__name__)
class ModelGateway:
    """Try candidate models in priority order, fall back automatically on failure"""
    def __init__(self, candidates):
        self.candidates = candidates  # [(name, client), ...]
    def complete(self, prompt, **kwargs):
        for name, client in self.candidates:
            try:
                return client.chat(prompt, **kwargs)
            except Exception as e:
                log.warning("model %s failed, fallback: %s", name, e)
        raise RuntimeError("all candidate models unavailable")

This infrastructure also has a hidden benefit: integrating the golden evaluation set into your CI pipeline automatically runs regression tests whenever you switch or upgrade models, so capability regressions are caught before deployment.

7. Conclusion

Back to the original question: which domestic large language model should you choose? From an engineering perspective, the answer is: first break down your scenarios, use leaderboards for initial screening, make decisions with a self-built evaluation set, validate with online A/B tests, and preserve the freedom to switch at any time with a gateway architecture. Models will keep iterating, so model selection is not a one-time decision — it is an ongoing process. Building solid evaluation and switching infrastructure has far more long-term value than obsessing over “who is the strongest right now.”