How to Choose a Domestic LLM: A Practical Evaluation Framework

1. The Big Picture: Domestic Models Are Now Worth Serious Evaluation

For the past two years, the top of various LLM benchmark leaderboards was long dominated by overseas open-source models. But recently, a notable shift has occurred: mainstream model inference service platforms—think of them as “model supermarkets”—no longer stock only overseas open-source models. A batch of newly released flagship domestic models has begun appearing on their shelves, callable directly through a unified interface.

The signal here matters more than the event itself. These platforms typically package models into standardized inference services: containerized deployment, unified authentication, unified billing, unified monitoring. A model being “placed on the shelf” means it has met the basic thresholds for service stability, interface standardization, and engineering usability. Combined with the fact that top domestic models now consistently rank in the first tier on multiple public code and web development benchmarks, the conclusion is clear: the question has shifted from “Can domestic models be used?” to “Which one should my business choose, how should I choose it, and how do I validate the choice?”

This article addresses all three questions with a practical selection methodology.

2. Before Selecting: Figure Out How You’ll Access the Model

Many people jump straight into comparing model scores, but the first step should be clarifying your access path, because it determines your evaluation costs and future migration costs. There are currently three main approaches:

1. Direct API from the model provider. This gives you the latest capability versions and the most complete parameter documentation, but different providers vary in interface details, rate-limiting policies, and billing structures. If you’re evaluating multiple providers simultaneously, you’ll need to maintain multiple sets of adapter code, which hurts evaluation efficiency.

2. Model inference service platforms (aggregated access). These platforms expose multiple models through a unified interface, with largely consistent request structures (message list + model name + sampling parameters). Switching models usually requires changing just one field. Their value lies in three areas:

   • Low cost for horizontal comparison—one set of evaluation code can run through all candidate models;
   • Architecturally natural support for multi-model redundancy—fast switching when the primary model fails or degrades;
   • Unified billing, monitoring, and alerting, which simplifies cost accounting and capacity planning.

3. Private / self-hosted deployment. Teams with high data compliance requirements (finance, healthcare, government, etc.) deploy open-weight models on their own infrastructure. The trade-off is that you must solve inference performance optimization (quantization, dynamic batching, KV caching), elastic scaling, and operational monitoring yourself.

A pragmatic path combines these: use an aggregation platform for rapid evaluation and small-traffic validation early on; once you’ve identified your primary model, evaluate whether to switch to direct API access or private deployment in exchange for lower unit costs or stronger data control.

3. Five Core Dimensions: Turning “Which Model Is Better” into a Quantifiable Evaluation Table

3.1 General Capabilities: Use Leaderboards for Screening, Not Decisions

Benchmark scores for general knowledge, mathematical reasoning, and instruction following are suitable for a first round of rough screening—for example, narrowing a dozen candidates down to three or five. But watch out for two traps:

• Data contamination: Benchmark questions may have been mixed into some models’ training data, inflating scores artificially;
• Distribution bias: The task distribution in benchmarks often differs dramatically from your business distribution. Ranking first on a leaderboard doesn’t mean ranking first in your scenario.

The correct approach: use leaderboards to narrow the field; use your own business test set for the final verdict.

3.2 Code and Engineering Capabilities: Hard Requirements for Agents and Coding Assistants

If your use case involves coding assistants, automated agents, or web development, the weight on code capability should increase significantly. Evaluation points include:

• Correctness and executability rate of complete code blocks, not just “looks right”;
• Cross-file, cross-context modification ability—when you change a function, does the model know to update its callers?
• Stability of function calling: whether parameter structures are valid and whether the model can self-correct after failures. This directly determines the reliability of agent pipelines;
• Stability of structured output (JSON, etc.). Parse failure rates are a pitfall many teams have encountered.

3.3 Context Capabilities: Look at “Effective Context,” Not Just the Nominal Window

The nominal context window is only one aspect. More critical is actual performance under long inputs: does instruction-following degrade, is mid-sequence information effectively utilized (needle-in-a-haystack tests), does output quality decline. Also note that the cost of long inputs rises non-linearly—not every task justifies filling the context. Retrieval augmentation plus summary compression is often a more economical alternative.

3.4 Cost and Throughput: Unit Price Is Only the Visible Cost

Cost calculation should account for at least four items:

• Whether input and output tokens are priced separately—the price gap can be significant;
• Whether cache hits receive discounts, which heavily impacts multi-turn conversation scenarios;
• Actual throughput under concurrency and rate limits—load test data is more realistic than a price sheet;
• Time to first token and overall latency, which directly affect interactive experience.

Run a load test with your own real workload. Calculate “unit price × real token distribution × peak concurrency” clearly, rather than comparing public price sheets directly.

3.5 Engineering and Compliance: Determines Whether You Can Go to Production

• Historical service availability and error rates, plus degradation plans for failures;
• Data compliance: whether data will be used for training and whether it meets your industry’s regulatory requirements—a hard threshold for finance and healthcare teams;
• Ecosystem compatibility: whether it’s compatible with industry-standard interface formats and can be smoothly integrated with mainstream development frameworks;
• Hosting and operations: if planning a cloud route, you can include platform services like Baidu AI Cloud in your evaluation scope, focusing on whether general capabilities such as inference services, monitoring and alerting, and permission auditing meet your team’s requirements.

4. Hands-On Validation: Building Your Own Evaluation Pipeline in Three Steps

Step 1: Build a Business-Representative Test Set

Sample from real business logs. A scale of one hundred to several hundred cases is recommended, covering four sample types: high-frequency routine requests, long-tail complex requests, boundary and adversarial examples, and real failure cases from historical incidents. Each sample needs a verifiable expected result: exact match, key-point checklist, or human scoring rubric.

Step 2: Unified Interface Batch Calls

Below is illustrative Python pseudocode showing the core idea of “one set of code evaluating multiple models”:

import time, json
MODELS = ["model_a", "model_b", "model_c"]   # Candidate model list
CASES  = json.load(open("eval_cases.json"))  # Business test set
def call_model(model, prompt):
    # Use a unified-format inference service; only the model name field changes
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0.2,   # Fixed sampling parameters during evaluation
    }
    start = time.time()
    resp = unified_client.chat(payload)
    return {
        "text": resp["choices"][0]["message"]["content"],
        "latency_ms": round((time.time() - start) * 1000, 1),
        "tokens": resp.get("usage", {}),
    }
results = []
for m in MODELS:
    for case in CASES:
        r = call_model(m, case["prompt"])
        results.append({"model": m, "case_id": case["id"], **r})
json.dump(results, open("raw_results.json", "w"), ensure_ascii=False)

A few variable-control details: use the same test set, the same sampling parameters (temperature, max output length), and ideally call during the same time window; run each model two to three rounds and take the average to smooth out random fluctuation.

Step 3: Scoring and Decision-Making

• Automated metrics: Key-point coverage rate, code executability rate, format parse success rate, latency distribution, token consumption;
• Human blind evaluation: Shuffle output order so raters don’t know which model produced which result, avoiding brand halo effects and preconceptions;
• Weighted aggregation: Combine by business weight, e.g., “correctness 50% + latency 20% + cost 20% + format stability 10%.” Weights must come from your business priorities, not someone else’s leaderboard.

Teams with the resources should add a final online canary round: route a small portion of real traffic to candidate models and observe behavioral metrics such as adoption rate, rework rate, and user feedback. Let real data make the final call.

5. Deployment Architecture: Leave Room for Switching Models

Model selection is not a one-time action. The pace of model iteration means you’ll likely switch foundations once or twice within a year. For deployment:

• Add a model gateway layer: Business code connects only to the gateway, which routes to specific models behind it. Switching models requires zero changes to business code;
• Dual-model strategy: Primary + backup model, with automatic degradation when the primary fails or degrades, avoiding single-point dependency;
• Tiered routing: Route complex reasoning to flagship models and simple tasks to lightweight models—a common cost-reduction technique;
• Continuous evaluation: Integrate test sets and evaluation scripts into CI. When a new model version is released, automatically run a round and let data decide whether to upgrade.

6. Common Pitfalls Checklist

1. Looking only at overall leaderboard scores, not sub-scores for your scenario;
2. Ignoring capability degradation under long context, leading to failures in long-document scenarios after launch;
3. Underestimating interface adaptation and migration effort, turning a model switch into a two-week refactor;
4. Load testing only throughput but not failure recovery, leaving no Plan B when rate-limited at peak;
5. Keeping the evaluation set static—models “memorize” the test questions and evaluation gradually loses validity.

7. Summary

Domestic large models reaching the first tier is an established fact, but “which model is best” still has no standard answer—only “which is most suitable for your business.” Treat model selection as an engineering project: leaderboard screening, five-dimension evaluation, business test set blind evaluation, online canary, gateway-based deployment. Complete these five steps, and your decision will be data-backed, reviewable, and rollback-capable. That’s more reliable than any reputation or hype.