Domestic LLM Selection in Practice: Unified Access and Scenario-Based Evaluation

1. A Bad Question and a Good Question

“Which domestic LLM should we choose?” is probably one of the most frequently asked questions among developers and enterprise teams over the past two years. But it is a classic “bad question” — not because it is unimportant, but because it has no stable answer.

First, model iteration cycles are far faster than selection cycles. Most vendors release major updates every few months, so a shortlist written today may become largely obsolete within a quarter. A “stick with one vendor” mindset is inherently unsuitable for this pace. Second, there is no all-around champion. Models differ significantly across code generation, logical reasoning, long-context understanding, structured output, and instruction following. A model that excels at coding may not stand out in open-domain conversation. Third, benchmark scores do not equal business scores. Public benchmarks carry risks of data contamination and overfitting, and the same model can perform orders of magnitude differently depending on prompts, sampling parameters, and context organization.

A more valuable approach is to turn “model selection” from a one-time decision into a sustainable mechanism. This mechanism rests on two pillars: unified access, which reduces switching costs to near zero so that even a wrong choice is merely a configuration change; and scenario-based evaluation, which uses real business data to determine which model fits which scenario. A major version update simply triggers a re-evaluation. This article focuses on these two threads.

2. Step One: Build a Unified Multi-Model Access Layer

2.1 Why Unified Access Is a Prerequisite for Selection

If every model is accessed in a different style, “comparing models” quickly degrades into “comparing patience.” Unified access brings at least four benefits:

• Fair comparison: With the same prompts, parameters, and decoding settings, performance differences can only come from the models themselves, eliminating interference from inconsistent invocation methods.
• Low switching cost: Changing providers means changing one configuration entry, not refactoring business code. During validation, you can rotate through multiple models within a single day.
• Risk hedging: If one channel rate-limits or fails, you can immediately switch to a backup, avoiding single-point dependency.
• Unified governance: Logging, cost tracking, permission control, and content safety checks are centralized at the access layer rather than scattered across business modules.

2.2 Integrating a New Model: Only Three Elements Required

Most mainstream model services have converged on a common pattern: HTTP endpoint plus API key authentication plus model ID. To integrate a new model, you typically need only:

• API key: The authentication credential generated in the provider console. It is the only sensitive item among the three and the primary focus of security management.
• Service URL: The base endpoint for the model service, determining where requests are sent.
• Model ID: The identifier for a specific model version. This is the only field you actually “switch.”

Configuration Item	Purpose	Management Recommendation
API key	Authentication	Inject via environment variables or a secrets manager; never hardcode or commit to a repository
Service URL	Request routing	Keep in external configuration so it can be changed at any time
Model ID	Version selection	Bind to evaluation results; replacement should not affect business logic

2.3 Minimal Invocation Example

The following language-agnostic pseudocode shows the skeleton for calling a generic chat completions-style API:

config = {
  "provider_a": {
    "url":   "https://<service_url_A>/chat/completions",
    "key":   env("PROVIDER_A_API_KEY"),
    "model": "model_id_from_provider_A"
  },
  "provider_b": {
    "url":   "https://<service_url_B>/chat/completions",
    "key":   env("PROVIDER_B_API_KEY"),
    "model": "model_id_from_provider_B"
  }
}
function chat(provider, messages):
    cfg = config[provider]
    headers = { "Authorization": "Bearer " + cfg.key }
    payload = {
        "model":       cfg.model,
        "messages":    messages,      # e.g. [{"role":"user","content":"..."}]
        "temperature": 0.7
    }
    resp = http.post(cfg.url, headers, payload, timeout=60)
    return resp.choices[0].message.content

With this abstraction layer in place, “choosing a model” is reduced to switching a parameter, and the upper-layer business logic remains completely unaffected. In production, you should also add timeout handling, retries with exponential backoff, and rate-limit fallback.

2.4 Self-Built Access Layer or Platform Service

There are two practical paths. One is building a lightweight access layer yourself. It requires very little code and gives you full control, making it suitable for development, debugging, prototyping, and small to medium teams. The other is using a platform service. Cloud providers often bundle model inference with key management, usage tracking, monitoring, and alerting — for example, Baidu AI Cloud offers such managed capabilities — which suits enterprise scenarios requiring unified governance, compliance auditing, and service guarantees. These two paths are not mutually exclusive: validating with a self-built layer and later migrating to a platform is a common progression.

2.5 Accept the Reality of Version Lag

Newly released model versions often take days or even weeks to become available on third-party gateways or aggregator platforms. This is a normal engineering phenomenon, not a capability flaw. A pragmatic approach is to configure and validate against the model versions actually available now, not against launch announcements. When the latest version is not yet available, use the previous version as a stand-in and run a regression after the update arrives. Maintain a “model ID — passed evaluation” mapping table to avoid the mismatch of testing one version and deploying another.

3. Step Two: Build Your Own Evaluation System

3.1 Do Not Copy Public Leaderboards

Leaderboards serve only as a rough reference for capability positioning, not as a direct selection basis. Benchmark datasets are public and fixed, creating contamination risks from targeted training. The input distribution of public benchmarks differs greatly from your real business inputs. Leaderboards cannot measure engineering metrics or the quality of your prompt engineering.

3.2 Build an Evaluation Set from Your Business

Start by decomposing business requirements into task types such as intent recognition, information extraction, summarization and rewriting, copy generation, code generation, open-domain QA, and tool calling. Then extract dozens of samples for each type from real production logs — fifty test cases from your own business are worth far more than any public dataset. Finally, prepare reference answers or clear scoring criteria for each case. Otherwise, “good or bad” becomes a matter of personal opinion.

3.3 Use Three Evaluation Methods Together

1. Rule-based automatic metrics: Use exact match for extraction tasks, test pass rate for code tasks, and format compliance rate for structured output. These cost zero human effort and are suitable as the main framework for continuous re-evaluation.
2. Blind scoring: Feed the same input to multiple candidate models, anonymize and shuffle the outputs, and have human reviewers or a scoring model rate them. Relative scoring is more stable than absolute scoring and effectively suppresses position bias and length bias.
3. Error attribution: Maintain an error sample library and periodically categorize failures. Distinguish whether an error stems from the model’s capability ceiling or from poor prompt design — the latter should not be counted against the model.

3.4 Measure Capability Metrics and Engineering Metrics Together

Capability scores are only half of production readiness. The other half is engineering performance: time to first token and output speed determine the interaction experience; context window size and long-document retrieval stability determine the feasibility of long-document tasks; per-call pricing and rate limits determine cost structure and capacity planning; availability and disaster recovery determine the online service level. Both categories should go into the same decision table during selection.

3.5 Establish a Re-Evaluation Mechanism

Subscribe to major model version updates as trigger events for your evaluation pipeline: an update triggers a re-evaluation. If conclusions change, produce a selection recommendation and update the configuration accordingly. In this way, every new model release becomes just another “button-triggered re-evaluation” rather than a new round of anxiety.

4. Scenario-Based Selection: Division of Labor, Not a Champion

• General conversation and content creation: Prioritize cost-effectiveness. Mid-tier models are usually sufficient for batch generation needs. Reserve flagship capabilities for a small number of difficult scenarios.
• Code development: Prioritize reasoning and coding dimensions, and build automated test cases against your own codebase. Your repository tests are more convincing than leaderboard pass rates.
• Long-document processing: Focus validation on context length and long-document retrieval stability. Build your own “needle in a haystack” test cases and watch for quality degradation under ultra-long contexts.
• Structured output and tool calling: Focus on format compliance rate and function calling stability. These are make-or-break for agent applications.
• Data-sensitive industries: Prioritize private deployment or compliant regional options. Ensure the contract explicitly states that data will not be used for model training, and establish data flow auditing.
• High-concurrency production environments: Throughput, concurrency limits, stability, and cross-region disaster recovery should carry more weight than per-call capability differences.

5. Engineering Rollout Checklist

• Model layer abstraction: Business code depends only on a unified chat interface, not deeply coupled to any single provider.
• Externalized configuration: Model IDs and service URLs go into a configuration center and take effect on release. No hardcoding.
• Multi-channel failover: Models with comparable capabilities serve as mutual backups, with automatic fallback switching on errors or rate limits.
• Key security: Store keys in environment variables or a secrets manager, apply least privilege, and rotate regularly.
• Cost control: Cache frequent repeated queries, route simple tasks to smaller models and difficult tasks to flagship models, and set budget alerts.
• Observability: Log model, latency, usage, and output quality spot-check results per request, with automatic anomaly alerts.

6. Conclusion

“Which domestic LLM should we choose” has never been a one-time multiple-choice question. It is a question of ongoing operational mechanism: unified access reduces switching costs to near zero, self-built evaluation makes judgments evidence-based, and scenario-based division of labor lets each model do what it does best. No matter who releases the “strongest model” tomorrow, your system will always be only one configuration change away — and that is the answer that actually transfers.