A Practical Guide to Selecting Domestic LLMs: From Capability Dimensions to Engineering Deployment

1. Why Selection Matters More Than Chasing the Newest Release

Over the past year, the release cadence of domestic large language models has accelerated noticeably. New flagship versions appear every few months, and leaderboard rankings shift frequently. In this environment, the most common question developers ask is: “Which one should I actually use?”

If you focus only on leaderboard positions or launch-event parameters, it is easy to fall into two traps:

• Blindly chasing new releases: Switching models the moment a new one ships causes frequent fluctuations in online services and makes regression testing extremely expensive.
• Relying on a single metric: For example, looking only at a score on one leaderboard while ignoring latency, cost, context length, tool-calling stability, and other key factors in real business scenarios.

The rational approach is to first clarify your business scenario and constraints, then build a quantifiable evaluation framework, and finally validate with small-scale stress tests and A/B experiments. This article addresses these layers and offers a practical selection methodology.

2. Four Constraints to Clarify Before Selection

Before comparing any models, answer the following four questions. They often determine the final choice more than “which model is stronger.”

2.1 What Is the Business Scenario?

Different scenarios impose very different requirements on model capabilities:

• Conversation and customer support: Emphasis on multi-turn context retention, instruction following, and response latency.
• Code generation and completion: Emphasis on code corpus coverage, long context, tool calling, and repository-level understanding.
• Document understanding and extraction: Emphasis on long-text processing and structured output stability.
• Agents and tool orchestration: Emphasis on stable function-calling formats and reliable multi-step reasoning.

2.2 Hard Requirements for Latency and Throughput

Online interactive scenarios typically require first-token latency in the hundreds of milliseconds, while offline batch processing can tolerate higher latency. Throughput directly determines unit cost. Both upper bounds must be defined before selection.

2.3 Cost Structure

Cost is not just the API unit price. Consider:

• The billing ratio between input and output tokens
• Whether you need to build your own inference cluster (GPU costs, operations costs)
• Actual discounts from cache hit rates and batch processing capabilities

2.4 Deployment and Compliance

• Whether data is allowed to leave the region
• Whether private deployment is required
• Whether industry-specific compliance requirements apply (e.g., finance, healthcare)

Once these constraints are fixed, the candidate pool often shrinks by more than half.

3. Six Technical Dimensions for Evaluating Domestic LLMs

The following dimensions can be used to compare any model horizontally, independent of specific brands.

3.1 General Reasoning and Knowledge

This includes mathematical reasoning, logical reasoning, and common-sense question answering. Public benchmarks are useful for initial screening, but do not rely on total scores alone—look at the subsets relevant to you. For code scenarios, focus on code-related benchmarks rather than general QA.

3.2 Long-Context Capability

Many models claim support for very long contexts, but “supported” and “effective” are two different things. What really needs testing:

• Whether key-information recall remains stable at 32K, 128K, or even longer inputs
• Whether information in the middle of long documents is “forgotten”
• Whether latency and cost at long context are acceptable

3.3 Instruction Following and Structured Output

For scenarios requiring JSON, XML, or fixed-format output, the stability of structured output is critical. You can construct a batch of test cases with schemas and measure the format error rate.

3.4 Tool Calling and Function Calling

In agent scenarios, models must reliably output call formats that conform to conventions. Key things to test:

• Whether multi-tool selection is correct
• Whether parameters conform to the schema
• Whether format drift occurs during multi-step calls

3.5 Ecosystem and API Compatibility

If a model provides an API compatible with mainstream interface specifications, migration costs drop significantly. This matters especially when you need hot-swapping between multiple models or implementing fallbacks.

3.6 Deployment Form and Observability

Whether private deployment is supported, whether quantized versions are available, whether common inference frameworks are compatible, and whether logging and monitoring metrics are comprehensive—all affect long-term operations costs.

4. A Practical Selection Workflow

Below is a complete workflow from initial screening to launch, suitable for small and mid-sized teams to apply directly.

Step 1: Build a Candidate Pool

Based on the constraints above, narrow down to 3–5 candidate models. Do not start by comparing a dozen models—that only increases workload.

Step 2: Construct a Business Test Set

Extract 100–300 representative samples from real business logs, covering:

• Typical success cases
• Edge and anomalous inputs
• Multi-turn conversations
• Scenarios requiring tool calls

Step 3: Run A/B Stress Tests with a Unified Interface

For fair comparison, wrap different models behind a unified abstraction layer. Below is a simplified Python example:

from dataclasses import dataclass
from typing import List, Dict
@dataclass
class ModelConfig:
    name: str
    endpoint: str
    api_key: str
    max_tokens: int = 4096
class ModelClient:
    def __init__(self, config: ModelConfig):
        self.config = config
    def chat(self, messages: List[Dict], temperature: float = 0.2) -> str:
        # Replace with the corresponding SDK or HTTP request in actual use
        raise NotImplementedError
def run_benchmark(clients, test_cases):
    results = {}
    for client in clients:
        latencies, outputs = [], []
        for case in test_cases:
            import time
            start = time.time()
            out = client.chat(case["messages"])
            latencies.append(time.time() - start)
            outputs.append(out)
        results[client.config.name] = {
            "avg_latency": sum(latencies) / len(latencies),
            "p95_latency": sorted(latencies)[int(len(latencies) * 0.95)],
            "outputs": outputs,
        }
    return results

The point of this code is not the implementation details but the principle: use the same test set and the same evaluation script to compare different models so that conclusions are comparable.

Step 4: Combine Human and Automated Evaluation

Automated metrics (such as exact match, BLEU, and format accuracy) cover only part of the picture. For generation quality, readability, and factuality, human spot checks are still needed. Double-blind review is recommended to reduce subjective bias.

Step 5: Gray Release and Rollback Plan

After choosing the primary model, do not switch all traffic at once. Start with 5%–10% of traffic and observe for a week. Keep a backup model ready to switch quickly if the primary model shows latency jitter or rising error rates.

5. Common Pitfalls and How to Avoid Them

Pitfall 1: Treating Leaderboards as the Only Basis

Leaderboards typically use public datasets that may differ significantly from your business distribution. The remedy is to build your own test set and use leaderboards only as an initial screening reference.

Pitfall 2: Ignoring Inference Cost

Some models perform slightly better but cost several times more in unit price or latency. In high-concurrency scenarios, this can make a project infeasible. Calculate the cost model early in the selection process.

Pitfall 3: No Version Management

Model services may update versions at any time, causing behavior changes. Record the model version at call time and run regression tests on critical business paths.

Pitfall 4: Overlooking Tool-Calling Stability

In agent scenarios, a single occasional format error can cause an entire workflow to fail. Strictly validate tool-call results and design retry and degradation logic.

6. A Future-Oriented Selection Strategy

The LLM landscape changes quickly, and today’s optimal choice may become obsolete within six months. Therefore, more important than “which model to choose” is building a sustainable capability for selection and switching:

• Use a unified abstraction layer to isolate model differences
• Continuously evaluate with standardized test sets
• Control risk with gray release and rollback mechanisms
• Drive decisions with observability data

With these in place, whatever new models emerge, the team can evaluate quickly and switch at low cost instead of being locked in to a single vendor.

7. Summary

Choosing among domestic LLMs is fundamentally an engineering problem, not merely a benchmark-scoring problem. Proceed in the following order:

1. Clarify business scenarios and hard constraints
2. Build a multi-dimensional evaluation framework
3. Run A/B stress tests with real data
4. Gray release while retaining rollback capability
5. Iterate continuously rather than deciding once and for all

Following this approach, developers can make more rational and explainable technical decisions among numerous candidates, and maintain architectural flexibility amid rapid model iteration.