A Practical Guide to Selecting Domestic LLMs: From Capability Dimensions to Engineering Deployment

1. The Nature of the Selection Problem: Matching, Not Ranking

When faced with a growing number of domestic large language models, the question developers ask most often is “which one should I pick?” But the question itself is vague—discussing strengths and weaknesses outside a specific scenario makes any conclusion meaningless. What actually needs to be answered are three progressive questions:

1. What capabilities does the task require? Is it long-document understanding, code generation, multi-turn dialogue, or structured extraction?
2. What are the engineering constraints? What are the acceptable latency, throughput, per-call cost, and data compliance boundaries?
3. In what form is the model delivered? Self-hosted inference, a managed API, or access through a unified model service platform?

Break these three questions apart, and model selection shifts from “going with your gut” to an operable decision process. This article follows that process and provides an abstract-level summary of the capability characteristics of current mainstream domestic models, helping readers build their own evaluation framework.

2. Capability Dimensions: Which Metrics to Compare

2.1 General Reasoning and Knowledge Coverage

General reasoning ability is typically observed through public benchmarks, such as comprehensive knowledge Q&A, mathematical reasoning, and code evaluation leaderboards. Note that leaderboard rankings can only serve as an initial screening reference, for three reasons: first, benchmark questions deviate from real business distributions; second, evaluation configurations (prompts, sampling parameters, whether tools are used) significantly affect results; third, some models have been optimized for specific benchmarks, and their generalization may not keep pace.

A pragmatic approach: shortlist several first-tier candidates from the leaderboards, then build a small evaluation set using your own business data (50–200 samples is enough) and compare them under unified prompts and parameters. The return on this step far exceeds repeatedly consulting leaderboards.

2.2 Context Window and Long-Text Processing

Long context has been one of the key competitive directions for domestic models in recent years. When evaluating, don’t just look at the nominal token limit—also pay attention to:

• Effective context: as input length grows, does the model’s recall of mid-section information degrade significantly (the “lost in the middle” phenomenon)?
• Stability under long text: does very long input make format corruption, repetitive generation, or instruction forgetting more likely?
• Non-linear cost: long context usually brings higher memory usage and inference latency, which must be estimated against actual call volume.

For document Q&A, contract review, and codebase understanding scenarios, it’s worth constructing test cases specifically where “key information is located in the middle of the document.”

2.3 Tool Calling and Structured Output

If the model needs to connect to external systems (retrieval, databases, business APIs), then function calling capability is a hard requirement. Evaluation points include:

• Adherence to tool descriptions: can it stably produce valid JSON under a given schema?
• Multi-tool orchestration: does it choose correctly when facing multiple candidate tools?
• Parameter-filling accuracy: especially for error-prone fields like enum values, date formats, and nested objects;
• Error recovery: can it reasonably retry or degrade when a tool returns an exception?

A simplified calling example is shown below, with the focus on constraining the output structure:

tools = [{
    "name": "query_order",
    "description": "Query order status by order ID",
    "parameters": {
        "type": "object",
        "properties": {
            "order_id": {"type": "string", "pattern": "^[A-Z0-9]{12}$"},
            "fields": {"type": "array", "items": {"type": "string"}}
        },
        "required": ["order_id"]
    }
}]
resp = client.chat(model=model_name, messages=msgs, tools=tools,
                   tool_choice="auto", temperature=0)

When evaluating, measuring the valid JSON ratio, field completeness rate, and end-to-end task success rate is more valuable than simply looking at “whether the tool was called.”

2.4 Code Capability

Code scenarios need to distinguish between two types of requirements: code completion (low latency, short context, high-frequency calls) and code understanding/generation (long context, complex reasoning, low-frequency high-value). The former is extremely sensitive to latency, while the latter demands greater reasoning depth. When selecting, test separately: unit test pass rate, cross-file dependency understanding, and adherence to the project’s existing code style.

3. Delivery Form: Why Managed Services Deserve Attention

Comparing model capabilities is only the first step. Whether a model can run stably and compliantly in production depends on the delivery form. There are currently three main paths:

Form	Characteristics	Suitable Scenarios
Self-hosted inference	Data stays in-domain, highest controllability; requires bearing GPU operations, scaling, and version upgrades yourself	Data-sensitive, high call volume, dedicated ops team
Vendor-hosted API	Ready out of the box, pay-as-you-go; data must leave the domain, subject to terms of service	Rapid validation, volatile traffic, no ops investment
Unified model service platform	Aggregates multiple models, standardized interfaces, facilitates horizontal comparison and switching	Multi-model routing, canary comparison, reducing lock-in risk

It’s worth noting that model inference service platforms that have emerged in recent years—infrastructure providing inference capabilities for multiple models as standardized microservices—are a typical example of the third form. The value of such platforms isn’t “one more shelf” but rather:

• Unified interface: the same request protocol adapts to multiple models, reducing switching cost from “changing code” to “changing configuration”;
• Managed operations: underlying inference optimization, batch scheduling, and health checks are handled by the platform;
• Canary capability: route a small amount of traffic to a new model and use real data for A/B comparison, rather than deciding based on leaderboards.

For enterprise users, another benefit of such platforms is reducing lock-in risk—when the model capability landscape changes rapidly, maintaining “replaceability” is itself an architectural capability. It’s recommended to abstract model calls into an internal gateway layer at the architecture design stage, shielding specific model differences:

class ModelGateway:
    def __init__(self, router_config):
        self.router = router_config  # route by scenario/cost/latency
    def chat(self, scene: str, messages, **kwargs):
        target = self.router.pick(scene)
        return target.invoke(messages, **kwargs)

This way, whether replacing models, adjusting versions, or distributing traffic across multiple services, no business code needs to change.

4. Cost Structure: Don’t Just Look at Unit Price

Model invocation cost includes at least four parts:

1. Token unit price: input and output are usually priced separately, and output is often more expensive;
2. Context amplification effect: long system prompts, multi-turn history, and retrieved concatenated content significantly inflate input tokens;
3. Retry and failure cost: retries caused by structured output parsing failures multiply actual overhead;
4. Engineering cost: human effort for integration, evaluation, monitoring, and fallback plans.

The recommended optimization order: first compress prompts and history windows (most direct benefit), then optimize output format constraints (reduce retries), and only then consider switching to a cheaper model. Many teams can recover over 30% of cost at the first step alone.

5. Compliance and Data Boundaries

Enterprise selection must answer: can data leave the domain? How is it retained after leaving? Is it used for training? There’s no technical shortcut here—you can only check the terms of service and data processing agreements item by item. For finance, healthcare, and government scenarios, private or dedicated instance deployment is usually required; in such cases, self-hosted inference or dedicated managed forms are necessary options. It’s recommended to set a “veto” weight for compliance items in the selection scorecard to avoid rework later.

6. A Reusable Selection Workflow

Combining the above dimensions, the following workflow can be put into practice:

1. Define scenarios and hard constraints: list task types, latency ceilings, data compliance requirements, and budget ranges;
2. Initial screening via leaderboards: shortlist candidates, keeping it within 3–5;
3. Build your own evaluation set: construct 50–200 test cases from real business samples, covering normal and edge cases;
4. Unified gateway access: call candidate models in parallel through standardized interfaces to ensure consistent evaluation conditions;
5. Multi-dimensional scoring: weight five items—capability, latency, cost, stability, and compliance—with weights confirmed by the business side;
6. Small-traffic canary: route 5%–10% of production traffic and observe real metrics;
7. Establish an exit mechanism: preserve switching capability, re-evaluate regularly, and avoid hidden regressions caused by model version upgrades.

7. Conclusion

The capability gap among domestic large models is rapidly converging, and first-tier models can already meet production requirements on most general tasks. The decisive factor in selection is increasingly not “which model is smarter,” but rather whether the evaluation methodology is solid, whether the architecture preserves replaceability, and whether cost and compliance are clearly accounted for. Abstract model calls into a gateway, turn evaluation into a repeatable process, and minimize switching costs—once these three things are done well, which specific model to choose becomes an operational decision you can adjust at any time, rather than a one-time bet.