Domestic LLM Selection Guide: A Practical Framework from Capability Dimensions to Production Deployment

1. Why Model Selection Is Harder Than Model Usage

Over the past year, domestic large language models have entered a period of rapid iteration. New versions ship every few weeks, leaderboard rankings shift frequently, and different vendors emphasize different strengths across reasoning, code generation, long-context processing, and multimodal capabilities. For developers, the question is no longer whether a model is capable enough — it is deciding which one to use.

A more practical reality is that model selection is not a one-time decision. It is an ongoing process that spans the entire project lifecycle. A single team may use multiple models simultaneously: lightweight models for low-complexity tasks, high-capability models for complex reasoning, and large-context models for long document processing. The right question is therefore not “which model is best,” but “how do we build a systematic selection methodology.”

This article proposes a reusable evaluation framework across four areas: capability dimensions, engineering constraints, cost structure, and implementation practices.

2. Capability Dimensions: Look Beyond Leaderboards

Leaderboards are an entry point, not the answer. Different benchmarks use different test sets, scoring methods, and focus areas, so a single ranking rarely reflects how a model will perform in real business scenarios. Evaluate models across the following dimensions instead.

2.1 General Reasoning and Instruction Following

This is the most fundamental capability layer. Focus on:

• Complex logical reasoning: multi-step reasoning, causal analysis, and mathematical word problems
• Instruction following precision: whether the model strictly follows output formats and avoids omitting constraints
• Hallucination resistance: whether the model tends to fabricate information when facing knowledge gaps, or clearly expresses uncertainty

Do not rely solely on standardized evaluation sets. Construct real prompts that closely match your business scenarios and observe how the model behaves in edge cases.

2.2 Long-Context Processing

Long-context support has become a baseline requirement, but “supporting 128K tokens” and “maintaining effective attention at 128K tokens” are two different things. When evaluating, distinguish between:

• Effective context window: the curve of information retrieval accuracy as input length grows
• Long document summarization quality: whether key details are lost or contradictions are introduced
• Multi-turn conversation memory: whether early constraints are forgotten during long dialogues

A practical technique is a variant of the “needle in a haystack” test: embed key information at different positions within a long text and observe whether the model can consistently recall it.

2.3 Code Generation and Engineering Capability

For developers, code capability is often the deciding factor. Evaluate at three levels:

# Level 1: Single function generation
# Given a docstring, observe correctness and style consistency of the generated code
# Level 2: Cross-file understanding
# Given a project structure and partial code, ask the model to complete a module or fix a bug
# Level 3: Engineering decision-making
# Ask the model to weigh trade-offs between multiple implementation approaches and explain its reasoning

The third level best distinguishes a model’s real engineering value — whether it can understand contextual constraints and propose maintainable solutions, rather than merely producing “code that runs.”

2.4 Multimodal and Tool Calling

If your business involves image understanding, document parsing, or agent scenarios, also evaluate:

• Granularity of visual understanding (charts, tables, handwriting)
• Format stability of function calling and tool calling
• Error recovery capability in multi-turn tool orchestration

3. Engineering Constraints: The Underrated Decision Factor

Capability is only the entry ticket. Whether a model can be deployed stably depends on engineering constraints.

3.1 Deployment Model and Data Compliance

• Public cloud API: fastest to integrate, but requires evaluation of data privacy and compliance requirements
• Self-hosted deployment: greater control, but demands compute resources and operational capacity
• Hybrid approach: sensitive data processed locally, general tasks routed to the cloud

3.2 Throughput and Latency

Inference efficiency varies significantly across models on identical hardware. Pay attention to:

• Time to first token (affects interactive experience)
• Output throughput (affects batch task costs)
• Concurrent load capacity (affects peak stability)

3.3 Context Caching and State Management

In multi-turn scenarios, whether context caching is supported and how cache hits are billed directly affects long-term costs. This is easy to overlook during selection but has significant impact after deployment.

4. Cost Structure: Account for Total Cost, Not Unit Price

The per-token API price is only one part of the cost. A complete cost model should include:

Cost Item	Description
Inference costs	Input/output token pricing, including tiered pricing
Caching costs	Storage and hit costs for context caching
Retry costs	Additional calls caused by failed responses and format corrections
Engineering costs	Integration, prompt tuning, monitoring, and alerting
Migration costs	Rework required when switching models

A common mistake is choosing the cheapest model based on unit price alone, only to face unstable output quality that leads to extensive retries and manual fallback — making the total cost higher in the end.

5. Implementation Practices: Building an Evolvable Selection Mechanism

5.1 Layered Routing Strategy

Do not try to solve every problem with a single model. Layer models by task complexity:

Simple classification/extraction  →  Lightweight model
Routine dialogue/summarization    →  Mid-tier model
Complex reasoning/code generation →  High-capability model

A routing layer provides unified dispatch, controlling costs while ensuring quality for critical tasks.

5.2 Build an Internal Evaluation Set

General-purpose leaderboards cannot replace business-specific evaluation. Recommendations:

1. Sample 100–500 evaluation cases from real business logs
2. Define quantifiable scoring criteria (accuracy, format compliance rate, human scoring)
3. Run regression tests whenever a model is updated or a new model is introduced

5.3 Abstract the Integration Layer

Regardless of which model you ultimately choose, add an abstraction layer between your business code and the model API:

class LLMProvider:
    def complete(self, prompt: str, **kwargs) -> str:
        raise NotImplementedError
class ProviderA(LLMProvider):
    def complete(self, prompt: str, **kwargs) -> str:
        # Adapt to the specific interface
        ...
# Business code depends only on the LLMProvider interface

This way, switching models only requires adding a new adapter rather than refactoring business logic.

5.4 Continuous Monitoring and Gradual Rollouts

After deployment, continuously monitor:

• Output quality drift (model version updates may cause behavioral changes)
• Latency and error rates
• Cost trends

When introducing a new model, start with a small traffic canary, validate against your internal evaluation set, and then gradually expand.

6. Conclusion

There is no universal answer to domestic LLM selection, but there is a repeatable methodology:

1. Capability evaluation should go beyond leaderboards and use real business prompts for multi-dimensional testing
2. Engineering constraints (deployment model, latency, compliance) often matter more to success than raw capability
3. Cost accounting should consider the total picture, including retry and engineering costs
4. Operational mechanisms such as layered routing, internal evaluation sets, and an abstraction layer turn model selection into an ongoing, evolvable process

Models will keep iterating, and today’s rankings may be rewritten tomorrow. What truly matters is not picking a “permanently correct” model, but building a technical mechanism that enables rapid evaluation, smooth switching, and continuous optimization.