A Practical Guide to Selecting Domestic Large Language Models: Evaluation Frameworks and Engineering Best Practices

1. Why Selection Has Become So Difficult

Over the past two years, domestic large language models have undergone a rapid shift from “do we have one?” to “which one should we choose?” New model versions are released every few months, with metrics such as parameter scale, context window, reasoning capability, and multimodal support constantly being refreshed. For developers and enterprise technical teams, this is both a blessing and a significant decision-making burden.

There are three core reasons why selection is so difficult:

First, evaluation dimensions are inconsistent. Different vendors emphasize different benchmarks—some focus on mathematical reasoning, some on code generation, and some on Chinese language understanding. Looking at a single score in isolation can easily lead to one-sided conclusions.

Second, business scenarios vary widely. A team building an intelligent customer service system and a team building a coding assistant have completely different requirements. The former cares more about response latency and concurrency costs, while the latter cares more about long context and logical rigor.

Third, the pace of iteration is too fast. A model chosen today may be surpassed by a new version three months later. Selection is no longer a “one-time decision” but a “continuous evolution” process.

Therefore, rather than asking “which model is the best,” it is more practical to build a reusable evaluation framework that allows teams to make quick judgments with every technology iteration.

2. Building a Selection Evaluation Framework

2.1 Capability Dimensions

When evaluating a model’s capabilities, consider the following directions:

• General reasoning ability: Includes logical reasoning, mathematical computation, multi-step reasoning, etc. Can be compared horizontally using standardized test sets.
• Code generation ability: If the business involves development assistance, focus on code completion, bug fixing, cross-file understanding, and similar capabilities.
• Chinese semantic understanding: Domestic models typically have a natural advantage in Chinese scenarios, but there are still gaps between different models, especially in specialized terminology, dialects, and classical Chinese.
• Long context handling: Context windows have expanded from early 4K to today’s 128K or even longer, but “supporting” and “using well” are two different things. Actual testing of long-document Q&A and information retention in multi-turn conversations is needed.
• Multimodal capabilities: Whether image understanding, document parsing, voice interaction, and other capabilities meet business requirements.
• Instruction following and formatted output: In scenarios requiring structured output (such as JSON or tables), the degree of model compliance directly affects engineering complexity.

2.2 Engineering Dimensions

Beyond capabilities, engineering implementation is equally critical:

• Inference latency and throughput: First-token latency and tokens per second directly affect user experience.
• Concurrency and rate limiting strategies: Whether API QPS limits and quota management meet business peak demands.
• Deployment options: Whether private deployment and containerized deployment are supported, and whether mainstream inference frameworks are compatible.
• Cost structure: Token-based billing, per-call billing, or annual/monthly packages—cost differences are enormous across different business models.
• Ecosystem compatibility: Whether common interface specifications are supported and whether integration with existing toolchains is easy.

2.3 Ecosystem and Sustainability

• Version iteration frequency: Whether the vendor has a stable update cadence.
• Documentation and community: Whether documentation is comprehensive and feedback channels are accessible.
• Compliance and security: Whether industry regulatory requirements are met and whether data handling is transparent.

3. Selection Strategies for Typical Scenarios

3.1 Intelligent Customer Service and Knowledge Q&A

The core requirements for this type of scenario are low latency, high concurrency, and stable Chinese understanding. The model does not need extremely strong reasoning ability, but it needs to accurately understand user intent and retrieve correct information from a knowledge base.

Selection recommendations:

• Prioritize medium-scale models with fast response times and controllable costs;
• Focus on testing context retention in multi-turn conversations;
• Combine with Retrieval-Augmented Generation (RAG) architecture to reduce reliance on the model’s own knowledge base.

3.2 Code Assistance and Development Tools

Code scenarios have high demands on the model’s long context and logical rigor. Developers want the model to understand the entire project structure, not just single-file snippets.

Selection recommendations:

• Test the model’s performance on cross-file references and function call chain analysis;
• Pay attention to the model’s coverage of mainstream programming languages and frameworks;
• Evaluate formatted output capability to ensure generated code can be used directly.

3.3 Content Creation and Document Processing

This type of scenario requires the model to have good language fluency and style control capabilities. At the same time, tasks such as long document summarization, translation, and rewriting have high requirements for context windows.

Selection recommendations:

• Test the model’s performance across different writing styles (technical documentation, marketing copy, academic papers);
• Pay attention to information loss when processing long documents;
• Evaluate whether custom style instructions are supported.

3.4 Data Analysis and Decision Support

Involving tasks such as structured data understanding, report generation, and trend analysis, these scenarios have high requirements for the model’s numerical reasoning and instruction following capabilities.

Selection recommendations:

• Test the model’s accuracy on tasks such as table understanding and SQL generation;
• Pay attention to whether the model “fabricates” data, i.e., the hallucination problem;
• Evaluate the difficulty of integration with existing data analysis tools.

4. Engineering Practices for Multi-Model Integration

In reality, few teams rely on only one model. A more common approach is multi-model integration with scenario-based routing. Below are some engineering practice recommendations.

4.1 Unified Interface Layer

Abstract a unified model calling interface to shield differences between vendor APIs. Illustrative code:

class ModelProvider:
    def chat(self, messages, **kwargs):
        raise NotImplementedError
class ProviderA(ModelProvider):
    def chat(self, messages, **kwargs):
        # Adapt to Vendor A's interface specification
        pass
class ProviderB(ModelProvider):
    def chat(self, messages, **kwargs):
        # Adapt to Vendor B's interface specification
        pass
class ModelRouter:
    def __init__(self):
        self.providers = {}
    def register(self, name, provider):
        self.providers[name] = provider
    def route(self, scenario):
        # Select model based on scenario
        if scenario == "code":
            return self.providers["provider_a"]
        elif scenario == "chat":
            return self.providers["provider_b"]
        else:
            return self.providers["default"]

The benefit of this approach is that when a model version is updated or needs to be replaced, only the adapter layer needs to be modified, without affecting upper-level business logic.

4.2 Canary Rollout and A/B Testing

Before launching a new model, it is recommended to gradually shift traffic through canary releases. Traffic can be split by user ID, request type, or random proportion to compare differences between old and new models in terms of latency, accuracy, user satisfaction, and other metrics.

4.3 Performance Monitoring and Alerting

Establish a comprehensive monitoring system covering at least the following metrics:

• Time to first token (TTFT)
• Tokens per second (TPOT)
• Request success rate and error code distribution
• Token consumption and cost trends
• Model output quality sampling

Once a metric becomes abnormal, it should be possible to quickly determine whether the issue is a network problem, rate limiting problem, or the model itself.

4.4 Caching and Fallback Strategies

For highly repetitive queries, semantic caching can reduce the number of model calls. At the same time, prepare fallback plans: when the primary model is unavailable, automatically switch to a backup model to ensure service availability.

5. Common Pitfalls in Selection

Pitfall 1: Blindly pursuing the largest parameter count. Parameter scale does not equal actual performance. Many medium-scale models perform better in specific scenarios with lower costs and smaller latency.

Pitfall 2: Looking only at benchmarks without considering scenarios. High benchmark scores do not mean the model works well in business scenarios. Testing with actual business data is essential.

Pitfall 3: Ignoring engineering costs. Model integration, prompt tuning, output parsing, and exception handling—these engineering tasks are often more time-consuming than the model selection itself.

Pitfall 4: Treating selection as a one-time decision. Model iteration is fast; it is recommended to re-evaluate quarterly to keep the technology stack up to date.

Pitfall 5: Neglecting compliance and data security. Especially in industries such as finance and healthcare, data handling methods and deployment models must comply with regulatory requirements.

6. Summary

Selecting a domestic large language model is essentially a multi-objective optimization problem. There is no absolute optimal solution—only the option that best fits the current business stage and team capabilities.

It is recommended that teams proceed through the following steps:

1. Clarify business requirements: Identify core scenarios, performance requirements, and budget constraints.
2. Build an evaluation set: Construct test sets using real business data rather than relying on public benchmarks.
3. Validate on a small scale: Select 2–3 candidate models for canary testing.
4. Engineer the integration: Reduce coupling through a unified interface layer and retain switching capability.
5. Continuously monitor and iterate: Establish a metrics system and a regular re-evaluation mechanism.

Selection is not the endpoint but the starting point of continuous optimization. Maintaining technical awareness and continuously adjusting based on business feedback is the key to staying competitive in the rapidly evolving large language model ecosystem.