Choosing a Domestic LLM: A Technical Framework from Inference Deployment to Application

In today’s landscape of rapidly evolving Large Language Models (LLMs), engineering teams are no longer asking “is there a model available?” but rather “which model is the best fit for our specific business constraints?” With a new generation of domestic foundational models recently released and integrated into mainstream inference infrastructure, model selection has become highly pragmatic. This article moves beyond simple parameter comparisons to offer a systematic evaluation and decision-making framework from an engineering perspective.

1. Core Principle: Select for “Best Fit,” Not “Strongest”

A common pitfall in model selection is relying solely on public benchmark rankings. However, real-world business scenarios are far more complex than benchmarks. A model excelling in general Q&A might struggle with structured extraction from long documents; a model impressive at code generation might deviate when following complex system instructions.

The first step in selection is defining your business scenario’s constraints. You need to answer:

• Inference latency sensitivity: Is this a real-time user interaction or offline batch data processing?
• Context length requirements: Are you processing a few conversational turns or analyzing dozens of pages of contracts or papers?
• Tool calling complexity: Does the model need to understand and invoke external APIs, databases, or code interpreters?
• Deployment environment constraints: Must it be privately deployed in an on-premise data center, or can it utilize public cloud APIs?

Documenting these conditions before evaluating model capabilities will rapidly narrow down your options.

2. Three Capability Tiers of Domestic LLMs and Selection Signals

Based on recent model releases and their performance on mainstream inference platforms, we can broadly categorize current domestic models into three tiers. Note that this is not a strict hierarchy but a reference based on comprehensive capability, ecosystem maturity, and specific scenario advantages.

Tier 1: General Flagship Models — Best for Complex Base Tasks

These models are characterized by massive parameter counts and balanced, standout performance in general capabilities like comprehensive knowledge, complex reasoning, multi-turn dialogue, and instruction following. They are typically the flagship products of their respective vendors and frequent top-rankers on benchmarks. For instance, several new models recently integrated into international mainstream inference microservice platforms fall into this category.

Selection Signal: If your business involves high-difficulty content generation (e.g., professional report writing, complex code generation) or requires strong logical reasoning (e.g., math problem solving, logical analysis) without being overly sensitive to cost, this tier is your primary choice. They are often labeled as “Flagship” or “Pro” versions in cloud providers’ model gardens.

Tier 2: High Cost-Efficiency Models — Best for Scaled Production Calls

Models in this tier maintain high capability levels while significantly reducing per-inference cost and latency through architectural optimizations like Mixture of Experts (MoE). They might slightly trail flagship models in extreme capabilities, but the experience gap is negligible for 90% of daily business scenarios.

Selection Signal: For massive text classification, extraction, summarization, or high-concurrency customer service bots, this tier is the perfect choice. Under cost constraints, their throughput advantages directly translate to business economics. Cloud providers often package these as “Lite” or “Turbo” versions.

Tier 3: Vertical-Specific Models — Best for Extreme Specialization

Some models do not aim to be generalists but are deeply optimized for specific domains (e.g., finance, legal, healthcare) or specific capabilities (e.g., OCR recognition, code translation, long-text understanding). These models have obvious “short boards” but might outperform general flagships in their specialized “long boards.”

Selection Signal: If your business has clear vertical domain attributes, such as preliminary legal document review or structured financial research reports, prioritize specialized models. Forcing a general flagship model into these tasks often yields suboptimal results and higher costs.

3. Four Key Dimensions of Technical Decision-Making

Once the tier is determined, the actual technical evaluation begins. The following four dimensions must be verified individually in practice.

1. Inference Cost and Resource Consumption: The Compute Math

This is the most hardcore metric. You must look beyond the API unit price (price per 1,000 tokens) and focus on the actual token consumption of your business. Some models have a low unit price but require longer prompts or more retries to achieve the desired effect, resulting in a higher overall cost.

Practical Advice: Select 100 real, typical samples from your business to build an evaluation set. Run inference using different candidate models and compare:

• Character count of generated results (Token consumption)
• Average number of requests required to reach a usable standard
• End-to-end latency per request

Calculate the “cost per completed business task” based on this data. This is far more reliable than simply looking at the unit price.

2. Long Context and Information Density: Beyond “Window Size”

“Supporting 128K context” has become standard, but the key is: as context grows, can the model still accurately locate critical information? This is the classic “needle in a haystack” test (inserting random facts into long text and querying the model). Many models boast large windows but have an effective attention range limited to the beginning and end.

Practical Advice: If you need to process long documents, thoroughly test the model’s information recall rate under long context. Place critical information in the middle of the document and see if the model can extract it accurately. Additionally, observe whether the model’s latency and cost growth curve remains flat as the context length increases.

3. Tool Calling and Structured Output: Agent Foundation Capabilities

Future applications will inevitably be Agent-based. Models must understand external tool definitions (Function Calling) and output structured call parameters. The gap here is significant: some models can precisely output according to JSON Schema, while others consistently “invent” their own formats.

Practical Advice: Build a mock environment containing multiple tools (e.g., weather query, calculator, database query). Test:

• Can the model accurately select the correct tool?
• Is parameter extraction correct? (e.g., extracting and invoking the weather query twice from “Check if Beijing or Shanghai is colder tomorrow”).
• What is the model’s success rate for complex instructions (e.g., “call three tools sequentially to complete the task”)?

4. Ecosystem and Deployment Flexibility: Avoiding Vendor Lock-in

Pay close attention to how easily a model can be deployed across various environments. Mainstream open-source models with open weights and compatibility with standard inference backends (like vLLM or TGI) are better choices. “Semi-closed” models accessible only via their proprietary cloud platforms carry high migration costs and lock-in risks unless they possess a cliff-like lead in capabilities. The recent integration of new domestic models into international mainstream inference microservices (such as NIM) is strong proof of ecosystem compatibility. The takeaway: prioritize models that comply with open standards (e.g., OpenAI API-compatible protocols) and can be loaded by mainstream inference frameworks. This ensures you can freely migrate between cloud vendors or between on-premise deployments and public clouds without being constrained.

4. Practical Path: A Three-Step Approach from Selection to Production

Step 1: Adopt a “Model Routing” Mindset

Do not attempt to solve every problem with a single model. Design a model gateway to route requests above your business logic. Route simple instructions (like greetings) to low-cost small models, and delegate complex creative tasks to flagship models. Mainstream cloud-native gateways (such as customized solutions based on Envoy or Spring Cloud Gateway) can achieve this via plugin mechanisms.

Step 2: Treat the “Evaluation Set” as Code Assets

Your evaluation set should be dynamic. Supplement it with new use cases upon every business upgrade. Automatically run the evaluation set after each batch of new model versions is released. The set should include not only “correct answers” but also “prohibited actions” (e.g., do not leak system prompts, do not output illegal content). This method allows you to quickly grasp whether a model upgrade improves or degrades performance.

Step 3: Predefine a “Cloud-Edge-Device” Collaborative Deployment Strategy

Determine deployment location based on constraints:

• Sensitive data scenarios: Utilize open-source model weights deployed on private cloud GPU resource pools, isolating data through the cloud provider’s private VPC environment.
• Cost-sensitive, public data scenarios: Directly use cloud inference APIs, leveraging their elastic scaling capabilities to handle traffic spikes.
• Ultra-low latency scenarios: Consider edge node deployment or on-device small models to compress first-token latency to under 100ms.

5. Conclusion: No “Best” Model, Only the “Most Suitable” Architecture

Returning to the original question: “which to choose?” The answer is: do not attempt to select a single “correct answer.” The flourishing of domestic LLMs grants us unprecedented options. The expert approach is mastering the trade-offs across the four dimensions above, then leveraging cloud-native technologies (containers, gateways, observability) to build a flexible, elastic, and replaceable AI application architecture. Within this architecture, “changing models” is merely a configuration update, not a project rewrite. This is the only correct posture to adapt to the current rapid iteration of AI technology.