How to Choose a Domestic LLM: Model Selection and Deployment from an Inference Serving Perspective

In the era of rapidly iterating deep learning technology, large language models (LLMs) have moved from the lab into production environments and become core components of many business systems. However, with the endless emergence of domestic models, technical teams often face a common dilemma during selection: models with similar parameter sizes and comparable public benchmark scores may differ dramatically in actual inference latency, throughput, and cost after deployment. This article aims to strip away marketing rhetoric and establish a scientific methodology for LLM selection from an engineering deployment perspective.

1. Understanding the Essence of Model Serving: The Leap from “Model” to “Service”

Before discussing which model to choose, one concept must be clarified: the model itself and model serving are two different things. Downloading an open-source model’s weight files locally is merely the first step of a long journey. To make a model truly serve business needs, it must go through a series of steps including quantization, compilation optimization, inference serving encapsulation, concurrent scheduling, and elastic scaling.

The current mainstream industry approach is to expose capabilities through a “inference microservice” architecture. An inference microservice encapsulates the model as a standard API interface, so callers do not need to concern themselves with underlying GPU resource management and inference engine details. This model resembles a “model supermarket”: shelves are stocked with various base models, and developers take what they need and pay by usage.

For domestic models, whether they can smoothly integrate into this system directly determines the difficulty of deployment. Specifically, three dimensions need to be examined: engine compatibility (whether it can run on mainstream inference frameworks), hardware adaptability (the degree of optimization for domestic and mainstream GPUs), and serving maturity (whether it supports production-grade features such as streaming output, batch processing, and dynamic batching).

2. Differentiated Positioning and Technical Profiles of Major Domestic Models

At present, domestic large models present a landscape of “multiple strong contenders coexisting,” but their technical routes and areas of expertise are clearly differentiated. To make a reasonable choice, one must first understand their genetic differences.

1. General Conversation and Instruction-Following Models

These models are optimized with dialogue interaction as the core objective, performing well in intent understanding, context coherence, and multi-turn dialogue consistency. They typically use large-scale SFT (supervised fine-tuning) and RLHF (reinforcement learning from human feedback) for alignment, making them suitable for scenarios such as intelligent customer service, virtual assistants, and educational tutoring. In terms of technical architecture, their design places greater emphasis on first-token latency (TTFT), because chat experience is extremely sensitive to “perceived speed.”

2. Reasoning and Code Generation Models

Another category of models focuses on strengthening logical reasoning and code generation capabilities. They introduce large amounts of code corpora and chain-of-thought data during training, excelling in mathematical problem-solving, complex logical reasoning, and code completion and repair. These models have extremely high requirements for “output throughput,” because code generation often involves long sequence outputs. When selecting such models, special attention should be paid to their context window length and attention mechanism efficiency under long sequences. If a model supports sparse attention or linear attention variants, it will have a clear performance advantage in long-text processing.

3. Multimodal and Ultra-Long-Context Models

Multimodal models not only process text but also support image, audio, and even video inputs. Ultra-long-context models, on the other hand, focus on understanding and summarizing thousand-page-level documents. These two technical directions often require larger memory footprints and more complex inference pipelines. During selection, one must clearly recognize that the introduction of multimodal capabilities significantly increases first-token latency, because the forward computation of the visual encoder (such as ViT) cannot be omitted.

3. Core Evaluation Framework for Technical Selection

Accurately assessing whether a model is suitable for a production environment requires far more than looking at benchmark scores. It is recommended to build an evaluation system from the following four levels:

(1) Inference Performance Benchmark

Under the same hardware conditions, construct a stress-test set using your own real business data. Core metrics include:

• TTFT (Time to First Token): The cornerstone of user-perceived waiting time; the target should be less than 300ms.
• TPOT (Time per Output Token): Affects generation speed; the target value depends on the scenario, with real-time chat requiring less than 50ms/token.
• Throughput (Tokens/s): The number of tokens generated per second per card or per instance, determining service capacity per unit cost.

Practical advice: Do not blindly trust officially published evaluation data, because evaluation environments often use top-tier H800 or A100 clusters, and batch sizes are carefully tuned. Always conduct actual tests under “simulated real concurrency” conditions.

(2) Ecosystem Compatibility and Deployment Cost

The key consideration is whether mainstream inference engines are supported (such as TensorRT-LLM, vLLM, MindIE, etc.). A model that cannot be optimized and compiled by mainstream engines will struggle to deliver performance in production even if its quality is excellent. Deployment cost requires comprehensive calculation:

• Memory footprint: A 7B model after INT8 quantization requires approximately 8GB of memory, while a 70B model even with FP8 quantization requires more than 70GB, determining how many cards you need for interconnect.
• Cluster scale and utilization: Evaluate whether PD separation (Prefill/Decode separation) is supported, as this technology can significantly improve GPU utilization and reduce per-request cost.

(3) Data Security and Private Deployment Capability

For industries such as finance, government, and healthcare, keeping data within domain boundaries is a bottom line. Therefore, it is necessary to evaluate the model’s degree of open-source availability and local deployment license. If a model is fully open-source and allows private deployment, then control remains in your own hands. In addition, examine whether federated fine-tuning or low-rank adaptation is supported, so that private data can be used to enhance the model in a targeted way while protecting data privacy.

(4) Continuous Evolution and Community Activity

Models are not static; selection should consider “future potential.” A model with an active community and continuous version iteration often has faster bug fixes and more complete toolchains. In this regard, ecosystem prosperity has greater long-term value than a single technical breakthrough.

4. Scenario-Based Selection Decision Matrix

Business Scenario	Primary Metric	Secondary Metric	Recommended Technical Route
Real-time voice conversation	TTFT < 200ms	Low hallucination rate	General conversation model, INT8 quantization, with streaming interface
Code auto-completion	TPOT < 40ms	Syntax accuracy	Code reasoning model, PD separation architecture, supporting FIM fill-in-the-middle mode
Long-text knowledge base Q&A	Context ≥ 128K	Retrieval recall rate	Ultra-long-context model, equipped with vector database and reranker
Multimodal content moderation	Throughput > 500 token/s	Multimodal recognition accuracy	Multimodal model, batch inference optimization, with asynchronous message queue

5. From Prototype to Production: A Deployment Practice Guide

After selecting a model, the real challenge begins with engineering. Below is a proven deployment workflow:

Step 1: Build an offline evaluation baseline
Randomly sample 1,000 examples from your business data and manually annotate standard answers. Compare them with the model’s outputs under the same prompt template using Rouge and BERTScore automatic evaluation, supplemented by manual blind testing. The purpose of this step is to eliminate data contamination effects and ensure the model’s true capabilities meet expectations.

Step 2: Design the serving architecture
A “gateway + inference instance group + queue” architecture is recommended. The gateway handles authentication and routing, the inference instance group auto-scales based on traffic, and the queue is used to smooth peaks and fill valleys. For high-concurrency scenarios, be sure to enable continuous batching, which concatenates tokens from different requests into the same batch and can increase throughput by 2-4x.

Step 3: Implement performance stress testing and tuning
Use stress-testing tools (such as wrk, Locust) to generate 100, 500, and 1,000 concurrent requests per second, monitoring GPU utilization, memory footprint, and latency distribution. Common tuning methods include: KV Cache quantization, prefix caching to hit similar requests, and adjusting generation parameters (temperature, top_p) to shorten invalid output.

Step 4: Establish monitoring and alerting systems
Focus on monitoring three metrics: “tail latency” (P99) in service level agreements (SLAs), because some users will experience larger delays; GPU utilization, because if it remains below 50% for a long time, it indicates underutilized resources and should trigger scaling down or lowering concurrency limits; and error rate, especially “context length exceeded” errors.

6. Future Trends and Risk Warnings

Domestic models are currently evolving toward “larger context, lower inference cost, and stronger agent capabilities.” For those making selections, here are two reminders:

1. Beware of the “benchmark trap”: Some models achieve high scores on public leaderboards through targeted training but perform mediocrely on real business tasks. Always re-test with private data.
2. Forward-looking compute planning: Large models iterate extremely quickly, and today’s optimal solution may become outdated in six months. Architecture design should maintain decoupling from the model serving layer, so that the adaptation cost when switching base models can be controlled within two weeks.

In summary, there is no absolute best choice among domestic large models—only the one with the highest degree of fit. Returning to the essence of engineering, speaking with data, validating with stress tests, and evaluating with cost—this is the correct approach to dealing with the model explosion. The future core competitiveness lies not in which model you use, but in how quickly and economically you can transform model capabilities into business value.