Domestic LLM Selection and Inference Service Integration: A Practical Guide
This article provides a practical framework for selecting domestic LLMs and integrating their inference services. It covers constraint definition, business-sample evaluation, service forms, key management, client integration, launch validation, common pitfalls, and cost and compliance considerations.
As the number of domestic LLMs continues to grow, developers often fall into two traps: looking only at public leaderboard rankings or comparing only unit prices. In practice, model capability, service form, engineering constraints, and compliance requirements often jointly determine the final result. A more reasonable approach is to turn “which model to choose” into a verifiable engineering problem: first define business boundaries, then build an evaluation set, and finally validate stability and cost through small-traffic canary releases.
1. Start with Boundaries: Selection Begins with Constraints
LLM selection is not about finding an all-around champion. It is about finding the option with the highest overall score under current constraints. It is recommended to answer the following questions first:
- What is the task type? General conversation, knowledge QA, code generation, text summarization, structured extraction, multimodal understanding, or complex reasoning? Different tasks place very different demands on model capability.
- How large are the inputs and outputs? Context length, single-output length, concurrent request volume, and peak QPS all affect service selection and cost estimation.
- What are the data compliance requirements? Is data allowed to leave the domain? Is private deployment or a dedicated instance required? Can logs be disabled? Is the data retention period controllable?
- What are the budget and SLA requirements? Is billing based on usage or on an annual/monthly instance subscription? Are rate limiting and queuing acceptable? What is the degradation strategy during failures?
- What is the team’s engineering capability? Does the team have key management, monitoring and alerting, retry and circuit breaking, an evaluation platform, and version management?
These constraints should be documented as a written checklist before selection. Without constraints, selection often ends up becoming “whoever has the loudest voice wins.”
2. Capability Evaluation: Use Business Samples Instead of a Single Leaderboard
Public leaderboards can serve as an initial screening reference, but they cannot replace business evaluation. It is recommended to build a small evaluation set that covers real scenarios and includes at least the following dimensions:
- Basic understanding and Chinese expression: instruction following, terminology accuracy, and whether long-document summarization loses information.
- Structured output: whether the model can stably output JSON, tables, and field extraction results, and whether exceptions can be repaired.
- Long-context capability: locating key information in long documents, cross-paragraph reasoning, and avoiding loss of middle information.
- Tool calling and function calling: whether parameters are complete, types are correct, and retries are possible after failures.
- Code and logic capability: code completion, error localization, unit test generation, and complex conditional reasoning.
- Stability and consistency: whether repeated calls with the same input drift, and whether changes in temperature parameters remain controllable.
- Safety and compliance: sensitive content blocking, prompt injection protection, and data masking capability.
During evaluation, do not only look at whether the answer “sounds right”; look at whether it is acceptable. For example, structured extraction tasks should use field accuracy, missing rate, and format validity rate as metrics. Customer service QA should use hit rate, hallucination rate, and human handoff rate as metrics. It is recommended to keep the same evaluation samples and scoring sheet for each candidate model, recording version, parameters, latency, and failure reasons to ensure reproducible conclusions.
A simplified scoring matrix can be designed as follows:
| Dimension | Weight | Evaluation Method | Recorded Items |
|---|---|---|---|
| Task accuracy | 30% | Manual or automated scoring on business samples | Correctness rate, hallucination rate |
| Structured output stability | 15% | Format validity rate across multiple calls | Validity rate, repair cost |
| Long context | 15% | Long-document QA and summarization | Recall rate, omission rate |
| Latency and throughput | 15% | Stress and concurrency testing | P95 latency, rate-limit rate |
| Cost | 10% | Per-request cost and peak cost | Unit price, cache benefit |
| Compliance and operations | 15% | Data retention, logs, permissions | Risk items, remediation cost |
Weights do not need to be fixed, but they must align with business goals. Internal knowledge base scenarios may place more emphasis on accuracy and compliance; high-concurrency online service scenarios may place more emphasis on latency, rate limiting, and cost.
3. Service Forms: Distinguish Hosted APIs, Dedicated Instances, and Private Deployment
Beyond model capability, service form directly affects integration cost and long-term operations. Common forms include:
- Hosted inference API: fast integration and usage-based billing, suitable for prototyping and small-to-medium traffic. However, pay attention to quotas, rate limits, peak availability, and data retention policies.
- Dedicated instance: dedicated resources and better stability, suitable for businesses with clear SLA requirements. Evaluate scaling, idle cost, and the boundary of operational responsibilities.
- Private deployment: data does not leave the domain and controllability is strongest, but it places higher demands on compute, operations, model upgrades, monitoring, and alerting.
Regardless of the form chosen, confirm access key management, permission isolation, call logs, monitoring metrics, and version change mechanisms. Keys in particular should not be written into frontend code or committed to code repositories. They should be injected through environment variables or a secrets management service, with least privilege and regular rotation.
4. General Integration Flow: Account Setup, Keys, Clients, and Model Identifiers
Console details vary by platform, but the general integration flow usually includes four steps: account and permission preparation, obtaining an access key, configuring a client or SDK, and setting the model identifier. The following pseudocode uses placeholder values; refer to the actual service documentation for endpoints and model identifiers:
# Pseudocode: manage configuration via environment variables to avoid hardcodingexport MODEL_ENDPOINT="<inference service endpoint provided by the vendor>"export MODEL_ACCESS_KEY="<your access key>"export MODEL_NAME="<selected model identifier>"
When configuring a client, three types of information are usually required: service endpoint, access key, and model identifier. It is recommended to manage configuration centrally rather than repeatedly pasting it into each business codebase. The calling layer should uniformly encapsulate timeout, retry, rate limiting, logging, and error code handling, so that business code does not directly depend on the details of a specific model vendor.
# Pseudocode: unified call wrapperrequest = {endpoint: MODEL_ENDPOINT,auth: "Bearer " + MODEL_ACCESS_KEY,body: {model: MODEL_NAME,messages: [...],temperature: 0.2},timeout: 30s,retry: { maxAttempts: 2, backoff: "exponential" }}
After integration, run a smoke test first: confirm that authentication succeeds, the model is callable, the response format meets expectations, and error codes are identifiable. Then run small-traffic validation: observe P95 latency, failure rate, rate-limit rate, and per-request cost. Do not assume integration is complete after testing a single “hello” before launch.
5. Five Validations Before Launch
- Connectivity and authentication: whether the key is valid, whether permissions are minimized, and whether the expiration policy is clear.
- Concurrency and rate limiting: stress test at 1.5x to 2x peak traffic, and record rate-limit thresholds and recovery time.
- Long context and structured output: test truncation, format errors, and repair cost using real long documents and complex fields.
- Failure retry and degradation: whether there are retry, circuit breaking, caching, or backup model strategies for timeouts, rate limiting, and service exceptions.
- Cost and observability: calculate per-request cost, cache hit rate, and failed request cost, and establish logs, metrics, and alerts.
6. Common Pitfalls and Implementation Recommendations
Common pitfalls include looking only at public leaderboards and not at business samples; looking only at unit price and not at peak rate limiting and retry costs; ignoring model version drift, which causes performance fluctuations; ignoring data compliance and log retention; and lacking a unified call wrapper, which leads to vendor lock-in. A more reliable implementation approach is to run a small-scope POC first, build a fixed evaluation set, and then scale through canary releases; use dual-model routing or degradation strategies for critical businesses; and bring prompts, evaluation samples, and call configurations into version management.
7. Conclusion
There is no absolute “best” domestic LLM. There is only a choice that better fits the current task, budget, compliance, and engineering constraints. During selection, use a business evaluation set as the core, treat service form, stability, and cost as constraints, and use canary validation and observability as safeguards. Define requirements first, validate with small traffic, and make decisions with data. This is more reliable than simply chasing leaderboards.
