Selecting Domestic LLM Inference Services: Key Evaluation Criteria and Technical Validation

1. Why Model Selection Is Moving from Benchmarks to Engineering Fit

For a long time, large language model (LLM) selection has been heavily influenced by leaderboard scores. Leaderboards do reflect a model’s relative performance on certain academic tasks, general Q&A, or code generation. However, engineering teams integrating a model into real systems often encounter more specific challenges:

• Is the inference service stable, and is time-to-first-token predictable?
• Does it support streaming output, tool calling, and structured output?
• As context windows expand, can long-text scenarios maintain acceptable latency and cost?
• Is the model’s understanding of private data, domain terminology, and business rules consistent enough?
• Are there clear security and compliance risks, such as sensitive information leakage or uncontrolled content generation?

A more practical approach is to divide model selection into two stages. The first is capability screening, using public benchmarks, technical reports, and small-scale internal tests to identify candidate models. The second is engineering validation, where candidates are integrated into a unified inference platform or evaluation framework and compared against real business data. The latter is often more decisive than leaderboard rankings alone.

2. Understanding the Role of Inference Platforms in Model Deployment

Many teams new to LLMs focus first on model weights, parameter counts, and training data, while overlooking the importance of the inference platform. In practice, the same model can behave differently depending on the inference service hosting it.

Inference platforms typically handle several key responsibilities:

1. Model hosting and version management
   The platform maintains model versions, dependency environments, and inference engine updates. Developers do not need to manually handle GPU drivers, operator optimization, or memory allocation.

2. API standardization
   Most platforms provide unified interfaces for text generation, chat completion, embeddings, and related capabilities. Standardized APIs reduce model-switching costs and decouple business code from specific models.

3. Performance optimization
   Platforms apply inference optimizations for different hardware, including quantization, batching, KV cache management, and dynamic batch sizing. These optimizations directly affect throughput and latency.

4. Availability and elasticity
   Production environments often require multiple replicas, auto-scaling, rate limiting, and retry mechanisms. If the inference platform is not mature enough, business teams must build substantial infrastructure themselves.

5. Security and compliance capabilities
   Mature platforms typically provide access control, audit logging, content filtering, and sensitive information detection. For enterprise applications, this layer is critical.

When a domestic LLM is hosted on a particular inference platform, developers should focus not only on whether the model is available, but also on how it can be integrated, whether the platform fits their tech stack, and whether it meets production-grade requirements. If a model is released as weights only and the team lacks sufficient operational capacity, self-hosting may not be cost-effective.

3. First-Round Candidate Screening: Starting from the Task

Before discussing specific models, it is important to clarify the actual task types required by the business. Different tasks place different demands on a model.

3.1 Text Generation and Content Creation

For use cases such as marketing copy, summarization, email drafting, and report polishing, focus on:

• Fluency and style control in the target language;
• Support for longer outputs;
• Adherence to instructions;
• Consistency of facts and avoidance of fabricating information.

These scenarios demand strong control over content style. Teams should prepare a set of real business samples and compare output quality across multiple models using identical prompts.

3.2 Code Generation and Development Assistance

For tasks such as code completion, code explanation, unit test generation, and code review, pay attention to:

• Proficiency in mainstream programming languages and frameworks;
• Understanding of project context;
• Compilability and runnability of generated code;
• Ability to explain technical approaches rather than only providing snippets.

Developers can build a small internal benchmark covering common business code, SQL, scripts, and configuration files to evaluate pass rates and readability across models.

3.3 Knowledge Q&A and Retrieval-Augmented Generation

In scenarios such as knowledge base Q&A, intelligent customer service, and document assistants, models typically need to work with external knowledge sources. Key considerations include:

• Whether the model correctly cites provided context;
• Whether it hallucinates when context is insufficient;
• Whether it supports long contexts for passing more material at once;
• Whether it can express uncertainty, such as stating “based on the provided materials, this cannot be confirmed.”

For these use cases, test through a retrieval-augmented generation (RAG) framework where models answer based on the same retrieval results. Compare answer accuracy and traceability.

3.4 Structured Information Extraction

When extracting fields such as dates, amounts, entities, and clauses from contracts, financial reports, meeting notes, or emails, structured output capability is essential.

Testing should focus on:

• Whether the model strictly outputs JSON or the specified format;
• Field completeness and accuracy of extraction results;
• Consistency in handling missing information;
• Understanding of complex layouts such as tables, lists, and multi-level headings.

3.5 Tool Calling and Agent-Based Applications

For scenarios where the model needs to call external tools, query APIs, or execute workflows, also evaluate:

• Whether it selects the right tool based on user intent;
• Whether it correctly passes parameters across multi-turn interactions;
• Whether it has some self-correction capability;
• Whether it tends to produce repeated or incorrect calls.

These capabilities are difficult to assess through static leaderboards. A minimal agent environment is usually required for validation.

4. Building a Unified Evaluation Set to Avoid Intuition-Driven Selection

A common mistake is drawing conclusions from only a few conversations. Since LLMs are highly sensitive to prompts, different phrasings can produce significantly different results. Therefore, establish a unified evaluation set early in the selection process.

4.1 Evaluation Set Design Principles

• Close to real business: Samples should come from real user questions, historical tickets, and internal documents rather than being entirely fabricated.
• Cover core tasks: At minimum, cover the key task types mentioned above.
• Include difficult samples: Do not limit the set to simple questions. Include long texts, ambiguous expressions, multi-turn conversations, and edge cases.
• Quantifiable criteria: Each sample should have a clearly defined expected result or scoring standard, such as accurate, partially accurate, incorrect, or refused.

4.2 Combining Automated and Human Evaluation

For tasks like code generation and information extraction, automated validation rules can be designed:

def evaluate_extraction(pred, gold):
    """
    Simplified example: check whether key fields match
    """
    score = 0
    for field in gold:
        if field in pred and pred[field] == gold[field]:
            score += 1
    return score / len(gold)

For subjective tasks such as copywriting and summarization quality, human scoring or multi-model cross-evaluation is still needed. Human evaluation should be conducted blind where possible to prevent model names from influencing judgment.

4.3 Recording Failure Cases

Do not focus only on average scores during evaluation. Failure cases are often more valuable than high-scoring samples. Teams can categorize errors by type, such as:

• Factual errors;
• Format errors;
• Logical breaks;
• Over-generation;
• Information omission;
• Refusing to answer or being overly conservative.

Analyzing failure cases helps teams understand the boundaries of each model’s capabilities and provides direction for prompt optimization and workflow design.

5. Performance, Cost, and Stability: Hard Metrics for Production

Even if a model delivers strong output quality, it may be difficult to deploy if inference latency is too high, costs are uncontrollable, or the service is unstable.

5.1 Latency and Throughput

Two core metrics require attention:

• Time-to-first-token: The time from sending a request to receiving the first output fragment. For interactive applications, this should stay within an acceptable range.
• Generation throughput: The number of tokens generated per second. For batch tasks, throughput directly determines processing time and resource cost.

Test under conditions close to real concurrency rather than with single requests. Latency may look good in isolation but degrade significantly once concurrency increases and queueing time grows.

5.2 Cost Structure

Model service costs typically consist of input tokens, output tokens, and fixed resource fees. Teams should estimate based on their specific workload:

• Average input and output lengths;
• Daily or monthly request volume;
• Peak concurrency;
• Whether long contexts or frequent tool calls are required.

Some models are inexpensive for short-text Q&A but become costly when processing long documents. Do not judge by unit price alone; calculate total cost based on actual business workloads.

5.3 Stability and Rate Limiting

Stability metrics include:

• API error rate;
• Timeout ratio;
• Rate-limiting frequency;
• Scaling response speed.

Teams should conduct stress testing over a period of time during the evaluation phase and record behavior under different concurrency levels. The business side should also implement retry, fallback, and caching mechanisms to prevent model service fluctuations from directly affecting user experience.

6. Security, Compliance, and Content Governance

In enterprise applications, security and compliance are often mandatory requirements. A model with outstanding capabilities may still be rejected by compliance teams if its safety features are insufficient.

6.1 Data Flow and Privacy Protection

Before integrating a model service, clarify the following:

• Whether data will be used for model training;
• Whether data is encrypted in transit and at rest;
• Whether private deployment or dedicated instances are supported;
• Whether data deletion and audit capabilities are provided.

For scenarios involving personal information, financial data, medical data, or other sensitive information, data security should be the top priority.

6.2 Content Safety and Output Control

Evaluate whether the model may produce inappropriate content, including:

• Illegal or non-compliant information;
• Violent, explicit, or hateful content;
• Leakage of sensitive information beyond the prompt;
• Incorrect medical, legal, or financial advice in business contexts.

Add safety test samples to the evaluation set and observe whether model outputs meet expectations. At the application layer, implement content safety filtering, sensitive word detection, and human review as additional safeguards.

6.3 Explainability and Traceability

For critical business decisions, models should provide supporting evidence or cite sources where possible. If a model only provides conclusions without traceability, troubleshooting and accountability become difficult. When designing prompts and workflows, require the model to explain its reasoning or reference source materials.

7. From Selection to Production: A Suggested Deployment Path

Based on the analysis above, here is an actionable path for model selection and deployment.

Step 1: Clarify Scenarios and Constraints

List business objectives, performance requirements, cost budgets, compliance requirements, and tech stack limitations. Avoid vague comparisons without defined boundary conditions.

Step 2: Define the Candidate Pool

Select several candidate models based on scenario characteristics. Keep the pool manageable—typically three to five models—to avoid excessive testing costs.

Step 3: Build a Unified Evaluation Environment

Integrate all candidates into the same evaluation framework. Test with identical prompts, parameters, and contexts. Record quality, latency, throughput, and failure rate metrics.

Step 4: Conduct Small-Scale Business Validation

Choose one or two real business modules for a pilot, such as internal knowledge base Q&A or a back-office tool. Focus on real user feedback and anomalies.

Step 5: Monitor and Iterate Continuously

Continue monitoring metrics after deployment. Establish log analysis and feedback mechanisms, and periodically re-evaluate model performance. When model versions update or business requirements change, return to the evaluation set and retest.

It is worth noting that model selection is not a one-time task. As new models are released, inference platforms evolve, and business scale changes, teams need a long-term evaluation mechanism rather than treating any single result as permanent.

8. Future Considerations

As LLM technology continues to evolve, keep an eye on the following trends during selection:

• Long-context capability: More businesses need to process entire books, complete codebases, or long meeting transcripts. Long context will gradually become a baseline capability.
• Multimodal input: Mixed text, image, audio, and video inputs will expand the range of business use cases.
• Agent capabilities: Moving from single-turn Q&A to multi-step reasoning, tool calling, and workflow orchestration requires stronger planning and execution abilities.
• Declining inference costs: Advances in quantization, distillation, and sparsification are expected to continue reducing inference costs at equivalent capability levels.
• Improved observability: Model services will provide richer logging, monitoring, and tracing capabilities to help teams diagnose complex issues.

For developers, the goal should not be finding the “best” model at any given moment, but building a sustainable evaluation and integration system. That way, regardless of how the model market changes, teams can validate quickly, switch smoothly, and focus their energy on business value rather than repeated low-efficiency manual comparisons.

In summary, there is no absolute answer when selecting a domestic LLM. The right choice depends on a combination of business scenarios, tech stack, performance requirements, cost structure, and compliance constraints. Through unified evaluation, engineering validation, and phased rollout, teams can more reliably identify the solution that truly fits their needs.