Evaluating and Deploying Domestic LLMs: From Capability Assessment to Production

1. What Problem Does an LLM Inference Platform Solve?

Before discussing model selection, it is essential to clarify the core function of an LLM inference platform. Its primary value can be summarized in one sentence: encapsulating trained large models into standardized services that business systems can call directly.

Without an inference platform, developers typically have to handle the following tasks manually:

• Obtaining model weight files, which can range from tens to hundreds of gigabytes.
• Preparing GPU or heterogeneous computing resources and completing the adaptation of drivers and runtime environments.
• Writing inference service code to handle request queuing, batching, and VRAM management.
• Implementing engineering capabilities such as authentication, rate limiting, monitoring, and logging.
• Continuously tracking model updates for version management and canary releases.

These tasks are complex and resource-intensive. An inference platform abstracts these capabilities into unified API interfaces. Developers only need to focus on inputs and outputs, while the platform manages the inference process. While this lowers the barrier to entry, it also introduces a new selection challenge: when multiple models are available on a platform, which one should you choose?

2. From Leaderboards to Business Scenarios: First Principles of Selection

Many teams instinctively look at public leaderboards when selecting a model. Leaderboards are indeed a useful reference, providing a horizontal comparison of model capabilities on relatively standardized test sets. However, leaderboard scores do not directly translate to business value for three main reasons.

First, evaluation tasks and business tasks are not always isomorphic. Leaderboards typically evaluate models on mathematical reasoning, code generation, general QA, and multilingual understanding. However, actual business use cases might focus on long-document extraction, structured output, domain-specific terminology comprehension, or safety and compliance checks. A model with a lower overall score might outperform a higher-scoring model on specific tasks.

Second, evaluation datasets may suffer from data contamination. Public benchmarks have been used for a long time, and some questions may have leaked into the training data, inflating the scores. Real business data consists of inputs the model has never seen, which can result in varied performance.

Third, deployment variations impact the final result. The same model weights can perform differently when subjected to various quantization strategies, inference framework optimizations, and hardware adaptations. Leaderboard scores are usually based on standard configurations and may not accurately represent the serviced version available on a specific platform.

Therefore, a more reliable approach is to use leaderboards as an initial filter to shortlist first-tier candidate models, and then conduct targeted validation on your own business evaluation datasets.

3. Building a Business-Specific Evaluation Dataset

Designing a business evaluation dataset is the most critical step in the selection process. It should adhere to the following principles:

Cover typical scenarios. If the business involves intelligent QA, the dataset should include high-frequency question types, edge cases, and multi-turn dialogue contexts. For code generation, cover different languages and difficulty levels. For document processing, include long documents, mixed formats, and cross-page tables.

Prioritize annotation quality over quantity. A carefully human-annotated dataset of 200 samples is often more discriminative than 2,000 unreviewed, auto-generated samples. Each sample must have a clear input, expected output, and quantifiable judging criteria.

Tiered difficulty. Simple tasks verify baseline capabilities, medium tasks differentiate model performance, and difficult tasks test the model’s upper limits. If all models achieve a 100% pass rate on simple tasks, those samples offer no discriminative value.

Reserve an update mechanism. Business requirements evolve, so the dataset needs regular updates—adding new samples and retiring outdated ones to maintain alignment with live business scenarios.

Once the dataset is ready, test candidate models one by one via a unified API, recording metrics like response time, output quality, and format compliance. It is recommended to conduct multiple rounds of testing and average the results to avoid single-sample bias.

4. Comprehensive Evaluation Checklist

Beyond output quality, selection should comprehensively consider the following dimensions:

4.1 Inference Performance

Response time directly impacts user experience. You must distinguish between first-token latency and total generation time. For interactive scenarios, first-token latency should be controlled within a few hundred milliseconds; for batch processing, total throughput is more critical. Stress testing under real business load levels is recommended rather than testing single requests only.

4.2 Context Window and Long-Text Capabilities

While LLM context windows have generally expanded, information extraction accuracy and the rate of intermediate information loss in long-context scenarios still require empirical testing. If your business involves long documents or extensive chat histories, this must be included in the evaluation.

4.3 Structured Output and Tool Calling

Many business scenarios require models to output structured formats like JSON or XML, or to call external tools. You need to test the model’s format compliance rate under complex schemas and parameter accuracy during multi-step tool calls.

4.4 Security and Compliance

Output content must meet enterprise security standards. Test the model’s ability to refuse and avoid risks related to sensitive topics, harmful instructions, and privacy leaks. Additionally, ensure the data flow chain complies with data privacy and regulatory requirements.

4.5 Cost Structure

In a token-based billing model, the unit price of input and output tokens, discount strategies at scale, and free quotas all affect the total cost. Calculate costs based on estimated monthly request volumes rather than simply comparing unit prices.

4.6 Stability and Technical Support

Engineering factors like service availability, rate limiting policies, incident response mechanisms, and technical documentation completeness are equally important. Run a small percentage of traffic for a trial period to observe stability.

5. Multi-Model Strategy: From Single Selection to Routing

In practice, many teams find that a single model rarely achieves optimal performance across all scenarios. In such cases, consider a multi-model routing strategy.

Scenario routing. Distribute traffic to different models based on request type. For instance, route code-related requests to one model, Chinese long-document extraction to another, and general QA to a third. Routing rules can be classifier-based or rule-based.

Quality tiering. Use higher-capability models for high-value requests and lower-cost models for low-priority requests. For example, route requests from key clients to a flagship model, while using a lightweight model for internal testing.

Failover and degradation. Automatically switch to a backup model when the primary model times out or hits rate limits, trading absolute certainty for availability. Degradation strategies must be configured in advance to prevent panic during live incidents.

Shadow testing and canary releases. Before launching a new model, route shadow traffic to it. Run it in parallel with the live model without affecting real users, and compare output differences. Gradually shift traffic once no major issues are confirmed.

These strategies require the inference platform’s unified API to be designed appropriately: model switching should be transparent to business code, achievable via configuration changes without requiring large-scale code modifications.

6. Integration Practice and Engineering Examples

Below is a simplified integration logic example demonstrating how to abstract the model invocation layer in business code:

import json
from typing import Dict, Any
class InferenceClient:
    """Unified model inference client, shielding underlying differences"""
    def __init__(self, model_config: Dict[str, Any]):
        self.endpoint = model_config["endpoint"]
        self.model_name = model_config["model_name"]
        self.timeout = model_config.get("timeout", 30)
        self.max_retries = model_config.get("max_retries", 2)
    def generate(self, prompt: str, **kwargs) -> str:
        """Generate text, returning the raw output"""
        payload = {
            "model": self.model_name,
            "prompt": prompt,
            "max_tokens": kwargs.get("max_tokens", 1024),
            "temperature": kwargs.get("temperature", 0.7),
        }
        # Actual request logic implemented according to the platform API specification
        return self._request(payload)
    def generate_json(self, prompt: str, schema: dict) -> dict:
        """Generate JSON and validate the format"""
        response = self.generate(
            prompt + f"\nPlease strictly output according to the following JSON schema:\n{json.dumps(schema)}",
            temperature=0.1,
        )
        return self._parse_and_validate(response, schema)
    def _request(self, payload: dict) -> str:
        # Handle retries, timeouts, and exceptions
        raise NotImplementedError
    def _parse_and_validate(self, text: str, schema: dict) -> dict:
        # Extract JSON and validate fields
        raise NotImplementedError

The core value of this encapsulation is that business code requires no extensive modifications when switching models or adjusting parameters. Even if different models respond differently to prompts, an adaptation layer can normalize the behavior.

In actual deployment, the following should also be considered:

Concurrency control. Upstream business request concurrency may far exceed the model’s throughput. Queuing, rate limiting, and backpressure mechanisms must be implemented at the client or gateway layer.

Caching strategy. For recurring similar requests, introduce semantic caching to reduce model invocation costs. Cache hit determination needs to balance accuracy and recall.

Logging and tracing. Record the input, output, latency, and token consumption for each request for cost analysis, quality monitoring, and anomaly tracing.

Canary release. When upgrading model versions, shift 5% of traffic first, observe key metrics, and then gradually increase the proportion. If output quality drops significantly, roll back quickly.

7. Decision Process Summary

Synthesizing the discussion above, a complete selection decision process can be formed:

1. Define business requirements: List core use cases, request volumes, response requirements, and data compliance boundaries.
2. Shortlist candidate models: Select 3 to 5 first-tier models based on public information and community feedback.
3. Build a business evaluation dataset: Cover typical scenarios, tiered difficulty, and human annotation.
4. Unified API testing: Conduct multiple rounds of testing on the target platform for candidate models, recording quality and performance data.
5. Cost estimation: Calculate the monthly cost of each candidate solution based on estimated call volumes.
6. Small traffic trial: Select 1 to 2 optimal candidates and integrate them into real business for canary validation.
7. Canary rollout: Gradually scale up based on live performance, establishing monitoring and degradation mechanisms.
8. Continuous evaluation: Regularly review model performance, track version updates, and keep the evaluation dataset evolving.

Selection is not a one-time action but a continuous iterative process. As model capabilities improve and new versions are released, the optimal solution may change. Establishing a reusable evaluation system is far more valuable than memorizing a single “best answer.”