LLM Selection, Evaluation, and Inference Cost Optimization: A Practical Guide for Chinese Models
This guide explains how to select and evaluate Chinese LLMs for production use without relying on leaderboard rankings alone. It covers translating business goals into measurable requirements, building layered offline evaluation sets, modeling tail latency and effective throughput, calculating cost per successful task, designing caches and model routing, handling rate limits and degradation, and operating a continuous observability and cost-governance loop. It also includes scenario-based guidance and a practical decision checklist.
When selecting a Chinese LLM, the most common mistake is to treat leaderboard rankings as the final answer. In real engineering, model selection is never a single-metric contest. It is a multi-constraint optimization problem: whether task quality meets the bar, whether response latency is acceptable, whether throughput is sufficient, whether token cost is controllable, whether rate limits and stability satisfy business peaks, and whether data compliance is satisfied. A very large model that performs well on general benchmarks may be worse in high-frequency, lightweight, latency-sensitive workloads than a smaller model with faster responses and lower cost. Conversely, in high-risk decisions, complex reasoning, or code review scenarios, pursuing only low cost can create higher rework and human-review costs.
A more reasonable approach is therefore not to ask “which model is best,” but to build a reproducible selection method: first define acceptance criteria from business tasks, then validate quality with a small high-quality evaluation set, then model latency, throughput, token cost, and cache benefits, and finally form a production loop through layered routing, rate-limit governance, and continuous monitoring.
1. Correct the Mindset First: Design a Model Portfolio, Not a Single Bet
In enterprise applications, using one model for everything is usually not optimal. A more common practice is “capability layering + scenario routing”:
- L1 small models: handle high-frequency, well-defined-format, short-context lightweight tasks, such as classification, field extraction, simple rewriting, and fixed-template generation.
- L2 medium models: handle tasks that require some reasoning ability, such as multi-turn Q&A, summarization, structured reports, and routine code completion.
- L3 large models: used only for complex reasoning, long context, difficult fallback cases, and offline tasks with extremely high quality requirements.
The core point is not merely to “save some money,” but to calculate quality, latency, and cost on the same ledger. When L1 can pass validation stably, there is no need to call L3. Escalation should happen only when validation fails, confidence is insufficient, task risk is high, or the user explicitly requests a high-quality mode. A model portfolio depends more on engineering governance than a single model does, but it is usually closer to a production-ready state.
2. Requirements Breakdown: Translate Business Goals into Measurable Metrics
Before selection, at least six types of constraints need to be clarified.
2.1 Task Type
Different tasks have very different requirements for model capabilities:
- Information extraction: emphasis on structural output stability, field completeness, and format validity.
- Summarization and rewriting: emphasis on information fidelity, length control, and style consistency.
- Multi-turn dialogue: emphasis on context retention, instruction following, refusal behavior, and safety policy.
- Long-document analysis: emphasis on context window, retrieval-augmented quality, and citation consistency.
- Code assistance: emphasis on syntax correctness, testability, and boundary-condition handling.
- Batch content processing: emphasis on throughput, unit cost, and asynchronous task orchestration.
The evaluation set for selection must be built around your own tasks; public general-purpose questions alone are not enough. Public benchmarks can help with initial screening, but they cannot replace business samples.
2.2 Quality Thresholds
Quality is not a vague “feels better.” It should be broken into checkable items:
- Factual consistency: whether the output is consistent with the given material and whether there is unsupported expansion.
- Structural compliance: whether the output satisfies the specified format, field names, enum values, and length limits.
- Task completion rate: whether the output actually answers the user’s question rather than giving generic statements.
- Stability: whether repeated calls on the same input produce controllable results.
- Safety: whether the output complies with requirements for sensitive information, unauthorized access, and content safety.
For structured output, prioritize “rule validation + sampled human review.” For subjective quality, use blind evaluation and double review. For code tasks, use unit tests and static checks as partial objective signals.
2.3 Latency Targets
Latency should distinguish at least:
- Time to first token or first segment: determines interaction feel.
- Full response time: determines task completion time.
- p50, p95, and p99 latency: averages easily hide tail latency.
- Queue waiting time: during peak periods, queuing may affect experience more than model inference.
For real-time dialogue, p95 latency is often more important than average latency. For offline batch processing, throughput and unit cost may take priority over single-request latency.
2.4 Throughput and Concurrency
Estimate peak request rate, average input length, average output length, number of concurrent users, and whether queuing is allowed. Capacity planning should not look only at “how many requests per second can be processed,” but at how many “successful tasks” can be completed per second within the latency threshold.
2.5 Cost Basis
Do not compare only the “price per million tokens.” The full cost should include:
- Input token cost;
- Output token cost;
- Cache creation cost;
- Cache read cost;
- Retry and failed-request cost;
- Safety review, human review, and rework cost;
- Logging, storage, bandwidth, and operations cost.
A more reliable metric is “cost per successful task”:
Cost per successful task = total cost during the measurement period ÷ number of successfully completed tasks
This metric reflects unit price, failure rate, and retry overhead at the same time, and is closer to a real business perspective than comparing unit prices alone.
2.6 Compliance and Deployment Boundaries
Confirm in advance whether data contains personal information, enterprise secrets, or regulated data; whether cross-border or cross-tenant processing is allowed; and whether private deployment, dedicated instances, audit logs, access isolation, and key management are required. Compliance should not be patched only before launch.
3. Offline Evaluation: Use a Small High-Quality Dataset Instead of Guesswork
It is recommended to build a layered evaluation set: normal samples, boundary samples, long-context samples, adversarial samples, safety samples, and historical production-failure samples. Each sample contains input, optional context, expected result or scoring rubric, task type, and risk level.
A simplified evaluation flow is as follows:
dataset = build_layered_eval_set()for each candidate_model in candidates:for sample in dataset:start = now()result = call_model(sample.input, sample.context, params)latency = now() - startsuccess = validate_structure(result) and validate_content(result, sample.gold)score = score_result(result, sample.gold)tokens = count_tokens(sample.input, result)record(candidate_model, sample.id, latency, tokens, score, success)output:quality_pass_rate = successful_samples / total_samplesp50/p95/p99 latencycost_per_successful_task = total_cost / successful_samplesbreakdown by task type, input length, and risk level
Three points need attention during evaluation:
- The sample size must be sufficient. A small number of samples can easily produce random fluctuations, especially for subjective and long-tail questions. For key metrics, provide confidence intervals or at least retain a re-test mechanism.
- Scoring rules must be fixed. Human scoring should have clear criteria to avoid inconsistent standards among reviewers.
- Evaluation must be consistent with production. Prompt templates, retrieval context, output format, safety policy, and parameter settings should be as close to production as possible; otherwise offline results will be distorted.
Evaluation is not a one-time task. Every prompt change, model version change, context-length adjustment, or cache-strategy change should trigger regression evaluation.
4. Performance Modeling: Focus on Tail Latency and Effective Throughput
LLM inference can usually be divided into two stages: input processing and output generation. Long input raises preprocessing cost, and long output extends full response time. Engineering teams should record separately:
- Time from request entry to first segment return;
- Time from first segment return to completion;
- Input token count and output token count;
- Queue waiting time;
- End-to-end successful duration;
- Number of timeouts, retries, and degradations.
Capacity planning can use an approximate estimate:
Required concurrency ≈ peak request rate × average end-to-end latency × safety margin
For example, if the peak requires a certain number of requests per second and the p95 latency of a single request is high, the required concurrency increases rapidly. The safety margin usually cannot be zero, because traffic is bursty and model services also jitter.
More importantly, define “effective throughput”:
Effective throughput = number of requests successfully completed within the p95 latency threshold / second
Counting only total calls is meaningless. Requests retried after timeouts amplify cost and further consume capacity. For real-time businesses, it is better to set a reasonable timeout and degradation message on the front end than to let requests queue indefinitely.
5. Cost Formula: Put Input, Output, Cache, and Retry on One Table
Pricing on different platforms may include multiple rate items such as input, output, cache write, and cache read. A general cost formula can be written as:
single_request_cost =input_tokens × input_rate+ cache_creation_tokens × cache_creation_rate+ cache_read_tokens × cache_read_rate+ output_tokens × output_rate
If a platform displays “billing units” or “points,” convert them into monetary cost using the same logic before calculating business metrics. Do not compare only the input rate, because the output rate, cache-write cost, and cache-read cost may significantly change the total bill.
A practical cost-accounting pseudocode is as follows:
function calculate_cost(usage, rate):return usage.input_tokens * rate.input+ usage.cache_creation_tokens * rate.cache_creation+ usage.cache_read_tokens * rate.cache_read+ usage.output_tokens * rate.outputfunction calculate_cost_per_successful_task(records):total_cost = 0success_count = 0for r in records:total_cost += calculate_cost(r.usage, r.rate)if r.success:success_count += 1return total_cost / max(success_count, 1)
Cost reduction should start with structure rather than fixating on unit price:
- Compress invalid context: deduplicate, rank, and truncate retrieval results to avoid stuffing irrelevant material into the prompt.
- Control output length: use field constraints and length limits for structured tasks; set target lengths for summarization tasks.
- Improve first-pass success rate: validate formats before generation, keep examples concise, and stabilize parameters to reduce retries.
- Use stable prefix caching: when the platform provides cache billing items, place fixed system prompts, fixed tool instructions, and stable knowledge prefixes in cacheable positions.
- Asynchronous batch processing: non-real-time tasks can use batch processing and off-peak scheduling to reduce peak pressure.
- Layered routing: let lightweight tasks go to small models first and escalate complex tasks later.
It is important to emphasize that caching is not an unconditional benefit. If cache-write cost is high, hit rate is low, or content changes frequently, caching may not be worthwhile. Evaluate it as “hit rate × saved cost − extra write/storage cost.” For content that contains personal information or tenant-isolation requirements, design cache isolation and invalidation policies as well.
6. Cache Design: Hit Rate, Isolation, and Invalidation Are All Required
Caches can be divided into multiple layers:
- Prompt prefix cache: stable system prompts, fixed format instructions, and public knowledge snippets.
- Retrieval result cache: reuse of retrieval results for identical or similar queries.
- Answer cache: cache final answers for questions with strong determinism and infrequent changes.
- Tool result cache: short-term reuse of external query results.
Cache benefit can be roughly evaluated as:
Cache benefit ≈ cache read tokens × input rate − cache creation cost − storage and invalidation cost
When designing a cache, answer four questions:
- How high is the hit rate? If requests are highly dispersed, cache benefit will be limited.
- Is the content allowed to be reused across requests? In multi-tenant scenarios, isolation by tenant is mandatory to avoid data leakage.
- What is the invalidation strategy? Invalidate based on time, version number, content digest, or business events.
- How are cache errors handled? Expired answers, incorrect recall, and consistency conflicts must have detection and rollback mechanisms.
For highly time-sensitive tasks, answer caching should be used cautiously. For stable content such as fixed instructions, template prefixes, and public code snippets, prefix caching is usually safer.
7. Model Routing: Use Validators to Decide Whether to Escalate
The key to layered routing is not “call a small model first and stop there,” but designing clear escalation conditions. Common validators include:
- Format validation: whether the output meets structure, field, enum-value, and length requirements.
- Business-rule validation: value ranges, citation sources, and permission constraints.
- Unit tests: whether code tasks pass the given tests.
- Confidence signals: rule hit rate, candidate-answer consistency, and retrieval support.
- Risk level: high-risk tasks go directly to a stronger model or human review.
Simplified routing pseudocode:
function handle_request(req):ctx = build_context(req)cheap_result = call_model(L1, ctx)if validate_format(cheap_result)and validate_business_rules(cheap_result)and risk_level(req) == lowand confidence_signal(cheap_result) >= threshold:return cheap_resultmedium_result = call_model(L2, ctx)if validate(medium_result):return medium_resultstrong_result = call_model(L3, ctx)if validate(strong_result):return strong_resultreturn safe_fallback(req)
Before routing goes live, run a shadow evaluation: let the escalation path run on offline data, and measure “direct pass rate,” “post-escalation fix rate,” “extra cost caused by escalation,” and “ratio still failing after escalation.” If the L1 pass rate is low, or L1 errors cannot be detected by validators, do not blindly pursue low cost.
8. Rate Limiting, Retries, and Degradation: Stability Is Part of Selection
Real systems must face rate limits. Platforms may limit requests per minute, tokens per minute, concurrency, or account quota. Engineering teams need to:
- Token bucket or leaky bucket rate limiting: control both request count and token count.
- Exponential backoff with random jitter: avoid a retry storm hitting the service at the same time.
- Idempotency keys: ensure timeout retries do not create duplicate side effects.
- Circuit breaker: stop calling temporarily after consecutive failures, fail fast, and degrade.
- Priority queue: ensure critical business gets capacity first.
- Asynchronization: put non-real-time tasks into queues to smooth peaks and fill valleys.
- Observable error classification: distinguish rate limiting, timeout, content refusal, format error, and internal error.
A simplified rate-limit and retry logic is as follows:
function call_with_retry(req):attempt = 0while attempt < max_retries:if token_bucket.allow(req.estimated_tokens):try:return call_model(req)catch retryable_error:passwait_time = backoff(attempt) + random_jitter()wait(wait_time)attempt += 1return degraded_result(req)
A degraded result is not necessarily a “failure message.” It can return a cached answer, use a simplified template, hand off to a human, ask the user to retry later, or switch to a cheaper but slightly lower-quality processing path. The key is to define in advance the impact of each degradation on user experience and business risk.
9. Observability and Cost Governance: Turn Selection into Continuous Operations
Selection does not end at launch; it becomes continuous operations. At minimum, monitor the following metrics:
- Success rate and quality sampling score for each task type;
- p50, p95, and p99 end-to-end latency and queue waiting time;
- Input, output, cache-creation, and cache-read token counts;
- Cost per successful task and budget consumption rate;
- Number of rate limits, retries, timeout rate, and degradation rate;
- Small-model direct pass rate, escalation rate, and post-escalation fix rate;
- Cache hit rate, cache error rate, and invalidation count;
- Safety blocks, content refusals, and human-review ratio.
Logs should include request ID, task type, prompt version, model version, token statistics, latency, validation result, and cost fields, while redacting sensitive information. Without prompt-version and model-version records, later quality regressions are difficult to locate.
Cost governance can use team quotas, project tags, budget alerts, and anomaly spike detection. Rather than looking at the total bill at the end of the month, it is more effective to break down cost by business line, task type, and call source, so that “who is consuming, why, and whether it can be optimized” becomes clear.
10. Scenario-Based Selection Preferences
Real-Time Dialogue and Customer Service
Prioritize first-segment response, p95 latency, rate-limit stability, and safety policy. A medium model plus caching and retrieval augmentation is suitable, with escalation for complex questions. Do not sacrifice obvious experience for extremely low cost, and do not call the largest model for every question.
Batch Extraction and Data Processing
Prioritize structural output stability, unit successful-task cost, and throughput. Small models can be used heavily with strict format validation and failure retries. Samples that fail validation can then go to stronger models; overall cost is usually more controllable than using large models for everything.
Long-Document Analysis and Knowledge Q&A
First decide whether to use “direct long-context input” or “retrieval-augmented input.” Long context increases input cost and may dilute key information; retrieval augmentation can reduce cost but requires handling recall quality, citation consistency, and context assembly. Both approaches should be evaluated with business samples rather than judged only by window size.
Intelligent Coding Assistance
Code tasks require attention not only to generation speed but also to testability, boundary conditions, and security risk. A combination of “small-model draft + test validation + strong-model review” can be designed. Code generation without tests and static validation is difficult to measure with cost metrics alone.
Strong Compliance Scenarios
Prioritize data isolation, audit capability, permission control, deployment boundaries, and traceability. Such scenarios should not choose a solution that cannot meet compliance requirements merely because the unit price is low.
11. Practical Decision Checklist
A weighted scorecard can be used for initial screening, with weights determined by business goals. For example:
| Dimension | Description | Example Weight |
|---|---|---|
| Task quality | Business-sample pass rate, human score | 35% |
| Latency experience | First-segment latency, p95 full latency | 20% |
| Cost | Cost per successful task | 20% |
| Stability | Rate limits, error rate, retry controllability | 15% |
| Compliance and governance | Data isolation, audit, permissions | 10% |
Note that the weights are only examples and cannot replace business judgment. For real-time products, the p95 latency weight may be higher; for offline processing, the cost weight may be higher; for finance, healthcare, and enterprise-confidential scenarios, compliance weight must take priority.
During selection review, answer each of the following:
- Does the quality pass rate on business samples meet the bar?
- Is p95 latency within an acceptable range for users?
- Is the cost per successful task within budget?
- Will peak traffic be rate-limited, and will retries amplify cost?
- Does caching bring net benefit, and are there data-isolation risks?
- What is the small-model direct pass rate, and is the escalation strategy effective?
- Are complete logs, monitoring, budget alerts, and regression evaluation in place?
- When model versions change, can quality regressions be detected quickly?
- Are multiple backup options retained to avoid being locked into a single capability?
- Have degradation, timeout, and human fallback been rehearsed?
Conclusion
Chinese LLM selection should not remain a static comparison of “who is stronger.” It should return to the business itself: evaluate quality with business samples, measure experience with tail latency, measure economics with cost per successful task, improve system resilience with rate limiting, caching, and routing, and ensure long-term controllability with observability and regression evaluation.
A robust conclusion is usually: use small and medium models to cover high-frequency tasks, use strong models for long-tail and fallback cases, use validators to control escalation, use caching and context governance to compress invalid tokens, and use rate limiting and degradation to protect stability. Models will continue to evolve, but the method of “task definition—offline evaluation—cost modeling—layered routing—online monitoring—continuous regression” will not become obsolete.
