Evaluating and Optimizing Inference for Chinese Large Language Models

    In recent years, Chinese large language models (LLMs) have made significant progress in general understanding, code generation, and inference services, with several entering the top tier in public benchmarks and attracting attention from the global tech community. However, for developers, the true focus should not be on benchmark rankings, but on whether models can maintain stable, explainable, and operable performance in real-world business scenarios. This article explores how to translate impressive benchmark scores into reliable production capabilities across three dimensions: evaluation methodologies, capability breakdown, and inference engineering.

    In recent years, Chinese large language models (LLMs) have made continuous progress in general understanding, code generation, and inference service capabilities. Some of these models have entered the top tier in public comparisons, attracting attention from overseas tech communities. For developers, what is truly worth focusing on is not a single benchmark ranking, but whether the model can maintain stable, explainable, and operable performance in real business operations. This article discusses how to transform “looks powerful” into “reliable to use” from three levels: evaluation methods, capability breakdown, and inference engineering.

    1. Shifting Focus from Benchmarks to Engineering Usability

    Early LLM selection often relied on parameter scale, benchmark scores, or single-turn Q&A performance. However, production systems face more complex issues: whether multi-turn instructions are stable, whether long context is effective, whether coding tasks pass hidden tests, whether peak throughput and median response meet user experience expectations, and whether anomalous inputs are controllable. The attention from overseas tech communities to Chinese models essentially reflects a shift in evaluation perspective: from “can the model answer” to “can the model complete tasks stably.”

    Therefore, enterprises should establish a layered evaluation system rather than covering all scenarios with a single total score. It is recommended to divide it into at least four layers: basic understanding and multi-turn dialogue, programming and tool usage, complex generation tasks, and inference service quality. Each layer must be bound to business samples, acceptance criteria, and regression mechanisms. A candidate model entering the top tier in comprehensive capabilities only indicates that its basic capabilities are relatively balanced; a leading performance in coding tasks suggests that specialized optimization may be effective; and stable performance in complex tasks like 3D scene generation implies potential in structured generation and long-chain execution. However, these conclusions must be cross-validated with your own business samples.

    2. Comprehensive Capability Evaluation: Avoiding the “Total Score Illusion”

    Comprehensive capabilities typically include knowledge understanding, instruction following, multi-turn context, information extraction, summarization, and tool calling. Evaluation design should follow three principles:

    1. Scenario Slicing: Break down business problems into subsets such as Q&A, summarization, classification, extraction, and planning. Track accuracy, human preference, and failure types separately.
    2. Difficulty Layering: Use easy questions for regression, medium and hard questions to differentiate capabilities, and boundary cases to detect hallucinations and refusals.
    3. Process Traceability: Retain inputs, outputs, prompt versions, model versions, and scoring rationales for easy regression diagnosis.

    Reporting only a single comprehensive score can easily mask unbalanced capabilities. For example, a model might excel in knowledge Q&A but perform unstably in multi-turn constraint following. A more reasonable engineering approach is to build a “capability matrix”: the horizontal axis represents task types, and the vertical axis represents accuracy, stability, latency, and cost. A candidate model being in the top tier for comprehensive capabilities means its capability matrix is relatively balanced; however, enterprises should still prioritize the dimensions that best match their core business, rather than blindly pursuing all-around perfection.

    3. Coding Capability Evaluation: Moving from Completion to Task Closed-Loop

    Coding capability is one of the dimensions developers care about most. Modern coding evaluation should not stop at function completion but should cover requirement understanding, repository context retrieval, code modification, unit test generation, bug fixing, and security constraints. A relatively complete evaluation pipeline can be abstracted as follows:

    1. for case in coding_evaluation_set:
    2. construct requirements and context
    3. call candidate model to generate patch or code
    4. compile and run hidden tests in an isolated environment
    5. execute static checks and security rules
    6. record pass rate, regression rate, time consumed, and failure reasons
    7. categorize failed samples and feed them back to prompt and retrieval strategies

    Recommended evaluation metrics include compilation pass rate, hidden test pass rate, regression failure rate, average repair rounds, human readability score, and dangerous code interception rate. It is not uncommon for a model to “overtake” others in a specific coding evaluation, as it often relies on specialized data, constrained decoding, test feedback, or retrieval augmentation. During enterprise implementation, you can combine retrieval augmentation, sandbox execution, static analysis, and test feedback loops, ensuring that the model’s generated results go through a verifiable process before entering the main branch.

    4. Inference Performance: Focusing on Median and Tail Latency Beyond Peaks

    Inference performance directly determines user experience and cost. Common metrics include Time To First Token (TTFT), time per output token, peak throughput, median response time, P95/P99 latency, and cost per request. Achieving a high Token/s peak in extreme speed mode indicates that the system has throughput capacity under ideal conditions; however, in production environments, more attention should be paid to the median response and tail latency as concurrency increases.

    Stage separation is also important: the throughput characteristics of the thinking phase differ from those of the answering phase, and the experience of long-output tasks is determined by the overall response time. Optimization methods typically include batching, KV cache reuse, continuous batching, quantization, model routing, result caching, and elastic scaling. It is recommended to record both throughput and latency during stress testing to avoid sacrificing stability for peak performance. For example, the following pseudocode illustrates the approach for concurrent stress testing and percentile statistics:

    1. async def benchmark(client, prompts, concurrency):
    2. tasks = []
    3. for prompt in prompts:
    4. tasks.append(run_one(client, prompt))
    5. results = await run_with_concurrency(tasks, concurrency)
    6. return {
    7. "qps": len(results) / elapsed,
    8. "p50_latency": percentile(results.latency, 50),
    9. "p95_latency": percentile(results.latency, 95),
    10. "tokens_per_second": sum(results.tokens) / elapsed
    11. }

    Stress testing conclusions should serve capacity planning: estimate the number of instances based on peak business concurrency, and reserve space for canary releases and failover. For interactive applications, the median response should prioritize meeting user experience targets; for offline batch processing, the batch size can be increased to trade for throughput.

    5. Engineering Architecture and Best Practices

    In a production architecture, it is recommended to encapsulate model capabilities into a unified inference gateway and route them based on task complexity: route simple tasks to a low-latency path, and complex tasks to a high-quality path; add caching for repetitive questions; add sandbox execution and test feedback for coding tasks; and add logging, monitoring alerts, and manual review for critical business operations. Container platforms, logging services, and monitoring alerts in general cloud environments can be used for elastic deployment and observability, but specific selections should align with the team’s existing infrastructure.

    Best practices can be summarized into five points: first, establish scenario-based benchmark sets and perform regular regressions; second, evaluate quality, latency, cost, and stability simultaneously; third, integrate the evaluation pipeline into the release process, where model or prompt changes must pass regression; fourth, retain failed samples and scoring rationales to form iterable assets; fifth, adopt canary releases and rollback mechanisms to avoid an all-at-once full switch.

    Overall, Chinese LLMs entering the mainstream top tier is just the starting point. For developers, what is more important is using rigorous evaluation methods to identify capability boundaries, using engineering practices to bridge reliability gaps, and using observable metrics to continuously optimize inference services. Only by truly translating comprehensive capabilities, coding task closed-loops, and high-throughput low-latency into business metrics will model competitiveness be transformed into sustainable productivity.