Chinese LLM Performance Evaluation: Balancing Accuracy, Latency, and Cost

    A practical analysis of a new Chinese large language model version, covering its gains in accuracy, reasoning, education, healthcare, finance, language understanding, and instruction following, as well as its higher token usage, latency, and operating cost. The article also examines a decline in agent and tool-calling performance and provides a framework for evaluating models against real enterprise requirements.

    Why Model Rankings Alone Are Not Enough

    Accuracy is usually the most visible metric when organizations select a large language model. However, it does not fully represent a model’s practical value. For production applications, suitability also depends on response speed, token consumption, API cost, stability, tool-calling capabilities, and performance in specific industries.

    For example, a model may achieve a high score in an overall benchmark but still deliver a poor production experience if its average response time is significantly longer or if each request consumes substantially more tokens. This can affect both user experience and operating costs in customer service, real-time question answering, and online productivity tools.

    A more useful approach is to build a multidimensional evaluation framework that considers effectiveness, efficiency, and cost together.

    This evaluation focuses primarily on Chinese-language tasks and covers approximately 15,000 test questions. It examines performance in general knowledge, reasoning and mathematical computation, education, healthcare, finance, language understanding, instruction following, and AI agent and tool-calling tasks.

    Overall Performance of the New Version

    The evaluation shows that the new model version achieved an overall accuracy of 71.5%. Average latency was approximately 96 seconds per call, average consumption was 3,922 tokens, and the measured cost was approximately RMB 52.5 per 1,000 calls.

    The most notable change is the improvement in overall capability. Compared with the previous version, accuracy increased from 68.1% to 71.5%, a gain of 3.4 percentage points. Its overall ranking also improved from 16th to 5th. This suggests meaningful progress in knowledge coverage, complex question understanding, and multi-step reasoning.

    However, the accuracy improvement came with additional costs:

    • Average token consumption increased from 2,851 to 3,922 tokens, a rise of approximately 37.6%.
    • Average response time increased from 59 seconds to 96 seconds, an increase of approximately 63%.
    • The cost per 1,000 calls rose from RMB 37.6 to RMB 52.5, an increase of approximately 40%.

    The new version is therefore not simply a universal replacement for the previous one. It exchanges higher accuracy for longer response times and higher operating costs. The right choice depends on the requirements of each application.

    Changes in Specific Capabilities

    1. Significant Improvement in Reasoning and Mathematical Computation

    The score for reasoning and mathematical computation increased from 66.7% to 72.7%, a gain of 6 percentage points. This indicates stronger performance in breaking down multi-step problems, identifying conditional relationships, deriving formulas, and verifying results.

    For applications that require complex logical judgment—such as data analysis, rule auditing, report generation, and technical question answering—better reasoning can directly reduce the workload involved in manual review.

    However, benchmark accuracy should not be interpreted as absolute correctness in a real business environment. For financial decisions, risk assessment, or safety controls, organizations should still use retrieval-augmented generation, rule-based validation, and human review rather than relying solely on model output.

    2. Moderate Improvement in Education Tasks

    The education score increased from 46.9% to 52.0%, an improvement of 5.1 percentage points. This suggests that the model has become more capable of explaining knowledge, analyzing questions, clarifying concepts, and supporting learning activities.

    Educational applications require answers to be complete and appropriately structured. The model must not only provide a conclusion but also explain the reasoning behind it and avoid unsupported answers. The new version’s stronger ability to decompose complex questions can help it generate more organized explanations.

    In production, however, the difficulty level must match the user’s knowledge. Responses for beginners should avoid unnecessary technical terminology, while responses for professionals should provide more complete assumptions, derivation steps, and boundary conditions.

    3. Stronger Performance in Healthcare and Mental Health

    The score for healthcare and mental health increased from 77.6% to 83.6%, a gain of 6 percentage points. This indicates improved performance in understanding medical knowledge, analyzing symptom descriptions, and responding to general health-related questions.

    Healthcare is a high-risk domain. Model output must be clearly positioned as informational guidance rather than a substitute for professional diagnosis. System designs should include risk detection and tiered handling for acute symptoms, medication recommendations, mental health crises, and emergency assistance. When necessary, users should be directed to seek professional help.

    4. The Largest Gain in Financial Tasks

    The financial task score increased from 77.7% to 84.2%, a gain of 6.5 percentage points, making it one of the strongest improvements across all evaluated areas.

    Financial question answering often involves concept interpretation, policy and rule analysis, data explanation, and risk assessment. Progress in these areas can provide a stronger foundation for intelligent customer service, knowledge-base question answering, business training, and document analysis.

    Financial information is also highly time-sensitive. Policies, interest rates, product rules, and market information can change. Financial applications should therefore not rely only on the model’s pretraining knowledge. They should connect to reviewed, up-to-date materials and identify the relevant time period, data source, and applicable conditions in generated answers.

    5. Better Language Understanding and Instruction Following

    The language understanding and instruction-following score increased from 64.6% to 69.3%, a gain of 4.7 percentage points. This means the model has improved at interpreting complex requirements, following output formats, recognizing contextual constraints, and completing tasks with multiple conditions.

    Instruction following is particularly important in enterprise automation. A system may require the model to output only structured fields, stay within a strict word limit, follow a fixed template, or satisfy several business rules at the same time. If the model cannot follow instructions consistently, post-processing and manual correction costs will increase.

    Capability Regression to Monitor

    Although overall performance improved substantially, the score for AI agents and tool calling declined from 69.9% to 68.1%, a decrease of 1.8 percentage points. This was the only evaluated area that showed a decline.

    The result demonstrates that stronger general knowledge and reasoning do not necessarily lead to better tool-use capabilities. Agent tasks typically involve several stages:

    1. Understanding the user’s goal
    2. Selecting the appropriate tool
    3. Constructing valid parameters
    4. Executing the tool call
    5. Reading the returned result
    6. Continuing the decision process based on that result

    An error at any stage can cause the overall task to fail.

    In practical systems, a useful design principle is to let the model handle understanding and planning while letting software enforce constraints and execute actions:

    1. User request
    2. ↓
    3. Intent recognition and task decomposition
    4. ↓
    5. Tool selection and parameter validation
    6. ↓
    7. Programmatic tool execution
    8. ↓
    9. Result validation and exception handling
    10. ↓
    11. Model-generated final response

    Parameter allowlists, call limits, timeout controls, result validation, and retry logic can reduce uncertainty during tool execution. High-risk operations should also require human confirmation to prevent the model from directly performing irreversible actions.

    What Higher Token Consumption Means

    Average token consumption increased from 2,851 to 3,922 tokens, or approximately 37.6%. This may indicate that the model is processing more context when generating answers or using a more complex internal reasoning process.

    Higher token usage can have three major effects.

    First, it increases the cost of each call. Even when input and output prices remain unchanged, processing more tokens per request raises total spending.

    Second, it increases response time. The model may require more time for queuing, reasoning, and output generation.

    Third, it creates greater pressure on context management. In long-document question answering, batch data analysis, and multi-turn conversations, token usage can accumulate quickly. Systems may need to control the length of conversation history and use summarization, segmented retrieval, and result caching.

    It is therefore too simplistic to assume that more tokens always produce better results. A better objective is to increase the amount of useful information generated per token and reduce unnecessary content while maintaining answer quality.

    Comparison with a Model in the Same Cost Tier

    At an approximate cost level of RMB 50, the new version costs about RMB 52.5 per 1,000 calls in actual testing. Another model in the same tier costs approximately RMB 53.5. Both models achieved an overall accuracy of 71.5%, but the new version ranked fifth overall, while the comparison model ranked sixth.

    The difference is more visible in response speed. The new version took approximately 96 seconds on average, compared with about 72 seconds for the comparison model.

    This means the new version is competitive in accuracy and overall ranking but has a disadvantage in latency.

    The result reinforces the point that model selection should not be based solely on aggregate scores. For offline report generation, batch document processing, or background auditing, longer response times may be acceptable, making higher accuracy more valuable. For user-facing applications—particularly real-time question answering, online customer service, and interactive assistants—latency becomes a critical metric.

    Recommendations for Enterprise Evaluation

    Before procurement or production deployment, organizations should build their own business-specific test set instead of relying entirely on public benchmarks. The test set should cover at least the following categories:

    1. High-frequency real-world questions: Common requests made by everyday users.
    2. High-risk questions: Scenarios that test whether the model produces seriously misleading answers.
    3. Long-context questions: Tasks involving long documents and multi-turn conversations.
    4. Format-constrained questions: Tests of whether the model can consistently produce structured output.
    5. Abnormal inputs: Ambiguous wording, missing information, and contradictory conditions.
    6. Tool-collaboration tasks: Tests of parameter generation, call order, and exception handling.

    The evaluation should also record first-token latency, total response time, success rate, average token consumption, retry rate, and the percentage of output requiring manual modification.

    Only by evaluating these metrics together can an organization determine whether a model is genuinely suitable for its business.

    Conclusion

    The new model version delivers clear improvements in Chinese-language general capability, reasoning and mathematical computation, education, healthcare, finance, language understanding, and instruction following. Accuracy increased from 68.1% to 71.5%, and its overall ranking improved significantly.

    These gains come with higher token consumption, longer response times, and increased calling costs. At the same time, performance in AI agents and tool calling declined slightly.

    The new version is therefore better suited to applications that prioritize answer quality, can tolerate some latency, and have sufficient budget. For strongly real-time applications, organizations should reduce latency and cost through model routing, caching, asynchronous processing, prompt compression, and result reuse.

    The final technical decision should not stop at asking which model has the higher ranking. It should answer three more important questions:

    • Can the model solve real business problems?
    • Is the cost per unit of useful performance acceptable?
    • Does the system provide sufficient safety validation?

    A model upgrade creates real engineering value only when accuracy, efficiency, cost, and risk are balanced together.