Evaluating LLM Chinese-Language Performance: Methods and Cost-Benefit Analysis

    This article explains how to build a systematic evaluation for large language models, with a focus on Chinese-language ability. It covers four core metrics: accuracy, response time, token consumption, and call cost. It also describes how to build a reliable question set, interpret results with statistical care, run a reusable benchmark script, and set engineering practices before launch.

    1. Why Systematic Evaluation Matters

    Before a model goes live, teams usually need answers to three questions. Can its accuracy meet business requirements? Can its response time support an interactive experience? Can its call cost fit within the budget? A few demos or scattered question-and-answer examples cannot reliably answer these questions. Systematic evaluation turns subjective impressions into repeatable, comparable numbers, and it applies the same standard to every model and version.

    Teams should also decide how the results will be used before starting. For model selection, the focus is comparing accuracy and cost on the same tasks. For version regression, the focus is confirming that the new version has not declined in key scenarios. Different goals call for different question set designs and metric weights.

    2. Breaking Down the Four Core Metrics

    1. Accuracy

    Accuracy is the proportion of correct answers out of the total number of questions. For multiple-choice questions, true/false questions, and calculation questions with standard answers, results can be compared directly. For open-ended questions, teams must define a scoring rule in advance. The rule should specify, for example, whether key points are covered, whether the conclusion is correct, and whether there are factual errors.

    Accuracy should be broken down by dimension. Common dimensions include language understanding, instruction following, mathematical and logical reasoning, and domain-specific knowledge. Two models with the same overall score can differ substantially across dimensions.

    2. Response Time

    Response time is the end-to-end duration from sending a request to receiving the complete result. Single measurements vary widely, so use the mean and also track percentiles such as P50 and P95. P95 reflects tail latency. It often affects user experience more directly than the mean does.

    Response time depends on output length, server load, and network conditions. When comparing two models, test them under the same network environment and at similar times of day.

    3. Token Consumption

    A token is the basic unit that a model uses to process text. For each call, record input tokens and output tokens separately, because most billing schemes charge different unit prices for each.

    The number of output tokens also affects response time. If one model produces noticeably longer output, its latency may be higher even when its per-token inference speed is the same. When analyzing latency, check token counts as well. This helps distinguish “faster inference” from “shorter output.”

    4. Call Cost

    The cost of a single call is calculated as follows:

    Cost per call = input tokens × input price ÷ 1,000,000 + output tokens × output price ÷ 1,000,000

    To measure cost as “cost per 1,000 calls,” multiply the average cost per call by 1,000. Read cost together with accuracy to determine the cost per correct result. This measure is more informative than price alone.

    3. Building a Reliable Question Set

    The quality of the question set determines whether the evaluation conclusions can be trusted. Keep the following points in mind when building one.

    Stratified sampling. Allocate questions by business domain, question type, and difficulty. This prevents any single category from dominating the set and skewing the overall score.

    Deduplication and leakage prevention. If identical or highly similar questions appear in both the evaluation set and the model’s training data, the scores will be overestimated. When this cannot be fully ruled out, record the source and version of each question, and state the risk in the conclusions.

    Annotation consistency. Subjective questions should be labeled independently by at least two people, and their agreement should be measured with a metric such as Cohen’s kappa. Low agreement indicates that the scoring rules need revision.

    Sample size. Larger samples produce more stable conclusions. For a binomial proportion, the standard error can be estimated as √(p(1−p)/n). Consider this hypothetical example: if accuracy p is about 0.75 and the question count n is 10,000, the standard error is about 0.43 percentage points, and the 95% confidence interval is about ±0.85 percentage points.

    4. Principles for Interpreting Results

    Do not look only at the overall score. When the total score rises, some dimensions may fall. Compare results by dimension, and flag the dimensions with the largest changes. These dimensions usually matter most to the business.

    Compare differences against confidence intervals. A 1 percentage point gap in accuracy between two models is not necessarily statistically meaningful. Using the example above, if each group answers 10,000 questions, the standard error of the difference is about 0.6 percentage points. The difference must exceed about 1.2 percentage points before you can reasonably conclude that a real difference exists.

    Read latency and cost together with accuracy. A model that is faster and cheaper but noticeably less accurate may not suit the business. Average cost per correct answer and average latency per correct answer are useful supplementary metrics.

    Align offline results with online metrics. Offline evaluation provides a stable baseline for comparison, but it cannot fully represent the distribution of real user requests. After launch, keep tracking key business metrics to verify the offline conclusions.

    5. A Reusable Evaluation Script

    The following example shows how to calculate accuracy, average latency, average token usage, and cost per 1,000 calls in a consistent way. call_model is the function that calls the model under test. Users implement it according to their own API. Its return value must include the reply text and the input and output token counts.

    1. import time
    2. def run_benchmark(samples, call_model, price_in, price_out):
    3. """
    4. samples: list of dicts, each containing prompt, answer, and domain
    5. call_model: call function that returns a dict containing text, input_tokens, and output_tokens
    6. price_in: price per million input tokens (in CNY)
    7. price_out: price per million output tokens (in CNY)
    8. """
    9. correct = 0
    10. latencies = []
    11. total_in = 0
    12. total_out = 0
    13. domain_correct = {}
    14. domain_total = {}
    15. for item in samples:
    16. start = time.perf_counter()
    17. resp = call_model(item["prompt"])
    18. latencies.append(time.perf_counter() - start)
    19. total_in += resp["input_tokens"]
    20. total_out += resp["output_tokens"]
    21. # Exact matching. For subjective questions, replace with a custom scoring function.
    22. hit = resp["text"].strip() == item["answer"].strip()
    23. correct += int(hit)
    24. d = item["domain"]
    25. domain_total[d] = domain_total.get(d, 0) + 1
    26. domain_correct[d] = domain_correct.get(d, 0) + int(hit)
    27. n = len(samples)
    28. avg_in = total_in / n
    29. avg_out = total_out / n
    30. avg_cost = (avg_in * price_in + avg_out * price_out) / 1_000_000
    31. latencies.sort()
    32. p95_index = min(int(n * 0.95), n - 1)
    33. return {
    34. "accuracy": correct / n,
    35. "avg_latency_s": sum(latencies) / n,
    36. "p95_latency_s": latencies[p95_index],
    37. "avg_tokens": avg_in + avg_out,
    38. "cost_per_1000_calls": avg_cost * 1000,
    39. "domain_accuracy": {
    40. d: domain_correct[d] / domain_total[d] for d in domain_total
    41. },
    42. }

    Keep three points in mind when using the script. First, run a small number of samples as a warm-up before the formal evaluation, so that first-connection overhead does not count toward latency. Second, fix the random seed and sampling parameters so that runs remain comparable. Third, record failed and timed-out requests separately instead of discarding them. Otherwise, accuracy will be overestimated.

    6. Engineering Recommendations Before Launch

    1. Run regression tests regularly. After any change to the model version, prompts, or service parameters, rerun the same question set. Save historical results so that changes can be traced.
    2. Set cost budgets and alerts. Include cost per 1,000 calls in monitoring, and use monitoring and alerting capabilities to set thresholds. This helps prevent costs from exceeding expectations during traffic peaks.
    3. Handle failures and timeouts. Set a timeout for each call. Use exponential backoff for retryable errors. When the failure rate exceeds a threshold, switch to a fallback plan.
    4. Log call details. Use a logging service to record token counts, latency, and result status for each call. These logs support post-incident review and provide real data for future evaluations.
    5. Set acceptance criteria for each scenario. Customer service, code assistance, and document generation differ in how sensitive they are to accuracy, latency, and cost. Set separate acceptance thresholds for each scenario instead of applying a single overall score threshold.

    The value of an evaluation system lies in letting a team make decisions against the same standard. Once the question set, metrics, and script are fixed, every later model change can produce conclusions that are directly comparable.