LLM Evaluation: Balancing Accuracy, Latency, and Cost
This article explains why leaderboard scores can mislead LLM selection and presents a practical evaluation framework based on a Chinese-language, logic-heavy benchmark. It covers benchmark design, metrics, cost calculation, cross-generation comparison examples, reproducible evaluation pipelines, and enterprise selection practices. All model names and results in the examples are illustrative.
Why Leaderboard Scores Can Mislead Model Selection
In the past year, leading large models have iterated quickly, and any benchmark report can become outdated within weeks. Many teams select models by looking only at overall leaderboard positions. After deployment, they discover two opposite problems: production traffic is mostly simple intent recognition, yet they paid for the latency and price of a flagship model; or their workload contains many multi-hop reasoning tasks, and a lightweight model suffers a sharp accuracy drop.
The root cause is simple: an evaluation score is not a constant; it is a function. It is jointly determined by model × question set × scoring method × invocation parameters. Conclusions drawn without the specific question set or task distribution are hard to transfer.
A valuable evaluation should answer at least four questions:
- Correctness: What is the accuracy under the target language and task distribution?
- Speed: What are the end-to-end latency, time to first token, and output throughput?
- Cost: What is the cost per 1,000 calls, and the effective cost per correct answer?
- Stability: What happens across repeated runs, rate limiting, timeouts, retries, and fallback behavior?
One note: this article is a methodology demonstration. Unless otherwise stated, model codenames, versions, rankings, and numbers are illustrative examples. They do not correspond to any public vendor leaderboard and should not be treated as conclusions about any specific product.
Benchmark Design: Strictness Determines the Value of the Results
Scale, Sources, and Annotation Consistency
The example question set contains about 15,000 questions. It covers Chinese language and instruction following, education, finance, law and public administration, healthcare and mental health, and multimodal tasks.
Question sources should include three categories:
- Self-built questions: constructed around real business needs, especially private formats, long instructions, and conflicting information.
- Adapted public questions: rewritten to avoid contamination through paraphrasing, context replacement, and difficulty enhancement.
- Desensitized production samples: converted into evaluation questions after removing sensitive information, making them closer to real traffic distributions.
Decontamination is also critical. For English benchmarks, a common issue is that public corpora have already entered pretraining data. For Chinese question sets, memorization hit rates can be reduced through newly created questions, private formats, recent contexts, and compositional constraints.
Domain ratios and annotation consistency should be documented. The example ratio is:
- Chinese language and instruction following: 35%
- Education: 15%
- Finance: 10%
- Law and public administration: 10%
- Healthcare and mental health: 10%
- Multimodal: 10%
- Codebases and long context: 10%
Annotation should use double review. Questions with agreement below 95% should be marked as disputed.
Why Chinese Text and Strong Reasoning Should Be Tested Strictly
Choosing “Chinese plus strong logic” as the main evaluation dimension is a high-value strategy:
- Higher discrimination: English leaderboards often compress the gap among top models because many have approached saturation. Chinese long instructions, entailment, implicit premises, and legal reasoning still show clear gradients.
- Stronger resistance to contamination: self-built Chinese questions, recent-context questions, and private-format questions are harder for models to memorize.
- Closer to real business requests: Chinese word segmentation ambiguity, honorific and formality levels, administrative divisions, and industry terminology are common in real requests.
Multimodal Capabilities Must Be Evaluated Separately
Native multimodal models are not merely models with one additional input type. They must handle complex materials such as video, audio, images, and codebases. A text-only question set cannot cover these capabilities, so a separate multimodal evaluation set is required.
The following multimodal results are illustrative examples used to explain the evaluation method:
| Capability | Example test | Example result |
|---|---|---|
| Long-video understanding | Locate event timestamps in videos lasting tens of minutes | Median timestamp error: 3.2s; event recall: 86% |
| Cross-modal retrieval | Answer structured questions given charts or audio clips | Numerical reading accuracy: 91.5% |
| Image-text mixed instructions | Combine images, long documents, and constraints | Item-level instruction-following rate: 84.2% |
| Dialect and noisy speech | Transcription-based QA with accents and background noise | Character error rate: 12.8%; answer accuracy: 79.4% |
| Codebase QA | Cross-file dependency localization and modification suggestions | Localization accuracy: 81.3%; compile pass rate: 74.6% |
| Long-context effectiveness | Multi-needle retrieval and global aggregation | 64k-context multi-needle retrieval accuracy: 78% |
These tests matter because text accuracy only reflects the language core. It cannot represent cross-modal capability. If a title or abstract claims “multimodal optimization,” the body must provide multimodal metrics rather than only recommendations.
Metrics: Quantifying Cost and Speed
The example evaluation collects the following core metrics:
- Accuracy: primarily rule-based scoring, with model-based scoring as support, measuring final-answer correctness.
- Average latency: end-to-end time from request initiation to the last returned token.
- Average token consumption: total tokens used per call, including input and output.
- Cost per 1,000 calls: monetary cost for every 1,000 successful calls.
- Effective cost: cost per 1,000 calls divided by accuracy, representing the cost per 1,000 correct answers.
- Latency percentiles: P95 and P99 are often more meaningful than the mean.
The cost example is defined as follows:
Input price: 10 CNY per million tokensOutput price: 122.3 CNY per million tokensAverage input tokens: 1200Average output tokens: 1950Measurement scope: billed on successful calls only; failed retries excluded; caching disabledPer-call cost = 1200 × 10 / 1e6 + 1950 × 122.3 / 1e6≈ 0.2505 CNYCost per 1,000 calls = 1000 × 0.2505 ≈ 250.5 CNYEffective cost = 250.5 / 74.8% ≈ 334.9 CNY per 1,000 correct answers
The key is to state the measurement scope explicitly: whether retries are included, whether caching is enabled, whether batch processing is used, and how input/output tokens are counted. Otherwise, cost numbers can be misread, sometimes with an error of several orders of magnitude.
Example Cross-Generation Comparison: From “More Accurate” to “Faster and More Accurate”
The following table uses illustrative data:
- Model codenames: M1 is the previous-generation preview version; M2 is the new-generation version.
- Version snapshots: M1 uses
m1-prev-snapshot; M2 usesm2-new-snapshot. - Question set source: self-built Chinese comprehensive set
Sample-CN-15k. - Ranking scope: internal illustrative ranking, not a public leaderboard.
- Scoring method: rule-based scoring first, model-based scoring as support, with sampled human review.
| Metric | M1 previous generation | M2 new generation | Change |
|---|---|---|---|
| Overall accuracy | 72.5% | 74.8% | +2.3 pp |
| Internal illustrative ranking | 6th | 2nd | +4 positions, not a public leaderboard |
| Chinese language and instruction following | 67.5% | 72.4% | +4.9 pp |
| Education | 64.8% | 68.6% | +3.8 pp |
| Finance / law and public administration | 65.1% | 66.5% | +1.4 pp, illustrative |
| Healthcare and mental health | 88.7% | 88.7% | Flat, near saturation |
| Average latency | 64s | 53s | -17.2% |
| Average token consumption | 3119 | 3157 | +1.2% |
| Average TTFT | 13.0s | 10.5s | -19.2%, illustrative |
Gains Are Concentrated in Instruction Following
The 4.9 pp improvement in Chinese language and instruction following is one of the most notable subdomain gains in the example. This metric is especially valuable to developers because real system prompts often contain many constraints, such as output must be JSON, fields must not be added, maximum length, no fabricated citations, and conclusion before reasoning. If each single constraint is weaker by one percentage point, the combined effect across ten constraints becomes a visible failure-rate increase.
The 3.8 pp improvement in education also reflects better performance in multi-step problem solving and format adherence.
Healthcare and Mental Health Plateau: How to Read Saturation
The flat result in healthcare and mental health does not mean the model has no value. It means this subdomain is close to saturation. Further improvement requires harder questions, such as multi-turn follow-up, conflicting information, boundary and taboo cases, and crisis intervention paths.
The evaluation boundary must also be stated: this content is used only for model capability evaluation. It does not constitute medical advice, diagnosis, or a crisis intervention plan. For mental health and medical-compliance scenarios, professional review, safety filtering, and human review mechanisms should be introduced.
Latency Improvement: End-to-End Versus Generation Stage
Average latency dropped from 64s to 53s while token consumption increased only slightly by 1.2%. If approximated by total tokens, the average end-to-end output rates are:
M1: 3119 / 64 ≈ 48.7 tokens/sM2: 3157 / 53 ≈ 59.6 tokens/s
However, this statement needs caution. This is an “average end-to-end output rate.” It mixes input processing, queueing, TTFT, network transmission, and the generation stage. It alone cannot prove that decoding throughput improved.
To determine whether the gain came from shorter queueing or faster generation, TTFT, output-stage throughput, and concurrency data must be collected.
In the example, TTFT decreased from 13.0s to 10.5s, suggesting improvements in queueing, first-token generation, or scheduling. Therefore, a more reliable conclusion is: this version achieved overall end-to-end latency optimization, possibly through a combination of scheduling, first-token latency, throughput, or inference optimization, rather than simply “faster decoding.”
Reproducible Evaluation Pipeline
A one-time evaluation has limited value. An evaluation that can be repeatedly executed and used for regression comparison is an asset. The following process should be fixed:
Pin versions. Evaluations must be pinned to specific snapshot versions, not rolling aliases. System prompts, question order, and few-shot examples must remain byte-level consistent throughout the evaluation run.
Fix parameters. Evaluations aim for reproducibility. Temperature is usually set to 0, and top_p, maximum output length, and stop sequences should be fixed.
Collect latency breakdown. Recording only total time is insufficient. TTFT, total latency, and output rate must be recorded separately to distinguish queueing latency, first-token latency, and generation latency.
Use dual scoring tracks. Rule-based scoring is used for deterministic answers such as numbers, options, and JSON fields. Model-based scoring is used for open-ended answers. Model-based scoring should fix the judge model version and rubric, and sample results should undergo human verification.
Design for robustness. Each test case should have a timeout and a limited number of retries. Retries must not be included in latency statistics, otherwise latency will be systematically overestimated. Concurrency, rate-limit thresholds, and failure-reason distributions should be archived.
Persist results and run regression. Raw responses, token usage, latency breakdown, and scoring details should be written to object storage and partitioned by version. After any model or prompt change, run a full regression. If any metric regresses beyond a threshold, trigger an alert.
Example invocation code:
def run_case(case):last_err = Nonefor attempt in range(3):t0 = time.perf_counter()try:resp = client.generate(model=MODEL_SNAPSHOT,messages=build_prompt(case),temperature=0.0,top_p=1.0,max_output_tokens=cfg.max_out,stream=True,timeout=cfg.timeout,)t_first = Nonetext = ""usage = Nonefor chunk in resp:if t_first is None:t_first = time.perf_counter()text += chunk.deltaif chunk.usage is not None:usage = chunk.usaget_end = time.perf_counter()if t_first is None:t_first = t_endif usage is None:usage = {"prompt_tokens": estimate_tokens(build_prompt(case)),"output_tokens": estimate_tokens(text),}return {"case_id": case.id,"passed": grade(case.gold, text),"ttft_s": t_first - t0,"total_s": t_end - t0,"out_tps": usage["output_tokens"] / max(t_end - t_first, 1e-6),"in_tokens": usage["prompt_tokens"],"out_tokens": usage["output_tokens"],"attempt": attempt + 1,}except (TimeoutError, ConnectionError, ValueError) as e:last_err = ereturn {"case_id": case.id,"error": str(last_err),"count_in_latency": False,}
The key points are: failed retries are not mixed into latency statistics; token usage has a fallback estimate when missing; and each successful call records TTFT, total latency, and output rate separately.
Enterprise Selection and Deployment Recommendations
Use Scorecards Instead of a Single Ranking
Scores should be weighted by scenario:
Score = w1 × accuracy_for_target_task+ w2 × (1 - normalized latency)+ w3 × (1 - normalized cost per 1,000 calls)+ w4 × stability score
Example weights:
- Customer support Q&A: accuracy 0.45, latency 0.25, cost 0.20, stability 0.10.
- Offline document processing: accuracy 0.45, latency 0.05, cost 0.15, stability 0.35.
- Complex reasoning and codebase tasks: accuracy 0.60, latency 0.10, cost 0.10, stability 0.20.
The stability score should include P95, P99, timeout rate, rate-limit trigger rate, and variance across repeated runs.
Tiered Routing Is Often More Effective Than Switching Models
Production cost efficiency usually comes from a combination of models rather than a single model:
- Simple requests such as intent recognition, format conversion, and summarization should be handled by smaller or faster models.
- Escalate to a flagship model only when confidence is insufficient, the task is judged to require multi-hop reasoning, or long-document understanding is needed.
- Engineering-wise, routing, fallback, caching, and monitoring can be implemented at a unified gateway layer.
Common Cost-Control Levers
- Semantic caching and exact caching: repeated questions can be served directly, significantly reducing call volume.
- Batch processing: offline tasks are not constrained by interactive latency and can use lower-cost modes.
- Output length constraints: much waste comes from verbose answers, so word or field limits should be set.
- Retrieval before long context: for long-context scenarios, retrieve first and then send relevant content, rather than inserting the entire corpus.
- Retry strategy optimization: failed retries should be counted separately to avoid hidden costs.
Three Disciplines for Interpreting Results
- Accuracy cannot be compared across different question sets. It can only be compared with historical versions evaluated on the same question set.
- Average latency is affected by platform queueing, regional network conditions, concurrency, and rate limiting. It is not the model’s inherent compute capability.
- Percentiles are more decision-relevant than means. Business experience is driven by P95 and P99 latency, not the average.
Frequently Asked Questions and Conclusion
Common questions fall into three categories:
How should I choose a model? First look at the target task distribution, then examine accuracy, effective cost, P95/P99 latency, and stability, rather than only the overall leaderboard.
What are common mistakes? Comparing scores across different question sets, looking only at average latency, ignoring token cost, ignoring retries and caching, and replacing multimodal capability with text-only questions.
What should be watched carefully? High-risk domains such as healthcare and mental health require professional review and boundary statements; cost numbers must state their measurement scope; evaluation data must be traceable and reproducible.
This example provides a reusable observation pattern: on a strict Chinese-language question set, accuracy improved to 74.8%, average latency dropped from 64s to 53s, token consumption increased only slightly by 1.2%, and the cost per 1,000 calls was about 250.5 CNY. Simultaneous improvement in accuracy, efficiency, and cost is uncommon. But the method matters more than the conclusion: making question-set design, metric definitions, latency breakdown, dual scoring, and the regression pipeline into repeatable assets allows technical selection to be based on data rather than impressions in a rapidly iterating model environment.
