A Guide to Evaluating Coding Ability and Inference Latency for China-Developed LLMs
Leaderboard scores do not tell you whether a model finishes your task, how long one request takes, or whether latency holds as concurrency rises. This guide defines shared latency and quality metrics, then covers stability checks, reproducible coding tasks, load tests, service-level targets, logging, and a report template you can recalibrate on your own hardware, prompts, and data.
1. Leaderboard scores cannot drive model selection on their own
A composite leaderboard score summarizes average performance on one benchmark set. It does not show whether the model is stable on your tasks, whether it is fast enough, or whether speed drops as concurrency increases. Selection needs three answers: can the model complete the task, how long does one completion take, and can it keep that speed as load rises.
This article does not cite unpublished numeric results. Thresholds and examples here are calibration starting points. Teams should remeasure them on their own hardware, prompts, and business data.
2. Agree on metric definitions first
The same word can mean different things across teams. “Throughput,” for example, can mean output tokens per second or completed requests per second. Before evaluation starts, write down each metric’s definition and how it is collected.
| Metric | Definition | How to collect it |
|---|---|---|
| Time to first token (TTFT) | Time from the client sending the request until the first useful output arrives | Client records the send time and the arrival time of the first non-empty streaming chunk |
| Reasoning-phase time | Time spent while the model emits reasoning content | Split by phase markers returned by the server; if there are no markers, state the fallback rule in the report |
| Answer-phase time | Time from the end of reasoning to the last answer token | Use the same timestamps as the reasoning phase |
| Total time | Time from request start until the stream ends | Single-point timing on the client |
| Answer speed | Answer token count divided by answer-phase time | Count tokens with the same tokenizer and record the tokenizer version |
| P50 / P95 / P99 | Percentiles of the latency distribution | At least 200 requests per group; do not report P99 when the sample is too small |
Two points matter. First, measure TTFT and total time on the client, because the gateway and network path are part of the latency users feel. Second, percentiles are sensitive to sample size. With too few samples, P95 swings widely, so the report must state the sample count.
The example below samples a client stream. Field names can be used directly in a logging system.
import timedef measure_stream(client, prompt, request_id):t_send = time.perf_counter()t_first = Nonet_reason_end = Nonet_last = Noneanswer_tokens = 0for chunk in client.stream(prompt):now = time.perf_counter()if t_first is None and chunk.text:t_first = nowif chunk.phase == "reasoning":t_reason_end = nowif chunk.phase == "answer" and chunk.text:answer_tokens += chunk.token_countt_last = nowif t_last is None:return {"request_id": request_id, "error": "empty_stream"}ttft_ms = (t_first - t_send) * 1000 if t_first else Nonetotal_ms = (t_last - t_send) * 1000if t_reason_end is not None:reason_ms = (t_reason_end - t_send) * 1000answer_ms = (t_last - t_reason_end) * 1000else:reason_ms = Noneanswer_ms = total_msreturn {"request_id": request_id,"ttft_ms": ttft_ms,"reasoning_ms": reason_ms,"answer_ms": answer_ms,"total_ms": total_ms,"answer_tokens": answer_tokens,}
If the API does not return a reasoning-phase marker, record reasoning_ms as empty and say so in the report. In that case you can compare only TTFT and total time. You cannot claim that the reasoning phase is faster or slower.
3. Capability evaluation: from a single score to stability
Overall capability can be split into four layers: basic skills (Q&A, summarization, rewriting), reasoning under constraints (multi-step calculation, conditional constraints), engineering output (JSON or table output, function-call descriptions), and business simulation (real user questions and past failure cases). The item count and source for each layer should be written into the evaluation notes.
Checking whether one answer is correct is not enough. You also need to see whether results are stable. Three common methods are:
- Run the same item five times and measure answer consistency. Open-ended items can be scored by a reviewer model or by people using one shared rubric.
- Run the same item set at different temperature settings and record how accuracy changes with temperature. A model whose accuracy falls sharply as temperature rises is a poor fit for tasks that need deterministic output.
- Apply small prompt perturbations—change punctuation, swap the order of constraints, or add an irrelevant prefix—and see whether the model still completes the task.
Record these fields for each case:
{"case_id": "reason-017","category": "constraint","correct": true,"format_valid": true,"consistency_5runs": 0.8,"perturbation_pass": false,"answer_tokens": 412}
format_valid should be decided by a program, for example a JSON parser, not by reviewing each output by hand. Programmatic checks can be reproduced and make batches easier to compare.
4. Coding evaluation: accept work with reproducible tasks
Coding evaluation should cover six task types: code completion, function generation, bug fixing, refactoring, code explanation, and multi-file applications. Each type needs a written input specification and pass criteria. Otherwise two evaluation runs cannot be compared.
Take a 3D application as an example. The source material mentioned a “3D solar system” and a “3D growing tree,” but did not give task specs or pass criteria. A reproducible spec looks like this:
- Deliverable: a single HTML file, with the 3D library pinned to one explicit patch version.
- Content: eight celestial bodies, each with orbital motion; the camera can be rotated and zoomed with the mouse; the page has pause and resume controls.
- Pass criteria: no console errors after load; the render loop keeps running; the pause control toggles state; the aspect ratio stays correct after the window is resized.
- Typical failure reasons: missing dependencies, calls to APIs that do not exist, a wrong camera aspect ratio, or memory that keeps growing after five minutes of animation.
The evaluation script should record the first generation and the result after repair separately.
def evaluate_codegen(model, spec, sandbox, max_repair=2, timeout_s=30):code = model.generate(spec)result = sandbox.run(code, timeout_s=timeout_s)first_pass = result.successrepairs = 0while not result.success and repairs < max_repair:code = model.repair(code, result.error_log)result = sandbox.run(code, timeout_s=timeout_s)repairs += 1return {"first_pass_success": first_pass,"final_success": result.success,"repair_rounds": repairs,"loc": code.count("\n") + 1,"wall_time_s": result.wall_time_s,}
The design point is that first_pass_success reflects one-shot generation quality, while final_success reflects repair ability in a closed loop. Report the two metrics separately. Do not merge them into one score.
Line count is not a capability metric on its own. Longer code may contain redundant logic, and shorter code may omit edge cases. A better practice is to read line count together with test pass rate, and to record the share of duplicated code.
5. Load testing and capacity planning
Load testing answers a different question: once request volume rises, does latency run away?
Capacity can be estimated with Little’s law. Average concurrency in the system equals arrival rate times average processing time. For example, a target of 2 requests per second and an average total time of 8 seconds implies about 16 concurrent requests. With a 30% margin, plan for about 21 concurrency slots. Replace this number with measured load-test results. Do not reuse it as-is.
A practical approach is to raise load in steps. Set concurrency to 1, 4, 8, 16, and 32. Hold each level for a period and record P95 latency and error rate. When P95 at a level is more than 50% above the baseline, or the error rate exceeds 1%, you have found the latency knee. The concurrency at that knee is the safe upper limit for one instance.
Long context needs its own test. Use requests with about 2k, 16k, and 64k input tokens and watch how TTFT changes with input length. If TTFT grows clearly faster than input length, prefill may be the bottleneck, and you should consider cache reuse or splitting the request.
A load-test report must record: model name and version, inference-engine version, GPU or CPU model, batching configuration, maximum output tokens, temperature, and prompt-template version. Results that omit these fields cannot be reproduced or compared with other batches.
6. Service-level objectives: define them, then calibrate
Service-level objectives turn business requirements into numbers you can check. The configuration below is a starting point. Calibrate every number with A/B tests or user feedback.
slo:ttft_ms_p95: 1500 # upper bound on time to first token for real-time Q&A; calibrate with UX teststotal_ms_p95: 15000 # acceptable wait for one code-generation request; set separately by task typeanswer_tokens_per_s_min: 40 # minimum answer-phase speed; below this, treat the service as degradederror_rate_max: 0.01 # measured on a rolling 1-hour windowdegradation:trigger: queue_wait_ms_p95 > 3000 for 5 minutesactions:- switch to a lower-intensity inference configuration- cap max_output_tokens at the smallest value the task allows- move non-real-time tasks into an async queue
Set trigger and recovery conditions as a pair. A trigger without a recovery condition can leave the service on a low-quality configuration for a long time. A reasonable recovery condition is staying under 70% of the threshold for 15 consecutive minutes.
7. Observability: log fields must answer questions
Logs should let the team answer three questions: where it is slow, where it fails, and when it started getting worse.
Each request log should include: request ID, task type, model version, input token count, output token count, TTFT, reasoning time, answer time, total time, queue wait, whether it timed out, whether degradation was triggered, and the error code.
Queue wait is often ignored, but it is frequently the main source of latency at peak. If queue time keeps rising as a share of total time, scale out or change routing before trying to optimize the model itself.
8. Evaluation report template
The report should include quality, performance, and operations data together. Fill unmeasured fields with null. Do not estimate them.
{"scenario": "code_generation","model": {"name": null, "version": null, "inference_engine": null},"hardware": {"gpu": null, "concurrency_tested": []},"quality": {"first_pass_success_rate": null,"final_success_rate": null,"consistency_5runs": null},"performance": {"ttft_ms_p50": null,"ttft_ms_p95": null,"total_ms_p95": null,"answer_tokens_per_s_median": null,"sample_size": null},"ops": {"error_rate": null,"queue_wait_ms_p95": null,"degradation_triggered": null},"reproducibility": {"prompt_template_version": null,"temperature": null,"max_output_tokens": null}}
