Chinese LLMs Keep Delivering Dark Horses: Evaluation Methodology and Technical Analysis
This article explains how to evaluate large language models (LLMs) in a reproducible, explainable way instead of chasing a single benchmark run. It defines three core evaluation dimensions—general capability, inference speed, and coding ability—and lays out the metrics and common pitfalls behind each. It then shows how to turn subjective impressions into data through percentile-based statistics, a streaming stress-test script skeleton, and a five-step method for building your own evaluation set. Finally, it breaks down the engineering behind "turbo" inference modes, the feasibility and limits of AI-assisted model development, and what enterprises should weigh when selecting and deploying a model.
1. Behind the Hype: What One Head-to-Head Comparison Revealed
Recently, an overseas information platform ran a lightweight comparison that pitted a newly released model against the two most-used lightweight models on an aggregated inference-calling platform. The results placed the new model in the middle on overall capability, let it overtake the others on some coding sub-tasks, and showed impressive output speed in turbo mode. What drew even more attention was that its architecture design, training system, and inference optimization all involved deep AI participation.
For developers, this comparison offers three takeaways:
- Evaluation criteria are diversifying: Overall capability, coding ability, and inference speed were assessed at the same time, so a lead on a single dimension is no longer enough to justify a “first-tier” verdict.
- Sub-scenarios are diverging: The same model may lag on algorithm problems yet overtake others on complex application generation. Capability structure matters more than the aggregate score.
- The R&D paradigm is evolving: AI is moving from “code assistant” to “R&D participant.” Architecture design and engineering optimization now bring in AI capabilities whose reliability must be validated through engineering-grade methods.
This raises a practical question: faced with a constant stream of new models, how does a team build a reproducible, explainable evaluation method rather than being led around by a single benchmark run? The rest of this article covers four layers: evaluation dimensions, testing methods, the principles behind speed, and selection and deployment.
2. Three Core Evaluation Dimensions: Define Metrics Before Ranking
Evaluating an LLM starts with breaking the vague “is it good?” into quantifiable, reproducible metrics.
| Dimension | Core metrics | Typical testing approach | Common pitfall |
|---|---|---|---|
| General capability | Sub-scores for knowledge Q&A, long-context understanding, instruction following, logical reasoning, etc. | General benchmarks plus a custom scenario-based question bank | Looking only at the aggregate score and ignoring specific sub-scores |
| Inference speed | TTFT, output throughput, reasoning/answer phase speed, median response time | Multi-concurrency load testing combined with streaming-output statistics | Using peak instead of median; inconsistent test conditions |
| Coding ability | Runnable rate, test pass rate, code quality, complex-application completeness | Layered evaluation across algorithm problems, engineering tasks, and application generation | Looking only at algorithm-problem pass rate and ignoring engineering and interaction ability |
2.1 General Capability: Look at Weak Spots, Not the Aggregate
General capability measures how well a model understands and responds to a broad range of tasks, usually expressed as a composite score across several public benchmarks. But an aggregate score hides weak spots and can diverge from real business experience:
- Weighting changes the ranking: Different dimensions carry different weights; change the weights and the ranking may change with them.
- Capability structure matters more than the total: If your team mainly does long-document summarization, the long-context sub-score should carry far more weight than the aggregate.
- Self-testing beats leaderboards: Build 100–300 evaluation samples based on your business scenarios, cover the typical input distribution, and cross-check them against actual conversational experience.
2.2 Inference Speed: Separate “First-Token Wait” from “Sustained Streaming”
Speed testing answers two different questions:
- How long do you wait after hitting Enter before you see feedback? This maps to TTFT (time to first token).
- Can it keep streaming at high speed while generating a long answer? This maps to output throughput.
Real systems should also track P50/P90/P99 percentile latency. For interactive products, the harm from a few slow requests far outweighs the benefit of a better average.
Many models first emit their reasoning process in a thinking phase, then move into an answer phase that produces the formal result. The throughput of the two phases can differ substantially, and reasoning sometimes consumes tens of times more tokens than the answer itself, so segmented measurement is more informative than a single number.
A common approximation formula:
total response time ≈ TTFT + reasoning time + answer time≈ TTFT + reasoning tokens / reasoning throughput + output tokens / answer throughput
Take a typical call with the parameters below:
| Parameter | Value |
|---|---|
| TTFT | 3.1 s |
| Reasoning tokens | 3,000 |
| Reasoning throughput | ~980 tokens/s |
| Output tokens | 1,800 |
| Answer throughput | ~1,800 tokens/s |
This puts total response time at about 7.2 s. When reasoning tokens grow to 15,000, total time rises to about 19.4 s—which shows how strongly reasoning-chain length affects end-to-end latency.
So whenever you compare speeds, record:
- Test conditions: Context length, concurrency, sampling parameters (temperature, top_p, max output length), whether thinking mode is enabled, and the input/output length distribution.
- Statistical convention: Peak, mean, or median; and whether cold-start requests have been excluded.
- Throughput calculation convention: Usually output tokens divided by “total time minus TTFT,” though some measurements fold in reasoning tokens. When the convention differs, the numbers cannot be compared directly.
2.3 Coding Ability: From a Single Function to a Complete Application
Coding ability roughly splits into three levels:
| Level | Task type | Key focus |
|---|---|---|
| L1 Algorithms and single functions | LeetCode-style algorithm problems, single-function completion | Correctness, edge-case handling, complexity |
| L2 Engineering and multi-file | Code refactoring, bug fixing, unit-test generation | Cross-file understanding, interface consistency, maintainability |
| L3 Complex application generation | Interactive 3D visualization, animation, mini-games | Code runs, complete interaction, performance and resource management |
In this comparison, the new model stood out on 3D-style application generation—the tasks that reveal the widest gaps:
- Rendering an object is easy; the hard part is organizing interaction, animation, and geometry so they are all correct.
- Logical correctness matters more than lines of code.
- Details like physics animation and resource cleanup quickly expose a model’s engineering ability.
3. Turning Subjective Feelings into Data: A Reproducible Evaluation Method
3.1 Statistical Convention: Prefer Median and Percentiles
The raw evaluation showed the model peaking at 2,100 tokens/s per second in turbo mode, with reasoning and answer phases around 984 and 1,816 tokens/s respectively and a median response time of about 3.1 s. These numbers say it sits in the first tier—but they also point to a few cautions:
- Peak is a ceiling, not the daily reality. Peak depends on hardware specs, batching strategy, and the specific request, and is rarely sustainable.
- Segmented throughput reveals capability structure. The answer phase is clearly faster than the reasoning phase, indicating the bottleneck lies more in scheduling and batching than in generation itself.
- Median response time reflects only half of requests’ experience; tail latency needs P90/P99 to complete the picture.
A reliable speed conclusion should therefore include at least: sample size, percentile statistics, and complete test conditions.
3.2 Stress-Test Script Skeleton
Below is an evaluation skeleton based on streaming output. The HTTP call follows a generic API style for illustration; adjust it to your target service’s actual interface:
"""Lightweight model inference evaluation skeleton: concurrent stress test + metric collection.Notes:1. Endpoint, port, and auth info are injected via environment variables; do not hardcode them or write them to logs;2. Request fields below are a generic API style for illustration; adjust them to your actual interface;3. Throughput is computed as "answer-phase tokens / (total time - TTFT)"."""import jsonimport osimport statisticsimport timefrom concurrent.futures import ThreadPoolExecutorimport httpxBASE_URL = os.environ["MODEL_BASE_URL"]HEADERS = {"Authorization": f"Bearer {os.environ['MODEL_TOKEN']}"}CHAR_TO_TOKEN = 1.5 # rough conversion for mixed Chinese/English text; approximation onlydef measure_once(client, prompt, model, max_tokens=2048):"""Run a single streaming request and record per-phase timings and output sizes."""payload = {"model": model,"messages": [{"role": "user", "content": prompt}],"stream": True,"stream_options": {"include_usage": True},"max_tokens": max_tokens,"temperature": 0.2,}t0 = time.perf_counter()ttft = t_answer = Nonereasoning_chars = answer_chars = 0usage = Nonewith client.stream("POST", f"{BASE_URL}/v1/chat/completions",json=payload, headers=HEADERS) as resp:resp.raise_for_status()for line in resp.iter_lines():if not line.startswith("data: "):continuedata = line[6:].strip()if data == "[DONE]":breakobj = json.loads(data)usage = obj.get("usage") or usage # the last frame usually carries token statsfor choice in obj.get("choices", []):delta = choice.get("delta", {})if delta.get("reasoning_content"): # reasoning-phase deltareasoning_chars += len(delta["reasoning_content"])ttft = ttft or time.perf_counter()if delta.get("content"): # answer-phase deltaanswer_chars += len(delta["content"])ttft = ttft or time.perf_counter()t_answer = t_answer or time.perf_counter()total = time.perf_counter() - t0ttft = ttft or totalthink_secs = (t_answer - ttft) if t_answer else 0.0think_tokens = (usage or {}).get("reasoning_tokens") \or reasoning_chars / CHAR_TO_TOKENreturn {"ttft": ttft,"latency": total,"think_tps": think_tokens / think_secs if think_secs > 0 else None,"answer_tps": (usage or {}).get("completion_tokens", answer_chars / CHAR_TO_TOKEN)/ max(total - (t_answer - ttft) - ttft, 1e-6),}def evaluate(prompts, model, concurrency=4):results = []with httpx.Client(timeout=120.0) as client:with ThreadPoolExecutor(max_workers=concurrency) as pool:futures = [pool.submit(measure_once, client, p, model) for p in prompts]for fut in futures:results.append(fut.result())for key in ("ttft", "latency", "answer_tps"):values = sorted(r[key] for r in results)p90 = values[int(len(values) * 0.9) - 1] if len(values) > 1 else values[0]print(f"{key:10s} P50={statistics.median(values):.3f} P90={p90:.3f}")if __name__ == "__main__":# Test prompts should cover real business: short Q&A, long-document summarization, code generation, etc.PROMPTS = ["Summarize the core advantages of vector databases in three sentences.","Implement an LRU cache in Python that is thread-safe.","Given a 500-character Chinese summary, compress it to 100 characters while keeping key numbers.",] * 10evaluate(PROMPTS, model="your-model-id", concurrency=4)
Key points of the script:
- Attribution: TTFT is measured from the first delta’s arrival, and the reasoning/answer phases are separated at the first answer-phase delta—more granular than measuring total time alone.
- Statistics: It reports P50/P90 to avoid the randomness of a single request.
- Safety: Auth info is injected via environment variables and masked in logs; once an evaluation set is used for production decisions, confirm it contains no sensitive data.
3.3 The Five-Step Method for a Self-Built Evaluation Set
- Sampling: Draw typical requests from real business logs, covering long-tail scenarios and hard cases.
- Stratification: Layer by task type (Q&A, summarization, extraction, code generation, etc.) and difficulty so each category carries a sensible weight.
- Standard-setting: Fix reference answers, scoring rules, and executable test cases (e.g., unit tests) up front—avoid “changing the standard while testing.”
- Automation: Call the model → parse output → score/run tests → record metrics, with the whole pipeline scripted so results are traceable.
- Continuous regression: Re-run the evaluation set after model iterations or swaps, and track production behavior via logging services and monitoring/alerting to close the loop.
4. The Engineering Behind “Turbo” Speed
A model’s high throughput in “turbo mode” rarely comes from a single technique; it usually results from coordinated optimization of model architecture and the inference system.
4.1 Model Architecture Side
- Sparse activation architecture: A Mixture-of-Experts (MoE) design activates only a subset of expert parameters for each token, cutting per-token compute while preserving total parameter scale.
- Attention optimization: Grouped-query attention (GQA) and multi-query attention (MQA) let multiple query heads share Key/Value, significantly shrinking the KV cache; some methods use low-rank compression to reduce cache footprint further, which is especially effective for long context.
- Quantization and sparsity: Post-training quantization (FP8/INT4, etc.) and weight/activation sparsification reduce weight-read bandwidth and compute.
- Vocabulary and decoding efficiency: Optimizing tokenization against domain corpora reduces tokens per unit of text, directly lowering latency and cost.
4.2 Inference System Side
- Paged KV cache management: Splitting the KV cache into fixed-size blocks allocated on demand sharply reduces memory fragmentation and redundant usage, freeing space for high concurrency.
- Continuous batching: Dynamically inserting new requests and evicting completed ones during generation—rather than queueing in static batches—greatly improves GPU utilization.
- Speculative decoding: A small model or draft mechanism quickly proposes candidate tokens, which the large model then verifies in parallel; accepted tokens are kept and rejected ones are rolled back. With high-quality drafts, output throughput can multiply.
- Prefill/Decode disaggregation: Deploying the compute-heavy prefill phase and the memory-bound decode phase on separate resource pools reduces interference and makes independent scaling easier.
- Quantization and operator fusion: Quantizing weights and the KV cache, and fusing computations such as attention, cut kernel-launch overhead and ease the decode-phase bottleneck.
4.3 Service Layer
- Inference accelerators and custom chips: Hardware tuned for memory bandwidth or compute pays off clearly for the memory-bound decode phase.
- Streaming output: Mechanisms like SSE return the first token as soon as possible, improving perceived experience when TTFT is otherwise fixed.
- Request scheduling: Multi-level caching reuses common prefixes to cut repeated prefill work; scheduling by request length and priority prevents long requests from blocking short ones.
- Elasticity and stability: Autoscaling, timeout and retry policies, and multi-availability-zone disaster recovery.
5. AI Participating in Model Development: Feasibility and Boundaries
“AI participating in architecture, training-system, and inference-optimization development” is worth watching, but should be viewed rationally:
- What AI is good at: Well-defined, automatically verifiable tasks such as hyperparameter search, parallel-strategy and communication tuning, operator-kernel generation and performance tuning, and automatic inference-config search. These have objective signals (throughput, memory footprint) as feedback and suit a “generate–verify–iterate” loop.
- What AI is not good at: Model architecture innovation, training-objective design, and data-quality control—tasks that lack clear verification signals and still require researchers’ judgment.
- Verifiability is the prerequisite: Whether an architecture is better or a kernel is faster should rest on unified benchmarks and ablation studies, not a single experiment.
- Engineering requirements:
- Training and inference code generated by an LLM must pass automated tests, static checks, and human review.
- Training and evaluation data must be strictly isolated to prevent “AI writes the questions, AI answers them” overfitting.
- Keep complete model versions, configs, and experiment results for each iteration to ensure reproducibility and traceability.
- Inference optimization also needs regression tests, so a new optimization doesn’t “speed up” typical workloads while “slowing down” long-tail requests.
6. Selection and Deployment: What Enterprises Should Really Focus On
Scores are only the starting point; production demands a broader view:
- Capability match: Use your own evaluation set—run candidate models against real business samples instead of relying only on public aggregate scores.
- Cost: Calculate by input/output price per million tokens, and watch the billing convention (whether cache hits are distinguished). Estimate monthly cost from actual prompt length, reasoning-token consumption, and concurrency. In thinking mode, reasoning tokens often dominate.
- Latency and stability: Track P99 latency, TTFT, and availability (SLA) as primary metrics. Turbo mode usually means lower latency but also lower priority or higher price—choose based on the business.
- Security and compliance: Confirm data flows and data residency, contract and audit capabilities; route sensitive data through dedicated deployment or masking.
- Ecosystem and migration: Use a unified calling abstraction to shield interface differences, and manage auth, rate limiting, and canary releases via a service mesh or API gateway to avoid being locked to a single interface.
- Observability: Connect logging and monitoring/alerting to continuously track latency, throughput, error rate, and cost; archive request logs to object storage for replay, auditing, and evaluation-set construction.
To reduce infrastructure investment, you can leverage general-purpose capabilities from cloud platforms such as Baidu AI Cloud. For example, call models through a model-serving platform, host inference services on a container platform, decouple evaluation data with object storage and message queues, and use logging and monitoring/alerting for observability—so you can focus your energy on the evaluation system and the business itself.
7. Conclusion
The steady arrival of new models shows that competition among Chinese LLMs has shifted from “parameters and leaderboards” to a multi-dimensional contest over capability structure, inference efficiency, and engineering delivery. For developers and enterprises, what matters most is building your own evaluation system:
- Define your dimensions and metrics;
- Fix your test conditions;
- Use percentile statistics;
- Test against real business scenarios.
Only then can you move from “hype” to “actually useful,” making reproducible, explainable technical choices amid rapid model iteration.
