LLM Evaluation: Measuring General Capability, Coding Skills, and Inference Performance
Selecting a large language model (LLM) is no longer a question of availability but of fit, and a single leaderboard score rarely settles it. This article breaks LLM evaluation into three largely independent dimensions — general capability, coding capability, and inference performance — and explains how to assess each one. General capability is treated as a matrix of sub-skills covering knowledge accuracy, instruction following, structured output, and multilingual transfer; coding is assessed in layers, from single-function problems to multi-file application builds such as interactive 3D scenes and data-visualization pages; and inference performance is measured through throughput, time to first token (TTFT), median response time, and separate decoding speeds for the prefill and decoding phases, with a Python benchmarking example. The article closes with a five-step evaluation workflow and notes on infrastructure practices that turn lab scores into stable production behavior.
Why Capability Evaluation Matters for Model Adoption
With the number of large language models growing quickly, the first question for developers and enterprises is no longer “is there a model available” but “how do we pick the one that actually fits our use case.” Discussions of “dark horse models” and leaderboard comebacks largely reflect an industry-wide need for evaluation methods that are objective, reproducible, and transferable.
Relying on a single aggregate score or one viral comparison tends to mislead. Strong performance on general question answering says nothing about how a model will hold up under complex coding tasks or high-concurrency inference. A rigorous evaluation system should therefore cover three related but largely independent dimensions: general capability, coding capability, and inference performance. Each is discussed below.
1. General Capability: Building a Multi-Dimensional Evaluation Matrix
General capability is a rough sketch of a model’s “overall intelligence.” It is not a single metric but a matrix of sub-capabilities, and an evaluation should break it down into at least the following areas:
- Knowledge breadth and accuracy: spanning general knowledge, science, humanities, and time-sensitive information, with emphasis on factual accuracy and resistance to hallucination.
- Language understanding and generation: long-context comprehension, instruction following, and the stability of structured output such as JSON or tables.
- Logical and mathematical reasoning: correctness of multi-step reasoning, constraint satisfaction, and numerical computation.
- Multilingual and cross-domain transfer: performance in non-native languages and in domains that are rare in the training data.
On methodology, two approaches are commonly combined: automated benchmarks plus human or model-based review.
- Automated benchmark testing: run a model against standardized public datasets and produce reproducible scores with standardized scoring scripts. It is cheap and highly comparable, though results may diverge from real business workloads.
- Subjective review (for example, head-to-head evaluation): two models produce anonymous outputs for the same prompt, and humans or a judge model vote in a blind comparison. This comes closer to the real usage experience, but it requires tight control over the prompt distribution and inter-rater consistency.
A practical recommendation: do not judge a model by its rank on a single leaderboard. A more sound approach is to build an internal evaluation set based on your own business’s prompt distribution, and let public benchmarks and the private set validate each other.
2. Coding Capability: From Code Generation to Complex Application Builds
Coding is one of the most closely watched dimensions in model evaluation, because it places heavy demands on reasoning chain length, syntactic rigor, and structured output — which tends to expose a model’s real weaknesses.
Coding evaluations can be divided into progressively harder tiers:
- Single-function and algorithm problems: correctness of basic syntax, common data structures, and standard algorithms. Good for fast screening.
- Engineering-level tasks: code completion, bug localization and repair, refactoring, and unit test generation — closer to a real development workflow.
- Complex application builds: the model starts from a requirement and produces a runnable multi-file project, such as an interactive 3D scene application, a frontend page with data visualization, or a graphics program that animates and renders progressively.
The third tier is especially discriminating. For a graphics application, the model must not only write correct rendering logic but also handle coordinate systems, lighting, animation loops, and performance optimization. Useful things to check:
- Runnability: does the code run as-is without errors?
- Feature completeness: are the core interactions described in the requirement actually implemented?
- Visual and logical quality: for instance, the layering of a 3D scene or how natural an animation looks.
- Maintainability: whether naming, comments, and module structure are clear.
The gap between models at this tier is often large: some produce a complete, runnable project in one pass, while others need several rounds of correction. Coding evaluations should therefore keep an observation window for multi-turn iteration rather than judging only the first output.
3. Inference Performance: Throughput, Latency, and Perceived Responsiveness
If the first two dimensions measure “how well it answers,” inference performance measures “how fast it answers.” In production, performance often determines both user experience and cost structure. At minimum, evaluate the following:
- Throughput (tokens/s): tokens generated per unit of time, reflecting overall generation efficiency.
- Time to first token (TTFT): the time from request submission to the first returned token, which directly drives whether users perceive lag.
- Median response time: the median total request time, a better indicator of the typical experience than the mean.
- Per-phase decoding speed: many inference systems separate a thinking or prefill phase from an answer or decoding phase, and token generation speeds can differ noticeably between them. Measuring them separately is more diagnostic.
For load testing in practice, a reasonable approach is:
- Compare under identical hardware and concurrency conditions, otherwise the numbers are not comparable.
- Measure decoding speed for the thinking phase and the answer phase separately to determine whether the bottleneck is in prefill or decoding.
- Report median and percentiles (P50/P90/P99), not just a single peak figure — peaks typically appear only in idealized, extreme scenarios.
The example code below shows a basic framework for measuring throughput and latency against an inference API (a demonstration of method only, not representative of any specific implementation):
import timeimport statisticsdef benchmark_inference(call_fn, prompt, runs=20):ttft_list, total_list, throughput_list = [], [], []for _ in range(runs):start = time.perf_counter()first_token_time = Nonetoken_count = 0for chunk in call_fn(prompt): # streaming responseif first_token_time is None:first_token_time = time.perf_counter()ttft_list.append(first_token_time - start)token_count += count_tokens(chunk)total = time.perf_counter() - starttotal_list.append(total)if total > 0:throughput_list.append(token_count / total)return {"TTFT_P50": statistics.median(ttft_list),"Total_P50": statistics.median(total_list),"Total_P90": sorted(total_list)[int(len(total_list) * 0.9)],"Throughput_avg": statistics.mean(throughput_list),}
With scripts like this, a team can establish its own performance baseline and run regression comparisons after a model upgrade or a change in deployment configuration.
4. Evaluation Methodology and Practical Recommendations
Bringing the three dimensions together yields a workflow that can actually be put into practice:
- Define the business scenario first: map out the real prompt types and their distribution, so you are not evaluating for the sake of evaluating.
- Build a private evaluation set: combine public benchmarks with internal real-world samples to balance comparability and relevance.
- Score dimensions independently: rate general capability, coding, and performance separately, so that a single aggregate score cannot hide weak spots.
- Control variables when comparing: keep hardware, concurrency, and prompts identical so the data is trustworthy.
- Watch long-term stability: after a model or inference service upgrade, run periodic regressions to catch capability or performance regressions.
On the deployment side, inference performance is closely tied to the overall architecture. Using a container platform for elastic scaling, monitoring and alerting for real-time observability, and log services for request-level tracing helps a team turn “benchmark scores from the lab” into “a stable experience in production.” With the general-purpose capabilities of cloud platforms such as Baidu AI Cloud — for example elastic compute, monitoring and alerting, and log analysis — this kind of evaluation and tuning workflow can be incorporated into the engineering process more systematically.
Conclusion
The debate over “which model is strongest” will never end, but what actually helps developers is a methodology that is reproducible, transferable, and grounded in the business. Break general capability, coding capability, and inference performance apart, validate them with a unified evaluation set and a rigorous load-testing process, and you can find the model that genuinely fits your scenario — beyond the noise of leaderboards. Evaluation is not about ranking models; it is about providing a basis for engineering decisions. That is what capability evaluation really means.
