Beyond Leaderboards: Evaluating LLMs on General Capability, Coding, and Inference Speed

    This article explains how to choose an LLM by evaluating three dimensions separately: general capability, coding ability, and inference speed. It covers combining public benchmarks with private, scenario-based tests, stratifying tasks by difficulty, assessing complex application builds, and measuring throughput, time to first token, tail latency, and the thinking and answering phases. It closes with a reusable evaluation workflow tied to business constraints.

    Introduction: Why One Score Is Not Enough

    Evaluation leaderboards for large language models multiply quickly, and different models trade wins in overall rankings. For engineering teams, though, a leaderboard position is only a starting point. What determines a good choice is what the business scenario actually requires. A model that performs well on general Q&A may frequently time out on long-context reasoning. Another model may excel at coding but show high time to first token on complex tasks that require extended thinking.

    A sound evaluation therefore separates three dimensions: general capability, coding ability, and inference speed. It also explains the technical meaning behind each metric and weighs them against business constraints. This article covers these three dimensions and presents a practical evaluation method.

    1. General Capability: Combining Multi-Task Benchmarks with Scenario Testing

    General capability describes how a model performs across a wide range of tasks. It is usually measured with a set of benchmarks covering knowledge, logical reasoning, mathematics, common sense, multilingual ability, and other areas. These benchmarks allow horizontal comparison, but they have two inherent limitations:

    • Data contamination risk: Public test sets may be included in training data, which inflates scores and fails to reflect true generalization.
    • Disconnect from business scenarios: Leaderboard questions are mostly standardized Q&A, while real business workloads often involve long documents, structured output, tool calls, and other complex constraints.

    A more reliable approach is a dual-track evaluation that combines benchmarks with scenarios. Use general benchmarks for initial screening, and build a private test set that matches your own business. The private set should cover high-frequency real requests and include a small sample of human-scored results. These samples calibrate the gap between automatic scoring and actual user experience.

    In scenario testing, group tasks by difficulty. For example, simple information extraction, medium-complexity multi-turn dialogue, and high-complexity reasoning should each have their own pass rates. Stratified results prevent an overall score from hiding weak spots. Many models are not uniformly strong. They often perform well up to a certain difficulty level and then drop off sharply.

    2. Coding Ability: From Code Snippets to Complete Applications

    Coding ability matters most to developers, yet it goes well beyond whether a model can write a single function. It breaks down into three progressive levels:

    1. Code completion and snippet generation: Function-level and line-level completion, checking syntactic correctness and fit with the surrounding context. Evaluation is relatively simple and can be quantified by unit test pass rate.
    2. Algorithmic and logical correctness: For algorithm problems and bug-fixing tasks, check how well the model handles edge cases, exception handling, and time complexity. Run the code to verify results instead of judging by eye.
    3. Complex application construction: Starting from requirements, the model generates a small runnable application, such as an interactive data visualization, a 3D scene, or an animation. These tasks test requirement understanding, module decomposition, multi-file collaboration, and front-end and back-end integration. Models differ most on this level.

    When evaluating complex applications, do not stop at whether code was generated. Assess three aspects: runnability, functional completeness, and visual and interaction quality. Combine automated checks with human review. Automated checks cover whether the project builds successfully and passes smoke tests. Human review covers whether the features match the requirements and whether the interactions run smoothly.

    Models often perform inconsistently across complex application tasks. Some excel at 3D scenes and animation, while others are more reliable on data-processing applications. Coding evaluation should cover a range of task types. A single problem should not decide the overall ranking.

    3. Inference Speed and Performance Metrics: Throughput, Latency, and Phase Breakdown

    Speed metrics directly affect user experience and inference cost, yet they are easy to misread. Start by clarifying several core concepts.

    3.1 Throughput (Tokens Per Second)

    Throughput is the number of tokens generated per unit of time. It is commonly expressed as TPS (Tokens Per Second) and reflects generation efficiency. Throughput depends on several factors, including hardware, batch size, context length, and concurrency. A single throughput figure without its test conditions is nearly meaningless for comparison. Fix the test environment, and state these conditions explicitly in every report.

    3.2 Time to First Token (TTFT)

    Time to First Token (TTFT) is the time from sending a request to receiving the first token. It largely determines how responsive a system feels. In streaming scenarios, TTFT often shapes the perceived experience more than overall throughput. Users would rather see output appear word by word at a slightly slower rate than wait with no response.

    3.3 End-to-End Latency and Percentiles

    An average latency figure can hide the real user experience. In production, look at percentile latency. The median (P50) reflects typical experience, while tail latency (P95, P99) reflects the worst cases users encounter. A service with a short median but very high tail latency produces frequent stalls under high concurrency, which is damaging for interactive applications.

    3.4 Thinking Phase and Answering Phase

    Models with reasoning or chain-of-thought capabilities typically respond in two phases. In the thinking phase, the model reasons internally. In the answering phase, it outputs the final answer. The two phases can have very different throughput and latency. The thinking phase often generates many intermediate tokens. If throughput is low, users wait a long time before the answer begins. Record metrics for each phase separately instead of measuring only the overall response.

    The following pseudocode shows how to measure these metrics. It illustrates the method only:

    1. import time
    2. def benchmark_stream(client, prompt, rounds=20):
    3. ttft_list, tps_list, total_list = [], [], []
    4. for _ in range(rounds):
    5. start = time.perf_counter()
    6. first = None
    7. tokens = 0
    8. for chunk in client.stream(prompt): # stream the response
    9. if first is None:
    10. first = time.perf_counter() # record the time of the first token
    11. tokens += count_tokens(chunk)
    12. end = time.perf_counter()
    13. ttft = first - start # time to first token
    14. decode_time = end - first # duration of the decoding phase
    15. tps = tokens / decode_time if decode_time > 0 else 0
    16. ttft_list.append(ttft)
    17. tps_list.append(tps)
    18. total_list.append(end - start)
    19. return {
    20. "p50_ttft": percentile(ttft_list, 50), # median
    21. "p95_ttft": percentile(ttft_list, 95), # tail latency
    22. "p50_tps": percentile(tps_list, 50),
    23. "p50_total": percentile(total_list, 50),
    24. }

    The values in the table below are illustrative only. They show how each metric should be recorded and do not represent the performance of any specific model. Real values vary with hardware, context length, and concurrency strategy.

    Metric Meaning Illustrative Value
    Peak throughput Maximum TPS under high concurrency with short context 2000+ TPS
    Thinking-phase throughput Tokens generated per second during the reasoning phase ~1000 TPS
    Answering-phase throughput Tokens generated per second during the answer output ~1800 TPS
    Median response latency (P50) End-to-end time for a typical request ~3 s

    4. Building a Reusable Evaluation Workflow

    Combining the dimensions above leads to the following workflow:

    1. Define business constraints: List the key constraints first, such as whether streaming output is required, average context length, concurrency, cost budget, and whether structured output is needed. These constraints determine how the weights are assigned.
    2. Build a private test set: Collect real high-frequency requests, group them by difficulty, and keep human-scored samples for calibration.
    3. Evaluate capability by difficulty level: Calculate stratified results for general and coding capability separately to locate where performance drops off.
    4. Run standardized performance tests: Fix hardware and test parameters, then measure TTFT, throughput, and P50 and P95 latency. Report the thinking and answering phases separately.
    5. Score with weights and make trade-offs: Combine scores according to business priorities, and focus on weak points rather than averages. Interactive scenarios should weight latency more heavily. Batch scenarios should prioritize throughput and cost.
    6. Run continuous regression tests: Models keep iterating. Version the evaluation sets and benchmark scripts, and rerun the regression suite on every upgrade to catch new versions that perform worse.

    5. Selection Guidance by Use Case

    • Interactive applications (such as intelligent assistants and code completion): Prioritize low TTFT and stable tail latency. Perceived speed matters more than peak throughput.
    • Batch and offline tasks (such as document processing and bulk generation): Focus on throughput and cost per token. Higher latency is acceptable.
    • Heavy coding workloads: Choose models with strong complex application construction and consistent results across diverse task types. Use “runnable with complete functionality” as the acceptance standard.
    • Tasks that require deep reasoning: Focus on throughput and overall latency during the thinking phase, so that slow reasoning does not degrade the experience.

    Conclusion

    Evaluating LLMs should be a systematic process guided by business constraints, not a simple list of leaderboard scores. General capability shows what a model can do. Coding ability shows whether its output can become a working application. Inference speed shows whether users will actually use it. Combining all three, supported by stratified testing, percentile latency analysis, and continuous regression, leads to selection decisions that hold up in production as models keep changing.