LLM Evaluation: Assessing General Capability, Coding, and Inference Performance

    This article explains how to evaluate large language models across three key dimensions: general capability, coding proficiency, and inference performance. It covers benchmark strategies, scenario-based testing, important metrics such as TTFT and throughput, and a reusable workflow for comparing models under consistent conditions.

    Why a Systematic Evaluation Framework Matters

    Over the past two years, the large language model market has entered a period of rapid iteration. New versions and specifications continue to appear. For developers and enterprise technical teams, the main challenge in model selection is often not a lack of options, but deciding which model is truly suitable for a specific use case.

    This is where evaluation matters. A rigorous evaluation framework helps answer three key questions: Is the model capable enough? Is it effective in real tasks? Is it affordable enough to run in terms of inference cost and latency? These questions correspond to three core dimensions: general capability, coding ability, and inference performance. Each dimension is discussed below.

    Dimension 1: General Capability — Is the Model “Capable Enough”?

    General capability reflects a model’s foundational competence as a general-purpose assistant. It typically covers language understanding, logical reasoning, knowledge Q&A, multi-turn dialogue, instruction following, and other related abilities.

    In practice, general capability is often evaluated by comparing a model with leading models in the same tier under the same benchmarks. The result is usually a relative positioning rather than an absolute score. For example, a model may perform between two mainstream flagship models and enter the first tier. This kind of conclusion is useful because it indicates that the model is sufficient for most general tasks, while it may still make trade-offs in certain subcategories.

    Common approaches for evaluating general capability include:

    • Standardized benchmarks: Use public general-purpose evaluation sets that cover reasoning, common sense, mathematics, reading comprehension, and similar task types.
    • Human preference evaluation: Have evaluators blind-score outputs from multiple models, which is closer to real user experience.
    • Scenario-based task sets: Construct representative questions based on your own business scenarios to test model stability in specific domains.

    It is important to note that no single leaderboard is universally reliable. A high ranking does not necessarily mean strong performance on your business data. Therefore, running a small-scale test tailored to your own scenarios is always an essential step.

    Dimension 2: Coding Ability — Can the Model Handle Real Engineering Tasks?

    Coding ability is critical for using large language models in development scenarios. Unlike general Q&A, coding tasks place higher demands on logical rigor, context consistency, and the ability to handle complex structures.

    A mature coding evaluation system usually includes two levels:

    1. Single-item benchmark evaluation: This includes function-level code generation, algorithm problem solving, code completion accuracy, and similar tasks. It is used to quickly measure basic coding competence. Notably, even if a model’s overall coding score falls between two leading models, it may outperform them on specific items. This shows that evaluation should look at structure, not just total scores.

    2. Complex application generation: This better reflects real capability. Asking a model to generate a small runnable application from scratch, such as an interactive 3D visualization program, exposes differences in architecture design, multi-file organization, and implementation correctness more clearly than a few algorithm problems. In such tasks, models can diverge significantly. Some models may produce a well-structured and smoothly running 3D solar system demo, while others may match the strongest models in specific graphical scenarios, such as recursively generated tree structures.

    For developers, the key takeaway from coding evaluation is this: do not rely on a single aggregate score. Construct realistic tasks based on your technical stack, such as frontend frameworks, backend languages, and domain-specific libraries, and observe how the model performs across the full workflow from requirements to runnable code.

    Dimension 3: Inference Performance — Can the Model Be Deployed Effectively?

    Even a strong model is difficult to put into production if latency is too high or throughput is too low. Inference performance evaluation focuses on speed and cost. Three core metrics are especially important:

    • Throughput (tokens/s): The number of tokens a model can generate per unit of time. This is a direct measure of generation speed. Higher peak throughput usually means a smoother experience for long-text output.
    • Time to First Token (TTFT): The time from sending a request to receiving the first token. It strongly affects the user’s perception of waiting and is especially important for streaming interactions.
    • Median response time: The median latency across multiple requests. It better reflects the typical experience than the average and is less affected by a few outlier requests.

    It is also worth noting that models with different operating modes, such as a thinking mode and a fast mode, can show significant performance differences. For example, in fast mode, peak throughput may reach several thousand tokens per second, and generation rates during reasoning and answering phases may differ. Median response time can also drop to a few seconds. This kind of mode design allows users to balance quality and speed flexibly: switch to deep thinking mode for complex reasoning, or enable fast mode for immediate feedback.

    The following example shows a general approach for benchmarking inference performance. It can be used to compare different models under real conditions:

    1. import time
    2. import statistics
    3. def benchmark_inference(stream_client, prompt, runs=20):
    4. """General inference performance benchmark: measure TTFT, median latency, and throughput."""
    5. ttft_list, total_list, token_list = [], [], []
    6. for _ in range(runs):
    7. start = time.perf_counter()
    8. first_token_at = None
    9. tokens = 0
    10. for _ in stream_client.stream(prompt): # Stream generation output
    11. if first_token_at is None:
    12. first_token_at = time.perf_counter() # Record time to first token
    13. tokens += 1
    14. end = time.perf_counter()
    15. ttft_list.append(first_token_at - start) # Time to first token
    16. total_list.append(end - start) # Total response time
    17. token_list.append(tokens)
    18. avg_ttft = statistics.mean(ttft_list) # Average TTFT
    19. median_latency = statistics.median(total_list) # Median response time
    20. throughput = statistics.mean(
    21. [t / d for t, d in zip(token_list, total_list)] # Average throughput in tokens/s
    22. )
    23. return {
    24. "Average TTFT (s)": round(avg_ttft, 3),
    25. "Median response time (s)": round(median_latency, 3),
    26. "Average throughput (tokens/s)": round(throughput, 1),
    27. }

    Inference performance is influenced by multiple factors, including model parameter count, quantization strategy, batch scheduling, and the optimization level of the underlying compute and inference framework. Therefore, when evaluating models for production, it is not enough to rely on theoretical peak values. Measurements should be taken under concurrency pressure close to production traffic to provide a credible basis for capacity planning.

    Building a Reusable Evaluation Workflow

    When the three dimensions are connected, a practical evaluation workflow can be summarized as follows:

    1. Define the scenario: First, clarify the core business requirement. Is it general Q&A, coding assistance, or high-concurrency real-time interaction? Use this to decide the weight of each dimension.
    2. Select benchmarks: For general capability, combine public general-purpose benchmarks with scenario-based task sets. For coding ability, combine single-item benchmarks with complex application generation. For inference performance, use a custom benchmark script.
    3. Compare under consistent conditions: Place candidate models in the same environment and keep inputs, hardware, and concurrency conditions consistent to avoid misleading comparisons.
    4. Make a balanced decision: Compare capability scores with performance and cost in the same framework, and choose the option that best fits your constraints.

    Cloud platforms can improve efficiency during evaluation and deployment. For example, a unified model service can be used to connect multiple candidate models for comparison. Monitoring and alerting, along with log services, can continuously track online latency and throughput. Elastic container platforms can support benchmark tasks under different concurrency levels. Cloud platforms such as Baidu AI Cloud typically provide such general AI infrastructure capabilities, helping teams focus more on evaluation itself rather than environment setup.

    Conclusion

    Scientific evaluation of large language models is fundamentally about finding the balance between capability, coding, and performance that best fits your business. There is no absolute “strongest model”; there is only the model most suitable for the current scenario. By mastering the methodology across these three dimensions and building a reusable, quantifiable evaluation process, teams can maintain clear technical judgment in a rapidly evolving model ecosystem and make selection decisions that hold up under production validation.