The Complete Guide to LLM Evaluation: Assessing Capability, Coding Skills, and Inference Speed
Choosing the right large language model requires a structured evaluation. This guide outlines a three-dimension methodology: general capability (knowledge accuracy and hallucination, instruction following, multi-turn consistency, safety), coding ability (function correctness, engineering completeness, complex application generation, debugging), and inference speed (TTFT, decode throughput, median response time, test conditions). It includes weighted-scoring and sandbox-based code-testing code examples, a measurement script, and the four stages of a reproducible evaluation pipeline.
As the number of large language models (LLMs) grows rapidly, the first practical question for developers is no longer whether a model is available, but which one to choose. With so many candidates, vendor claims and a single leaderboard are rarely enough for a reliable decision. A useful evaluation must answer three questions at once: Is the model capable? Can it write good code? Is it fast enough? This article walks through a practical evaluation methodology around these three dimensions.
Start with a Framework: Why Single Metrics Mislead
In practice, many teams rank models by a single “overall score,” but this is a dangerous oversimplification. A model may excel at knowledge question answering yet frequently fail at structured code generation. It may also deliver excellent quality but be too slow to support a high-concurrency online service.
A sound evaluation, therefore, is a multi-dimensional weighted assessment rather than a single-point comparison. A practical approach is to define the weights for your business use case first, then score each dimension separately:
# Example evaluation weights — adjust them to your use caseweights = {"general_capability": 0.3,"code_generation": 0.4,"inference_throughput": 0.3,}def overall_score(item_scores):return sum(item_scores[k] * weights[k] for k in weights)
For an online coding assistant, the code generation weight should be higher. For content moderation or summarization, inference throughput and response latency are the central concerns. Defining the weights before comparing scores is the first step toward avoiding leaderboard-driven decisions.
Dimension 1: General Capability
General capability is the hardest dimension to quantify because it covers several sub-areas: knowledge breadth, logical reasoning, instruction following, and multi-turn conversational consistency. The evaluation should not rely on automated scoring alone. Instead, combine automated metrics with human review.
Key aspects to cover include:
- Knowledge breadth and accuracy: Check correctness on factual Q&A, and watch for hallucination, meaning answers that sound authoritative but are fabricated.
- Instruction following: Assess whether the model reliably satisfies complex constraints such as output format, length, and role settings. In real-world applications, this often matters more than sheer knowledge volume.
- Multi-turn consistency: In ongoing conversations, check whether the model keeps context coherent and avoids contradictions between turns.
- Safety boundaries: Evaluate how the model refuses or redirects leading or out-of-scope requests.
Build a test set that covers typical business scenarios, with dozens of samples for each category. Use a scoring matrix to record how different models perform.
Dimension 2: Coding Capability
Coding ability carries high weight in enterprise model selection today. It is also the dimension where models differ most. Checking whether a model can generate runnable code is not enough; the evaluation needs to cover several levels:
- Function-level correctness: Given a natural language description, the model should produce a function that is logically correct and runnable. Run test cases to verify that it actually meets the requirements.
- Engineering-level completeness: The model should generate clear, well-structured code with error handling and edge-case checks, not just toy-level snippets.
- Complex application generation: For example, generate an application with interactive features and visualization. Tasks like these test the model’s grasp of the overall architecture. The quality of a generated 3D visualization or interactive page directly reflects the model’s ability to turn requirements into a finished product.
- Debugging and repair: Given buggy code, the model should accurately locate and fix the problem.
A practical approach to automated evaluation is “generate and verify”: have the model produce code, then automatically run it against test cases.
def evaluate_code_model(model, problem, test_cases):"""Generate code and verify its functional correctness with test cases"""generated = model.generate(problem.prompt)code = extract_code_block(generated) # extract the code blockpassed = 0for case in test_cases:try:result = execute_safely(code, case.input) # run in a sandboxif result == case.expected:passed += 1except Exception:continuereturn {"pass_rate": passed / len(test_cases),"sample_code": code,}
Note that code execution must take place in an isolated sandbox so that untrusted code cannot put the system at risk. When comparing results across evaluations, emphasize functional pass rate rather than whether the code merely looks right.
Dimension 3: Inference Speed
At equal quality, inference speed often determines whether a model can move into production. Many models look impressive in demos but are too slow to handle high-concurrency production traffic. Performance therefore needs to be evaluated independently and quantitatively.
3.1 Key Performance Metrics
Inference speed cannot be captured by a vague “tokens per second” figure. Instead, break it down into metrics for separate stages:
- Time to First Token (TTFT): The time from sending the request to receiving the first token, which corresponds to the model’s prefill stage. TTFT directly drives the user’s sense of waiting: the sooner the first token appears, the smoother the interaction feels.
- Decode throughput: The token generation rate once the response is underway, corresponding to the decode stage. It determines how quickly content flows during long-form generation.
- Median response time: The overall wait for a typical request. It represents the real user experience better than the average because it is not skewed by a small number of extremely slow requests.
For models that use a two-phase “think, then answer” mode, measure the thinking and answering phases separately. Their speeds often differ significantly.
3.2 Measuring Staged Performance
A mature inference framework typically offers two modes. The first is a regular mode that prioritizes stability and cost. The second is a “fast mode” that trades lower precision or parallel scheduling for higher throughput. In fast mode, peak throughput can be several times that of the regular mode, making it well suited to latency-sensitive interactive scenarios. The regular mode, in turn, balances quality against resource consumption.
When evaluating, clearly state the test conditions, including input length, output length, concurrency, and whether batching is enabled. Otherwise, the speed numbers are not comparable. The same model can show vastly different throughput under different batching strategies.
3.3 Measurement Example
Below is a general framework for measuring the key performance metrics:
import timedef measure_inference(model, prompt, max_tokens=512):"""Measure core inference metrics: time to first token and decode throughput"""start = time.perf_counter()result = model.generate(prompt, max_tokens=max_tokens)end = time.perf_counter()first_token_time = result.first_token_time # the framework must return a first-token timestampttft_ms = (first_token_time - start) * 1000decode_time = end - first_token_timeoutput_tokens = len(result.tokens)return {"ttft_ms": round(ttft_ms, 1), # time to first token"decode_tps": round(output_tokens / decode_time, 1), # decode throughput"total_latency_s": round(end - start, 2), # total latency}
In real-world scenarios, run multiple requests and use the median rather than a single result. This filters out noise from cold starts and network jitter.
Building a Reproducible Evaluation Pipeline
Bringing these dimensions together, a complete evaluation pipeline has four stages:
- Test set management: Store test samples and expected outputs in object storage so versions can be traced.
- Batch execution: Schedule evaluation tasks in parallel through a container platform, isolating the runtime environment of each model.
- Data collection: Write metrics such as latency, throughput, and accuracy to a logging service to support side-by-side comparisons.
- Monitoring and alerting: Continuously monitor the response latency and error rate of production models, and trigger alerts when thresholds are exceeded.
You can build this pipeline in the cloud and draw on general cloud infrastructure capabilities for elasticity and observability. Use a container platform to host evaluation tasks, object storage to manage test data, and logging and monitoring services to provide end-to-end visibility. Evaluation then becomes an ongoing, repeatable engineering capability rather than a one-off run.
Conclusion
Choosing a large language model is essentially about finding the right balance among quality, coding ability, and inference performance for your use case. General capability answers whether a model is good enough. Coding ability answers whether it can get real work done. Inference speed answers whether it can hold up under production traffic. All three are essential, and they must be compared quantitatively within a unified, reproducible evaluation framework.
For developers, the real skill is not memorizing where a model ranks, but mastering the methodology of breaking down metrics, designing tests, and comparing results quantitatively. When business needs change, this methodology lets you reassess quickly and make rational model choices instead of following a single leaderboard.
