LLM Inference Performance Evaluation: Key Metrics and Model Selection Practices
Inference speed is quickly becoming as important as raw capability when teams choose a large language model. This article explains why latency and throughput matter in real deployments, breaks inference speed down into measurable metrics such as time to first token (TTFT), token throughput, peak throughput, and P50/P95/P99 latency, and describes how to evaluate coding capability across scenarios that resemble real development work. It also includes a reusable streaming benchmark script, a scenario-based selection checklist covering real-time chat, IDE code completion, batch generation, multi-step agents, and capability-heavy tasks, and a set of general engineering optimizations for improving the inference experience.
1. Why Inference Performance Is Becoming a Core Selection Criterion
In the past, models were mostly compared by their aggregate scores on public leaderboards. In real engineering work, however, users in scenarios such as real-time conversation, code completion, and multi-step tool calls by an AI agent perceive one thing first: how fast the model feels, rather than the fraction of a point it gains on a benchmark.
Even a top-tier model can hurt the product experience if the first token arrives late or generation stutters noticeably. Building a systematic evaluation framework around inference performance is therefore becoming a necessary part of model selection. The “fast” or “Flash” class models that have emerged in recent years are essentially a product-level response to this demand: through model architecture simplification, inference pipeline optimization, quantization, and operator-level acceleration, they look for a better balance between capability and speed.
2. Three Core Evaluation Dimensions
Comparing models in this category usually requires covering three dimensions, none of which can be skipped.
- General capability: language understanding, knowledge Q&A, multi-turn dialogue, and instruction following. This determines whether the model can serve as the primary model for a wide range of tasks.
- Coding capability: code generation, completion, refactoring, cross-file understanding, and the quality of complex frontend and visualization output. This is one of the capabilities developers care about most.
- Inference speed: the latency and throughput of the entire path from request submission to result delivery. This directly affects the interaction experience and cost per unit of work.
It is worth emphasizing that these three dimensions often involve trade-offs. Chasing extreme speed may cost some capability, and vice versa. The value of evaluation lies in comparing all three in the same frame of reference instead of looking at any single one in isolation.
3. Inference Speed: Breaking Down the Key Metrics
Inference speed is the dimension users notice most easily in fast models, but assessing it objectively requires breaking it into several measurable sub-metrics.
3.1 Time to First Token (TTFT)
Time to first token is the interval between sending the request and receiving the first output token. It determines how long the user feels they are “waiting for things to start”. TTFT is heavily influenced by queuing, cold starts, and input length (the prefill stage), so it should be the primary metric when assessing the interaction experience.
3.2 Throughput (tokens/s)
Throughput describes how many tokens are generated per unit of time, and determines how quickly the output “fills the screen”. Two stages need to be distinguished:
- Thinking/prefill stage: processing the input and building the context. This is usually compute-intensive and relatively slower.
- Answering/decoding stage: generating the result token by token, often at a higher speed.
Many evaluations report the rates for these two stages separately to reflect the model’s real performance at each step.
3.3 Peak Throughput and Median Response Time
- Peak throughput: the instantaneous maximum generation rate under ideal conditions. It represents the ceiling of the model’s capability; for example, certain fast modes can reach a scale of thousands of tokens per second under specific loads.
- Median response time (P50): the median of end-to-end latencies across multiple requests. It represents the experience of a “typical user” better than the average, because it is not distorted by extremely long tail requests.
When measuring in practice, record P50, P95, and P99 latencies together rather than relying on a single average, which can hide long-tail jitter.
4. How to Evaluate Coding Capability
Evaluating coding capability should not rely on the accuracy rate of a single algorithm problem. It should cover multiple scenarios that are closer to real development:
- Code generation and completion: given a comment or a function signature, can the model produce a correct, runnable implementation.
- Refactoring and debugging: can it understand the logic of existing code and make equivalent rewrites or fix defects.
- Complex application output: for example, generating an interactive visualization application (such as a 3D solar system or a dynamically growing tree), which tests how well the model organizes structured output and reproduces details.
- Multi-language and toolchain coverage: how well the model handles different programming languages and frameworks.
Graphical and 3D tasks in particular tend to expose a model’s weaknesses, because they place higher demands on code structure, dependency relationships, and detail consistency. Code that merely “looks right” does not guarantee that it runs correctly.
5. A Reusable Inference Benchmark Script
Credible speed data requires a disciplined measurement methodology. The following is an illustrative benchmark approach that uses a generic interface placeholder and is not tied to any specific vendor:
import timedef benchmark_stream(client, prompt: str, runs: int = 10):latencies = []for _ in range(runs):start = time.perf_counter()first_token_time = Nonetoken_count = 0# Consume the response as a stream and count chunk by chunkfor chunk in client.stream(prompt):if first_token_time is None:first_token_time = time.perf_counter() - start # Time to first tokentoken_count += 1total = time.perf_counter() - startlatencies.append({"ttft": first_token_time, # Time to first token"total": total, # End-to-end latency"tps": token_count / total, # Average throughput (tokens/s)})# Compute median and percentiles so the average cannot hide the long taillatencies.sort(key=lambda x: x["total"])p50 = latencies[len(latencies) // 2]["total"]p95 = latencies[int(len(latencies) * 0.95)]["total"]return {"p50": p50, "p95": p95, "samples": latencies}
The key points of this code are measuring time to first token with a streaming response, recording percentile latencies separately, and normalizing throughput statistics. In engineering practice, it is advisable to test in tiers across input lengths and concurrency levels, because the same model can perform very differently under different loads.
6. A Scenario-Based Selection Checklist
The ultimate purpose of evaluation is selection. Below is a practical decision framework:
- Real-time conversation and voice interaction: prioritize TTFT and median response time, then throughput, with low jitter as a requirement.
- Code completion in an IDE: throughput determines how “responsive” completion feels, while coding accuracy remains a requirement.
- Batch generation and data processing: peak throughput and cost per token matter more, while TTFT is less sensitive.
- Multi-step agent calls: focus on the P95/P99 long tail for stability, so that jitter in a single step does not break the whole chain.
- Complex tasks that demand high capability: capability comes first and higher latency is acceptable; a fast model may not be the best choice here.
7. Engineering-Side Optimization Tips
Regardless of which class of model you choose, the inference experience can be further improved with general engineering practices:
- Set sensible output limits: overly long generations significantly increase tail latency, so truncate as needed.
- Caching and reuse: cache results for frequent, repeated requests to reduce unnecessary calls.
- Concurrency and queue management: introduce message queues and request scheduling to smooth out peaks and control P99.
- Observability: use logging services and monitoring with alerting to continuously collect metrics such as TTFT, TPS, and percentile latencies, establish a baseline, and configure threshold alerts.
- Tiered deployment: combine general cloud product categories such as model capabilities, container platforms, and object storage, and scale elastically on demand.
8. Conclusion
The rise of fast large models reflects a shift in the industry from “capability above all” to “capability and efficiency in equal measure”. For developers and enterprises, rather than chasing a single leaderboard ranking, it is better to build an evaluation method oriented toward business scenarios: measure general capability, coding capability, and inference speed in the same frame of reference, support decisions with disciplined percentile latency and throughput data, and combine this with reasonable engineering optimizations. Only then can you truly select the model that fits your own scenarios. The purpose of evaluation is not to rank models, but to help you find the right balance point among speed, capability, and cost.
