LLM Inference Performance Evaluation: Balancing Speed, Cost, and Concurrency

    This article explains why asking which LLM is "the strongest" tends to miss the point, and breaks down the metrics that actually matter for inference performance: throughput (tokens/s), latency (TTFT and TPOT), concurrency, and cost structure. It outlines how to run reproducible load tests, how to match metrics to scenarios such as real-time chat, code completion, offline batch jobs, and multi-turn agents, and which levers to use for tuning and model selection. Readers can use this to set concrete performance targets, build a repeatable evaluation process, and make decisions based on measurable data rather than marketing numbers.

    1. “Which model is the strongest” is a question that needs to be broken down

    In an engineering context, directly asking “which LLM is the strongest” rarely produces a useful answer. “Strong” is a composite concept: generation quality, inference speed, unit cost, concurrency capacity, context length, and stability. No single dimension can fully capture how a model actually performs in a specific business.

    A code-completion tool prioritizes extremely low first-token latency and stable throughput. A batch task for long-document summarization cares more about overall throughput and unit cost. An application serving high-concurrency online traffic is acutely sensitive to concurrency scheduling and tail latency. The right approach, then, is not to hunt for the “strongest model,” but to first clarify how sensitive the business is to each metric, and then build a quantifiable, reproducible evaluation system.

    This article focuses on inference performance—the area that is easiest to quantify and also the easiest to blur with marketing language—and breaks down its key metrics.

    2. The core metrics of inference performance

    1. Throughput: what tokens/s really means

    A token is the basic unit a large model uses to process text. Depending on the language, a single token maps to several characters or a fragment of a word. Measuring generation speed in tokens per second is therefore closer to the underlying mechanism than counting words or characters per second, though for external communication it can be roughly converted to an approximate character count.

    What needs to be understood is that throughput is not a single number. It depends on input length, output length, and batching strategy together. When the input is short and the model is mainly decoding output, speed is usually higher; when the context is long and the model has to process a large prefill computation, overall speed drops noticeably. Quoting a “speed number” detached from the specific input and output lengths has limited meaning.

    2. Latency: TTFT and TPOT

    The metric users actually feel is latency, usually split into two parts:

    • Time To First Token (TTFT): the time from sending the request to receiving the first token. It is governed mainly by the prefill phase and directly affects the sense of “responsiveness” in streaming interactions. Conversational applications are the most sensitive to this metric.
    • Time Per Output Token (TPOT): the average time to generate each subsequent token, corresponding to the steady-state speed of the decoding phase.

    One model may have high TTFT but low TPOT, and vice versa. Only by combining both can you accurately assess the interaction experience.

    3. Concurrency

    In online-serving scenarios, single-request speed is no longer the only goal. The real challenge is whether the system can maintain stable throughput and latency when many requests arrive at once. Concurrency depends on the underlying batching and scheduling, the reuse efficiency of memory and compute resources, and the queuing mechanism. A common phenomenon under high concurrency is a sharp rise in tail latency (P95/P99)—the average looks good, but part of the user experience collapses. When evaluating concurrency, you must look at the latency distribution, not just the average.

    4. Cost structure

    Inference cost is usually billed separately for input tokens and output tokens, and the unit prices for the two often differ. Real cost must be estimated against the business’s input/output ratio: a long-input, short-output scenario and a short-input, long-output scenario can differ enormously in total cost even at the same unit price. A sound practice is to account for cost per “thousand typical business requests” rather than looking only at the per-token price.

    3. How to run a scientific load test

    Marketing-style “speed comparisons” often lack methodology, and their conclusions cannot be reproduced. A reliable evaluation should meet the following conditions:

    1. Fix the input/output length distribution: use request samples close to real business traffic, not random short sentences.
    2. Measure TTFT, TPOT, and overall throughput separately: avoid letting one vague number hide the details.
    3. Observe metric changes under increasing concurrency: record the curves of throughput and latency distribution against concurrency, and find the inflection point.
    4. Repeat multiple times and study the distribution: focus on P50/P95/P99, not a single average.
    5. Control variables: keep the hardware environment, network, and batching parameters consistent.

    The following is a vendor-agnostic example of a general load-testing approach (pseudocode, showing only the measurement dimensions):

    1. import asyncio
    2. import time
    3. import statistics
    4. async def single_request(client, prompt, max_tokens):
    5. start = time.perf_counter()
    6. first_token_time = None
    7. token_times = []
    8. # Stream the response so TTFT and TPOT can be captured separately
    9. async for chunk in client.stream_generate(prompt, max_tokens=max_tokens):
    10. now = time.perf_counter()
    11. if first_token_time is None:
    12. first_token_time = now
    13. ttft = now - start
    14. token_times.append(now)
    15. total_time = token_times[-1] - start
    16. # TPOT: average inter-token interval excluding the first token
    17. tpot = (total_time - ttft) / max(len(token_times) - 1, 1)
    18. return {"ttft": ttft, "tpot": tpot, "total": total_time,
    19. "tokens": len(token_times)}
    20. async def benchmark(client, prompts, max_tokens, concurrency):
    21. semaphore = asyncio.Semaphore(concurrency)
    22. async def wrapped(p):
    23. async with semaphore:
    24. return await single_request(client, p, max_tokens)
    25. start = time.perf_counter()
    26. results = await asyncio.gather(*(wrapped(p) for p in prompts))
    27. elapsed = time.perf_counter() - start
    28. ttfts = [r["ttft"] for r in results]
    29. tpots = [r["tpot"] for r in results]
    30. total_tokens = sum(r["tokens"] for r in results)
    31. return {
    32. "throughput_tokens_per_s": total_tokens / elapsed,
    33. "p50_ttft": statistics.median(ttfts),
    34. "p95_ttft": sorted(ttfts)[int(len(ttfts) * 0.95)],
    35. "avg_tpot": statistics.mean(tpots),
    36. "elapsed": elapsed,
    37. }

    The value of this framework is that it does not presume any “best model”; instead, it hands the judgment back to reproducible measurement.

    4. Matching business scenarios to metric weights

    Discussing metrics detached from a scenario is meaningless. Here are weight references for a few common scenarios:

    • Real-time conversation / intelligent customer service: prioritize TTFT, then watch tail latency under high concurrency. Users can tolerate slightly slower generation, but they struggle to accept “freezing in place.”
    • Code completion / intelligent coding tools: TTFT and TPOT are equally important, and stability requirements are extremely high, because the editing experience is very sensitive to jitter.
    • Offline batch processing (summarization, translation, data processing): prioritize overall throughput and unit cost; latency is not sensitive, and you can fully leverage batching to improve cost-efficiency.
    • Multi-turn agent tasks: require weighing throughput, concurrency, and long-context processing efficiency together, because they are often composed of multiple consecutive inference steps.

    It is recommended to set a clear target threshold for each business dimension (for example, “P99 TTFT must not exceed a certain number of milliseconds”) and treat it as a hard acceptance criterion, rather than vaguely pursuing “fast.”

    5. Selection and tuning practices

    Once the evaluation system and business weights are defined, you can optimize inference performance along the following directions:

    1. Use streaming output sensibly: getting the first token on screen as soon as possible markedly improves subjective experience, even when the total generation time is unchanged.
    2. Control context length: overly long inputs drive up prefill cost; techniques such as retrieval augmentation and context compression can reduce wasted tokens.
    3. Adjust batching and scheduling strategies: find a balance between throughput and latency—online scenarios lean toward low latency, offline scenarios toward high throughput.
    4. Introduce caching: cache results for high-frequency or structured inputs to reduce repeated inference.
    5. Improve observability: use general capabilities such as logging and monitoring/alerting to continuously track TTFT, TPOT, concurrency error rate, and cost, forming a closed optimization loop.
    6. Scale out horizontally at the architecture level: use container platforms for elastic scaling and message queues to smooth out traffic spikes, avoiding instantaneous high concurrency overwhelming the service.

    6. Summary

    The question “which LLM is the strongest” should essentially be reframed as “which model best fits my business metrics and cost constraints.” Real engineering capability lies in whether you can build an objective metric system—throughput, first-token latency, per-token generation speed, concurrency capacity, and cost structure—and validate it with reproducible load-testing methods, rather than being swayed by a single marketing number.

    For developers and businesses, rather than chasing leaderboard rankings, it is more worthwhile to invest effort in defining clear performance targets, building a scientific evaluation process, and continuous online monitoring and tuning. Only then does selection stop being a roll of the dice and become a durable, reusable technical decision-making capability.