High-Throughput LLM Inference: How to Get Both Speed and Quality
An in-depth look at what drives LLM inference speed and why high throughput and model quality are not inherently at odds. The article breaks down the key metrics (time to first token, output tokens per second, and end-to-end latency), explains the engineering levers behind fast inference such as MoE, continuous batching, PagedAttention, quantization, and speculative decoding, and outlines a reproducible approach to benchmarking both performance and output quality. It closes with practical engineering advice for adopting a high-speed inference service.
Why Output Speed Became a New Experience Metric
In the past, evaluating a large model came down to one question: was the answer correct? Today, in scenarios like intelligent coding, real-time customer support, and document generation, how fast the answer arrives matters just as much to the user experience. Whether an inference service is “fast” can be broken down into at least three metrics:
- Time To First Token (TTFT): The time from sending a request to receiving the first token. It is mainly affected by queue wait time, input prefill (Prefill), and first-packet network overhead.
- Output speed (tokens per second): Once the output phase begins, this reflects how fast the model generates content and captures the efficiency of a single forward pass.
- End-to-end latency: The total time to generate a complete answer, roughly equal to “TTFT + output length ÷ output speed.”
Only when you combine all three can you fully characterize the interactive experience. With streaming output, users no longer stare at a blank screen; they watch text appear word by word, and perceived latency drops sharply. But when output speed clearly outpaces reading speed, the wait shrinks from tens of seconds to just a few seconds. The old back-and-forth of queuing and refreshing to get lucky has been reshaped into a conversational rhythm where users can ask follow-up questions continuously, interrupt at any time, and iterate quickly.
“Faster Must Mean Dumber” Is an Intuition Worth Unpacking
The idea that you can’t be both fast and smart seems contradictory, and its root is equating “model size” with “compute per single pass.” In reality, output speed depends on how long a single forward pass takes, while model capability depends on training data, parameter count, and training methods. The two are not a simple trade-off. The factors that affect output speed can be layered as follows:
- Model architecture: In sparse architectures like Mixture of Experts (MoE), each token activates only a subset of expert networks. The total parameter count can be very large, yet the compute per token stays relatively manageable, creating a structural possibility for “flagship capability plus high-speed output.”
- Inference engine: Continuous batching lets new requests join a running batch immediately after their first token is generated, avoiding whole-batch waiting; techniques like PagedAttention significantly reduce memory waste.
- Parallelism and quantization strategies: Tensor parallelism and pipeline parallelism split the model across multiple accelerator cards to compute collaboratively; weight quantization and KV Cache quantization reduce memory access and bandwidth pressure while keeping accuracy loss under control.
- Accelerated decoding techniques: Speculative decoding first uses a lightweight draft model to guess several tokens in a row, then the target model verifies them in a single pass. With a high acceptance rate, this can noticeably boost output speed without changing the target model’s output distribution.
- Server-side scheduling: Reasonable request queuing, context length limits, and elastic instance scaling determine whether latency stays stable under high concurrency.
It’s clear that output speed is an “engineering problem” while model capability is a “training problem.” The intuition that fixates on parameter count happens to miss the engineering layer that truly makes the difference. Of course, pursuing speed has a cost: aggressive quantization, an overly short maximum output length, or overly restrictive context limits can indeed hurt quality. That is why “high speed” should come from architecture and inference optimization, not from “swapping in a smaller model.”
Benchmarking: How to Measure It Right
To verify whether “high speed comes at the cost of quality,” you need a reproducible evaluation process.
Performance
- Test matrix: Cover short input / long input, short output / long output, and gradients of single concurrency, medium concurrency (e.g., 16–32), and high concurrency (e.g., 64+).
- Key statistics: Look at both TTFT and output speed in terms of median and P95/P99 percentiles. Averages easily mask long tails, and the long tail is precisely the source of degraded experience during peak times.
- Watch the methodology: If you count only the number of streamed data chunks, network fragmentation will introduce bias. Rely instead on the usage information returned by the API, or on re-encoding the complete output with the tokenizer.
The example code below demonstrates how to run asynchronous concurrent load tests against an inference service:
import asyncio, json, os, timeimport aiohttpBASE_URL = os.environ["LLM_BASE_URL"] # Inference service endpointAPI_KEY = os.environ["LLM_API_KEY"]MODEL = os.environ["LLM_MODEL"]PROMPT = "Please implement a local cache with an expiration time in Python, and explain the key edge cases."async def one_call(session: aiohttp.ClientSession) -> dict:payload = {"model": MODEL,"messages": [{"role": "user", "content": PROMPT}],"stream": True,"stream_options": {"include_usage": True}, # Final chunk returns exact usage}headers = {"Authorization": f"Bearer {API_KEY}"}t0 = time.perf_counter()ttft, tokens = None, 0async with session.post(f"{BASE_URL}/chat/completions",json=payload, headers=headers) as resp:resp.raise_for_status()async for raw in resp.content:line = raw.decode("utf-8").strip()if not line.startswith("data:"):continuebody = line[len("data:"):].strip()if body == "[DONE]":breakchunk = json.loads(body)if chunk.get("usage"):tokens = chunk["usage"].get("completion_tokens", tokens)continuedelta = chunk["choices"][0]["delta"].get("content") or ""if delta and ttft is None:ttft = (time.perf_counter() - t0) * 1000elapsed = time.perf_counter() - t0return {"ttft_ms": ttft, "tokens": tokens,"tps": tokens / elapsed if elapsed > 0 else 0}async def benchmark(concurrency: int, rounds: int = 3):results = []async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=300)) as s:for _ in range(rounds):results += await asyncio.gather(*[one_call(s) for _ in range(concurrency)])for key in ("ttft_ms", "tps"):vals = sorted(r[key] for r in results if r[key] is not None)p95 = vals[int(len(vals) * 0.95) - 1] if vals else Noneprint(f"{key}: p50={vals[len(vals)//2]:.1f}, p95={p95}")
Quality
Meeting performance targets is only the first step; you also need a fixed “regression test set” to validate quality. Recommended coverage includes:
- Code generation: Whether it can produce runnable code in one pass and pass unit tests;
- Instruction following: Whether format constraints (e.g., strict JSON), length limits, and role settings are respected;
- Long context: Whether it can accurately retrieve and locate key information from long documents or large codebases;
- Multi-turn consistency: Whether it stays consistent with requirements and style across multiple conversation turns;
- Edge cases and error handling: Whether it adequately considers illegal input, null values, concurrent conflicts, and similar scenarios.
The Felt Difference: A Coding Assistant Example
After integrating a high-speed inference service into an intelligent coding assistant, the most direct change is that the “interaction rhythm” is restructured:
- From “state the requirement → wait → review” to “watch and adjust as it goes”: The generation process runs in sync with the reading process, so users can spot a problem mid-output, interrupt, and add constraints in time, greatly reducing the sunk cost of “waited a minute, only to find the direction was wrong.”
- High-frequency, small-step iteration becomes the norm: Paste in an error message and a fix suggestion appears almost instantly; the test → fix → re-test loop can quickly run through many rounds. The speed advantage keeps compounding across multi-turn tasks.
- Agent-style long tasks see a clear improvement in experience: When a coding assistant needs to call tools in sequence to complete a “read files → edit code → run tests” loop, the latency of each step adds up; a faster single step means the whole task chain finishes sooner.
- Scenarios migrate naturally: High-frequency tasks like code completion, natural language to SQL, log troubleshooting, and test case generation can all benefit from high-speed output.
Engineering Considerations When Adopting It
- Pair it with streaming: When using streaming output, the frontend must correctly handle incremental concatenation and rendering, and the backend should have solid timeout, retry, and interrupted-error handling.
- Use engineering means to speed things up further: Combine prompt caching to reuse stable system prompts and context, and use a rules engine or a small model to handle lightweight tasks like simple classification and formatting, reserving the high-speed flagship model for complex reasoning.
- Don’t lower the quality bar: Faster output doesn’t mean you can skip review and testing. Generated code must go through static checks and unit tests; when it comes to production changes, follow existing practices like small, incremental commits, canary releases, and fast rollback.
- Keep monitoring and cost in sync: Log metrics such as TTFT, output speed, and error rate through a logging service, and connect them to monitoring and alerting; the cost of a high-speed service should be assessed against actual throughput and billing rules, not unit price alone.
- Security and compliance: API keys should be injected through key management or environment variables, not hardcoded into the repository; input and output data should follow your organization’s data security standards.
For enterprises that want to build or select an inference service, cloud platforms such as Baidu AI Cloud typically offer model services covering a range of parameter sizes along with supporting toolchains, making it easier to make comprehensive performance and quality trade-offs by scenario. For specific capabilities and specifications, refer to the official documentation.
Conclusion
Being “absurdly fast without losing intelligence” is possible because output speed and model capability are determined by engineering optimization and training methods, respectively. For developers, rather than choosing between “fast” and “good,” it’s better to build an evaluation system that includes TTFT, output speed, long-tail percentiles, and a quality regression set—let the data speak, then choose the service configuration that best fits your own scenario.
