LLM Code Generation Speed: Throughput Benchmarking and Workload Selection
This article provides a practical framework for evaluating fast-output LLM variants for code generation. It explains which tasks benefit from higher throughput, how to measure TTFT and output tokens per second, how to run concurrent stress tests, and how to account for cost, data privacy, and infrastructure. It also covers common mistakes and a task-selection process for deciding which workloads should use high-speed models and which should use default models with fallback strategies.
Why Speed Deserves Separate Evaluation
For a long time, developers have assumed that “faster model equals smaller model.” Choosing a faster model often meant accepting weaker code-generation capability. Over the past year, some providers have introduced high-speed output versions that claim to preserve flagship-model benchmark scores while increasing output speed to about 400 tokens/s and reducing time to first token (TTFT) to under one second.
These claims are worth attention, but they should not be accepted at face value. Many public numbers come from vendors or individual single-run tests, with different input lengths, sampling parameters, concurrency levels, and hardware conditions. The results are difficult to compare directly. Teams need to test with their own real tasks and then determine whether the speed improvement translates into actual workflow efficiency.
How Faster Output Changes Workflows
The value of speed depends on whether waiting interrupts work. In Usability Engineering (1993), Jakob Nielsen summarized classic response-time thresholds: within 0.1 seconds, users perceive the system as responding instantly; around 1 second, their train of thought is generally not interrupted; beyond 10 seconds, users usually switch their attention. These thresholds come from early human-computer interaction research and were not specifically developed for large language models, but they provide a useful starting point.
By this standard, several task types differ clearly:
| Task type | Typical output size | Problem at slow speed | Possible benefit at high speed |
|---|---|---|---|
| Single-file function completion | 100 to 300 tokens | Waiting interrupts the coding rhythm | Nearly instantaneous; write and review as you go |
| Full-page front-end prototype | 3,000 to 8,000 tokens | Staring at token-by-token output for a long time | A previewable result in a short period |
| Batch refactoring or test generation | Multiple files, tens of thousands of tokens in total | Long total duration; difficult to parallelize | Parallel jobs complete faster |
| Complex algorithm design | Depends on reasoning length | Reasoning time dominates | Limited benefit; prioritize quality |
Take a full-page front-end prototype as an example. Generating 4,000 tokens takes about 80 seconds at 50 tokens/s and about 10 seconds at 400 tokens/s; both figures still require adding TTFT. This is a direct conversion based on speed. Actual elapsed time is also affected by network conditions, queueing, and code length.
The last row is worth noting. If the bottleneck is model thinking rather than output, simply increasing tokens/s provides limited help.
How to Measure: Separate Two Metrics
During testing, record two metrics separately:
- TTFT (time to first token): The time from sending the request to receiving the first token. It is affected by queueing, network conditions, and input processing (prefill). Longer inputs usually make it more noticeable.
- Output throughput (tokens/s): The number of tokens generated per second during the generation phase. Calculate it as total output tokens divided by (end time minus first-token time).
Looking only at averages can hide problems. Teams should record P50 and P95 and run each task type at least 30 times. Use the same prompt set, the same max_tokens, and the same temperature during testing. Take token counts from the usage field returned by the API rather than estimating them from character counts.
Speed is only half of the picture. For the same batch of tasks, also record code-quality metrics such as compile pass rate, unit-test pass rate, and human ratings. A model that is fast but frequently wrong simply shifts time into rework.
Running Concurrent Multi-Instance Stress Tests
Concurrent stress testing is used to observe how a service performs under sustained load. A common approach is to use a terminal multiplexer such as tmux and divide it into multiple panes, with each pane running an independent test process. The pseudocode below illustrates the flow; it is not a directly runnable script:
# Pseudocode: illustrates the stress-test flow; replace with actual SDK and endpointstasks = load("tasks.jsonl") # prompts and inputs fixed for each taskconcurrency_levels = [1, 2, 4, 8] # increase step by stepfor each concurrency_level:for each task, run concurrently:record start timecall API in streaming mode(model, prompt, max_tokens, temperature)record first-token time, end time, and output token count from usageon timeout or 429/5xx, retry with exponential backoff, max 3 timeswrite result(task ID, concurrency level, TTFT, tokens/s, status, error code)aggregate: calculate P50, P95, error rate, and rate-limit count at each concurrency level
In concurrent stress testing, failure handling requires more upfront design than speed measurement:
- Rate limiting (429): Use exponential backoff and record the number of retries. Frequent rate limiting means concurrency has exceeded the quota; the measured speed does not represent model capability.
- Timeouts: Set explicit timeouts for requests and distinguish between connection timeouts and generation timeouts.
- Stream interruption: Save the partial output already received for troubleshooting; do not discard it.
- Excessively long context: Check input length in advance to avoid distorted results caused by truncation.
- Result validation: Compile or test the generated code, and archive failed samples separately.
Cost, Data, and Infrastructure
Concurrent testing costs far more than single calls. The cost of a single task can be estimated with this formula: input token count multiplied by the input unit price, plus output token count multiplied by the output unit price. Before rollout, estimate total cost with a small sample and set budget alerts.
Do not place production code, secrets, or customer data in test prompts. Before use, confirm the provider’s data-retention and training policies, and redact data where necessary.
If test machines are deployed in the cloud, instance charges accumulate over time. Release resources promptly after testing. In addition, the network distance between the test machine and the API endpoint directly affects TTFT; choosing a geographically nearby region helps produce stable results.
Common Mistakes
- Looking only at average latency and ignoring long-tail requests.
- Using character counts instead of token counts to calculate throughput.
- Treating a single test result as a stable metric.
- Exceeding the account quota, allowing rate limiting to interfere with results, and mistakenly concluding that the model itself has become slower.
- Comparing speed only without checking the correctness of generated code.
Selection Process
- List the five most common types of development tasks for the team.
- Select 10 to 30 real tasks for each type and fix the inputs.
- Measure both speed and code quality.
- Calculate the unit cost for each task type and estimate the value of waiting time saved.
- Decide which tasks should use a high-speed model and which should use the default model, and define a fallback strategy for failures.
Conclusion
High-throughput output has clear value for interactive tasks, especially front-end prototyping, visual debugging with view-and-edit workflows, and batch parallel generation. It cannot automatically solve complex reasoning problems, nor can it replace code review. Before purchasing or migrating, teams should re-measure with their own tasks, their own concurrency levels, and their own quality standards, then base their investment decisions on the results.
