Open-Source LLM Benchmarking: Inference Speed vs. Coding Ability
As open-source LLMs evolve rapidly, leaderboard scores alone rarely predict real-world performance. This guide shows how to build a reproducible, business-relevant evaluation covering three dimensions—capability, speed, and cost. You'll learn to define measurement criteria, break "fast" into measurable metrics (throughput, TTFT, phased timing, and P50/P90/P99 percentiles), run a reproducible streaming speed-test script, evaluate coding ability through real tasks instead of a single score, choose models by scenario, and control deployment cost. It also includes a common-mistakes checklist to help you avoid pitfalls that make results incomparable.
Open-Source LLM Benchmarking: Inference Speed vs. Coding Ability
1. What an Open-Source LLM Evaluation Should Solve
Open-source LLMs evolve quickly, with leaderboards, benchmark scores, and vendor claims appearing constantly. Many teams fall into a common trap: treating leaderboard scores as the sole basis for model selection. In real deployments, three things usually matter most—whether the model can actually complete the task, whether responses are fast enough, and whether the deployment cost is acceptable.
For this reason, a sound evaluation cannot rely on a single “overall capability” number. It should break the question down across three dimensions: capability, speed, and cost. Especially when developers face questions like “how do I choose an open-source LLM,” “how do I measure inference speed,” and “what are the common mistakes in coding benchmarks,” they need a standard that is reproducible, comparable, and close to their actual workloads.
2. Define the Measurement Criteria Before Testing
Many evaluation results are not comparable—not because the models differ, but because the test conditions differ. Before running a formal test, fix the following variables first:
- Whether you are running single-request tests or concurrent load tests.
- Whether the input length is fixed (for example, 512, 2048, or 8192 tokens).
- Whether the output length is fixed (for example,
max_tokensset to 512 or 1024). - Whether decoding parameters are consistent, including
temperature,top-p,top-k, and repetition penalty. - Whether warm-up requests are used to avoid cold-start skewing the data.
- Whether you record the hardware, inference framework, quantization method, context length, and version number.
- Whether you distinguish thinking tokens from user-visible tokens.
- Whether you track first-packet latency, first thinking token, first visible token, total time, and percentiles.
Without a unified set of criteria, data across different models, versions, and test batches simply cannot be compared.
3. Inference Speed: Breaking “Fast” into Measurable Metrics
3.1 Throughput: Don’t Look at the Average Alone
Throughput usually refers to the number of tokens generated per unit of time, commonly expressed in tokens/s. When evaluating, you should distinguish peak throughput from average throughput. Peak throughput may only appear early in generation; as the context grows longer and decoding complexity increases, speed can drop. It is therefore recommended to report average throughput, stable generation throughput, and thinking-phase throughput separately.
For reasoning models, a single response may contain two phases: an internal thinking phase and a final answering phase. Tokens generated during the thinking phase are invisible to the user but still consume time; the tokens in the answering phase are what the user actually sees. Measuring thinking throughput and answering throughput separately is therefore more valuable than reporting a single overall average.
3.2 Time to First Token: From Request to the First User-Visible Token
TTFT (Time To First Token) should be defined as the time from sending the request to receiving the first user-visible token—not the time at which the first SSE packet arrives. In many streaming APIs, the first packet may be nothing more than a role placeholder, empty content, or metadata, and does not represent content the user actually sees.
If the model has a thinking phase, a single TTFT number is often not enough. It is better to split it into three metrics:
- First-packet latency: the time to receive the first streaming packet.
- First thinking token time: the time to receive the first thinking content.
- First visible token time: the time to receive the first user-visible content.
For interactive scenarios such as real-time chat and code completion, the latency of the first visible token usually best represents the user’s perceived wait.
3.3 Look at the Thinking Phase and the Answering Phase Separately
Reasoning models may generate a long thinking process before the formal answer. The thinking phase may improve the quality of the final answer, but it can also noticeably slow the user’s perceived experience. When evaluating, you should observe:
- Whether the thinking phase produces a large number of invisible tokens.
- Whether the answering phase can maintain fast throughput once the thinking phase ends.
- Whether the user waits a long time before seeing the first visible content.
- Whether the thinking brings verifiable quality gains.
Models vary widely in their thinking behavior, hardware environment, inference framework, and sampling parameters. For this reason, it is not advisable to cite unconditional performance numbers directly; your own measurements should be the source of truth.
3.4 Percentiles: Don’t Be Misled by the Average
The average is easily skewed by extreme values. When measuring speed, report at least the median (P50), P90, and P99. P50 reflects the typical experience, P90 reflects a worse experience, and P99 reflects the worst experience. For online services, P99 is especially important because it determines stability under high concurrency or long-context scenarios.
4. A Reproducible Speed-Test Script Example
The example below is based on a common streaming completion API and uses only the standard library. It measures first-packet latency, the first thinking token, the first visible token, the number of thinking tokens, the number of visible tokens, and approximate throughput. Before use, make sure your API is compatible with the OpenAI Chat Completions streaming format.
import jsonimport timeimport urllib.requestdef benchmark_stream(prompt, base_url, model, timeout=120):"""Measure key speed metrics of an open-source LLM's streaming response.Notes:1. This script uses the number of chunks with non-empty delta.content asan approximate count of visible tokens.2. If the server returns usage, prefer completion_tokens / reasoning_tokensfrom the usage field.3. If the API does not support stream_options, you can remove that field."""url = base_url.rstrip("/") + "/v1/chat/completions"body = json.dumps({"model": model,"messages": [{"role": "user", "content": prompt}],"stream": True,"stream_options": {"include_usage": True},}).encode("utf-8")req = urllib.request.Request(url,data=body,headers={"Content-Type": "application/json"},method="POST",)start = time.perf_counter()first_packet_at = Nonefirst_reasoning_at = Nonefirst_visible_at = Nonetoken_count = 0reasoning_token_count = 0completion_tokens_from_usage = Nonereasoning_tokens_from_usage = Nonewith urllib.request.urlopen(req, timeout=timeout) as resp:for raw in resp:line = raw.decode("utf-8", errors="ignore").strip()if not line or not line.startswith("data:"):continuepayload = line[len("data:"):].strip()if payload == "[DONE]":breaknow = time.perf_counter()if first_packet_at is None:first_packet_at = nowtry:obj = json.loads(payload)except json.JSONDecodeError:continueusage = obj.get("usage")if isinstance(usage, dict):completion_tokens_from_usage = usage.get("completion_tokens", completion_tokens_from_usage)reasoning_tokens_from_usage = usage.get("reasoning_tokens", reasoning_tokens_from_usage)details = usage.get("completion_tokens_details") or {}if reasoning_tokens_from_usage is None:reasoning_tokens_from_usage = details.get("reasoning_tokens")choices = obj.get("choices") or []if not choices:continuedelta = choices[0].get("delta") or {}content = delta.get("content")reasoning = delta.get("reasoning_content") or delta.get("reasoning")if reasoning:if first_reasoning_at is None:first_reasoning_at = nowreasoning_token_count += 1if content:if first_visible_at is None:first_visible_at = nowtoken_count += 1end = time.perf_counter()if first_visible_at is None:raise RuntimeError("No user-visible content was received. ""Check the model, prompt, or streaming output configuration.")total_time = end - startafter_ttft_time = max(end - first_visible_at, 1e-9)return {"first_packet_s": first_packet_at - start if first_packet_at else None,"first_reasoning_s": first_reasoning_at - start if first_reasoning_at else None,"ttft_s": first_visible_at - start,"total_s": total_time,"token_count_approx": token_count,"reasoning_token_count_approx": reasoning_token_count,"completion_tokens_from_usage": completion_tokens_from_usage,"reasoning_tokens_from_usage": reasoning_tokens_from_usage,"visible_tps_total": token_count / total_time,"visible_tps_after_ttft": token_count / after_ttft_time,}
At a minimum, this script helps developers confirm three key questions:
- How long it takes the model to produce the first user-visible content.
- Whether the thinking phase consumes a large amount of time.
- What the actual generation speed is once the initial first-token wait is excluded.
5. Coding Ability: From a Single Score to Real Tasks
Coding ability is one of the dimensions developers care about most, yet it is also the one most easily misled by a single score. A model may perform well on algorithm problems but fail to produce runnable code in real engineering tasks; conversely, it may be mediocre at single-function completion yet understand cross-file context well.
5.1 Don’t Look at the Overall Leaderboard Only
It is recommended to build your evaluation checklist across the following dimensions:
- HumanEval or MBPP: to assess basic function-generation ability.
- LiveCodeBench: to reduce the impact of training-data contamination and stay closer to recent tasks.
- Custom bug-fix cases: to assess the model’s ability to locate problems, modify code, and avoid introducing new errors.
- Cross-file understanding tasks: to assess the model’s ability to establish relationships across multiple files and modules.
- Real application-generation tasks: for example, generating a data dashboard, a visualization page, a command-line tool, or a simple backend API.
5.2 Scoring Items for a Coding Evaluation
To make coding evaluations more actionable, it is recommended to include at least the following scoring items:
- Runnability: whether the generated code can start or execute normally.
- Test pass rate: the proportion of unit tests or hidden cases passed.
- Bug-fix rate: whether it can correctly fix a given failing case.
- Requirement completeness: whether the main functional points are covered.
- Cross-file understanding: whether modules, functions, or interfaces are referenced correctly.
- Code quality: whether there are obvious security vulnerabilities, resource leaks, or logic errors.
- First-pass rate: the proportion of tasks that pass acceptance on the first generation.
5.3 An Example of Acceptance Criteria
If a team needs to establish an executable baseline, the following acceptance criteria can be used as a reference:
- Basic function generation: the hidden-case pass rate meets the team’s threshold, with no obvious runtime errors.
- Runnability: in real application tasks, the generated project can build or start successfully.
- Requirement coverage: coverage of the main functional points is not lower than a pre-defined proportion.
- Bug fixing: the fix does not introduce new blocking errors.
- Cross-file tasks: the model correctly identifies file dependencies, call relationships, and key interfaces.
- Real application tasks: satisfy at least the three conditions of “runnable, demonstrable, and open to further iteration.”
With this approach, coding evaluation is no longer a vague aggregate score, but a set of engineering criteria that can be reproduced, compared, and continuously improved.
6. A Scenario-Based Selection Decision Table
Different business scenarios prioritize different metrics. It is recommended to establish a selection table like the following:
| Scenario | Priority Metrics | Focus |
|---|---|---|
| Real-time chat | First visible token latency, P99, stable throughput | User wait time |
| Code completion | First-token latency, P90, completion pass rate | Instant response and accuracy |
| Offline batch processing | Average throughput, batch concurrency, unit cost | Total task completion time |
| Long-context understanding | Context stability, first-packet latency, VRAM usage | Usability with long inputs |
| Reasoning tasks | Thinking-phase time, answering-phase quality | Whether the thinking wait is worth it |
| Low-resource deployment | Quantization effect, VRAM usage, throughput ceiling | Hardware threshold and deployment cost |
When selecting, you should not look at the overall ranking alone. If your business is mainly code completion, then first-token latency and completion pass rate matter more than the overall leaderboard score; if your business is offline report generation, then throughput and unit cost deserve higher priority.
7. Engineering Deployment and Cost Control
Evaluation conclusions ultimately have to land in production. It is recommended to establish long-term mechanisms in the following areas:
- Tiered deployment: use low-latency strategies for interactive scenarios and high-throughput strategies for offline batch processing.
- Continuous measurement: continuously collect latency, throughput, error rates, and retry/failure counts in canary or staging environments.
- Elastic scaling: scale inference instances on demand through mainstream cloud platforms or container platforms to balance cost against peak load.
- Observability: combine logs, monitoring, and alerting to continuously track P50, P90, P99, and anomalous requests.
- Cost estimation: record GPU-hour cost, QPS, cache hit rate, batch throughput, and unit token cost.
A simple cost template can include:
- Per-instance hourly cost.
- Average number of tokens generated per hour.
- Scale-out threshold under peak concurrency.
- Cost savings from the cache hit rate.
- The quality-versus-cost tradeoff of different quantization schemes.
Through continuous measurement, evaluation can shift from a one-time experimental conclusion to an operable quality baseline.
8. A Common Mistakes Checklist
The following issues are especially common in practice:
- Looking only at average throughput, without checking P90 and P99.
- Not warming up and testing cold-start results directly.
- Not fixing decoding parameters, making results non-reproducible.
- Inconsistent concurrency conditions, making data across batches incomparable.
- Counting only SSE packets without distinguishing user-visible tokens.
- Running only standardized problems without testing real business tasks.
- Ignoring the wait cost introduced by the thinking phase.
- Looking at a single coding leaderboard without checking cross-file and engineering-task ability.
- Using unconditional performance numbers to represent all scenarios.
- Testing only speed without combining hardware, quantization method, and deployment cost in an overall judgment.
9. Conclusion
The core of open-source LLM evaluation is not finding a permanently correct leaderboard, but building a measurement system suited to your own business. Break “fast” into throughput, first-token latency, phased timing, and percentiles; break “strong” into real tasks, coding benchmarks, and acceptance criteria; then combine these with cost to form an actionable selection conclusion. Only by moving beyond single-score thinking can developers find a truly suitable balance among performance, quality, and cost.
