Open-Source LLM Inference Evaluation and Deployment: A Practical Guide
This guide explains how to evaluate open-source large language models without relying on vendor-published scores, and how to apply evaluation results to production deployment decisions. It covers execution-based code evaluation, inference latency and throughput metrics, configuration-aware benchmarking, and engineering practices such as model routing, prefix caching, constrained decoding, observability, and canary rollout. It also outlines a model selection workflow and common pitfalls, emphasizing that results should be tied to real workloads and business SLAs.
1. Why model evaluation should be vendor-neutral
The open-source LLM community releases a large number of lightweight models every month, each with different parameter sizes, architecture choices, and inference acceleration strategies. When enterprises evaluate models, they often face a difficult problem: benchmark scores published at launch events look impressive, but the actual experience in their own business scenarios may differ significantly from the claimed results.
There are two common reasons for this gap. First, public availability of test sets varies, making data contamination difficult to rule out. Second, evaluation methods are inconsistent. Some teams focus only on generation speed, while others focus only on accuracy, leaving little room for fair comparison.
As a result, an unbiased evaluation framework should satisfy at least three requirements: public evaluation criteria, execution-based judging, and reproducible metrics. This framework benefits three groups: model providers can identify bottlenecks, integrators can choose suitable deployment configurations, and business teams can assess return on investment.
The following sections break down evaluation methods across three core dimensions.
2. Evaluating coding ability: from runnable code to high-quality code
Coding ability should not be evaluated only through static scoring based on natural-language descriptions. A more reliable approach is execution-based evaluation: run the generated code in an isolated sandbox and use unit test pass rates as objective signals.
Common evaluation approaches include:
- Function-level evaluation: Given a function signature and docstring, ask the model to complete the implementation and determine whether it passes a set of hidden test cases. Metrics such as pass@1 and pass@k are commonly used to measure first-attempt success and coverage across multiple samples.
- Repository-level evaluation: Extract issues and corresponding patches from real open-source projects, then ask the model to generate fixes under multi-file, cross-context conditions. This tests long-context understanding and dependency management.
- Frontend and visualization tasks: Ask the model to generate a runnable 3D rendering page in one pass, such as a solar system orbit simulation or a recursively growing tree. These tasks evaluate understanding of geometry and transformation matrices, as well as the correctness of the rendering loop.
A key engineering requirement for execution-based evaluation is sandbox isolation and resource limits. CPU, memory, and execution time should be constrained, network access should be disabled, and runtime versions should be unified. Otherwise, results are not comparable across different runs.
An example judgment script is shown below:
import jsonimport subprocessdef evaluate(sample, tests, timeout=10):"""Execute the model-generated code in a subprocess and return whether all tests pass."""script = sample + "\n" + teststry:proc = subprocess.run(["python", "-I", "-c", script],capture_output=True,timeout=timeout,text=True,)except subprocess.TimeoutExpired:return Falsereturn proc.returncode == 0def pass_at_k(samples, tests, k=1):ok = sum(1 for s in samples[:k] if evaluate(s, tests))return ok / k
Beyond pass@1, it is also useful to record the distribution of results, including the median number of passed test cases and the category of the first failing test case, such as boundary conditions, exception handling, or performance constraints. This detailed information helps distinguish whether a model has a capability gap or a style mismatch.
3. Evaluating inference speed: do not look only at peak throughput
“Tokens per second” is the most commonly used promotional metric from vendors, but it usually describes the decoding upper limit under ideal conditions. In real business scenarios, three types of metrics deserve more attention:
| Metric | Meaning | Business impact |
|---|---|---|
| TTFT (time to first token) | Time from request submission to the return of the first token | Determines the perceived responsiveness of the system |
| TPOT / ITL (time per output token / inter-token latency) | Average time spent generating each token during the output stage | Determines the smoothness of streaming output |
| Effective throughput | Number of usable tokens produced per unit of time | Determines concurrency capacity and cost |
It is also important to note that if a model first generates a long chain of reasoning before providing the final answer, tokens produced during the reasoning stage also consume compute resources and time. During evaluation, the “reasoning stage” and the “answer stage” should be measured separately. In addition, median response time should be used rather than the average, so that long-tail requests do not distort the conclusions.
The following script can be used to send asynchronous concurrent requests and record timestamps for each token:
import asyncioimport timeimport statisticsasync def bench(session, url, payload, rounds=50):ttfts, inter, total = [], [], []for _ in range(rounds):t0 = time.perf_counter()first = Noneprev = t0async with session.post(url, json=payload) as resp:async for chunk in resp.content:if chunk:now = time.perf_counter()if first is None:first = nowttfts.append((first - t0) * 1000)else:inter.append((now - prev) * 1000)prev = nowtotal.append((time.perf_counter() - t0) * 1000)return {"ttft_ms_p50": statistics.median(ttfts),"tpot_ms_p50": statistics.median(inter),"total_ms_p50": statistics.median(total),"tokens_per_s": 1000 / statistics.mean(inter) if inter else 0,}
Factors affecting inference speed are systemic. Evaluation conclusions are meaningful only when they are tied to a clear configuration description:
- Model structure: Sparse activation, such as Mixture-of-Experts (MoE), makes a tradeoff between total parameter count and the number of parameters activated per token. It can often achieve higher decoding speed with relatively little compute overhead.
- Parallelism strategy: Tensor parallelism reduces memory pressure per GPU but introduces communication overhead. Pipeline parallelism can improve throughput but may increase time to first token.
- Decoding optimizations: KV Cache reuse, speculative decoding (small draft model plus large verification model), and quantization (INT8/FP8) can significantly change inter-token latency.
- Serving-layer scheduling: Dynamic batching improves overall throughput but may increase queueing latency for individual requests. SLA priorities should be made explicit during stress testing.
4. Turning evaluation results into engineering practice
After collecting evaluation data, what ultimately determines user experience is the scheduling design on the engineering side. Several practices have been repeatedly validated in production:
Tiered routing. Use lightweight models to handle high-frequency simple tasks such as intent classification, summarization, and format conversion. Escalate to larger models only for complex reasoning and code generation. This approach offers a good balance between cost and latency.
Prefix caching and request reuse. System prompts, tool definitions, and few-shot examples often account for the majority of tokens in a request. When prefix caching is hit, time to first token can drop significantly.
Structured output constraints. Moving constraints such as JSON Schema or function signatures into the decoding stage through constrained decoding can reduce extra latency caused by retries and rework.
End-to-end observability. Request latency, token consumption, and failure reasons should be collected through logging services. Monitoring and alerting should set thresholds for TTFT and error rates so that performance regressions can be detected continuously in production. On cloud platforms, this observability and elastic-scaling pipeline can be assembled using general services such as model services, container platforms, object storage, and monitoring/alerting.
Canary release and rollback. Any model or parameter change should first be validated through traffic mirroring or shadow traffic. Rollout should proceed gradually only after both quality metrics and latency metrics meet the required thresholds.
5. Selection checklist and common pitfalls
For production evaluation, the following order is recommended:
- Define business SLAs, such as the maximum allowed time to first token and peak concurrency.
- Build an evaluation set from your own data, ensuring that it covers the real input distribution at least broadly.
- Use execution-based evaluation to assess quality.
- Stress-test latency distributions.
- Run a small-traffic canary release.
- Observe the cost curve.
Three common pitfalls should be avoided:
- Treating peak throughput as online capacity while ignoring latency amplification caused by queueing and batching.
- Replacing evaluation on your own data with public leaderboard rankings while ignoring differences in domain distribution.
- Measuring speed without measuring quality, which can result in fast but incorrect outputs and increase the cost of human review.
Overall, relative positioning statements such as “between two leading models” are far less valuable for decision-making than conclusions tied to specific scenarios, such as “pass rate improved by 12% on our dataset and P95 latency decreased by 300ms.” This is the form that evaluation results should take.
