Lightweight LLM Selection: How to Evaluate Capability and Inference Speed

    A practical framework for selecting lightweight large language models by measuring task quality, code execution success, latency, throughput, and cost under consistent testing conditions. The article also explains how to design business-relevant test sets, report latency percentiles, validate benchmark credibility, and turn evaluation results into clear model-selection thresholds.

    1. Why Lightweight Model Selection Requires a Methodology

    Lightweight models generally refer to models with smaller parameter counts, lower inference costs, and shorter response times. They are commonly used for latency-sensitive scenarios such as online customer service, code completion, document summarization, and batch data processing.

    When selecting a model, teams often consider leaderboard rankings, speed figures from marketing pages, and the results of a few manual tests. When these sources produce conflicting conclusions, the problem is usually inconsistent evaluation methods rather than the models themselves.

    A reliable evaluation should answer three questions:

    1. What level of performance can the model achieve on the target tasks?
    2. Can its outputs be used directly in production workflows?
    3. Can its response speed meet the business latency budget?

    The following sections examine these three dimensions.

    2. Overall Capability: Evaluate with Business-Relevant Test Sets

    Overall capability covers knowledge question answering, logical reasoning, instruction following, and format compliance. Public benchmarks can provide useful reference points, but two issues require attention.

    First, benchmark questions may already be included in a model’s training data, resulting in inflated scores. It is therefore best to include some private, previously unpublished samples.

    Second, the distribution of a general-purpose benchmark may differ significantly from the distribution of real business requests. Teams should build a test set based on their own request types, retain multiple samples for each task category, and document either the expected answers or the scoring criteria.

    Evaluation methods generally fall into three categories:

    • Exact match: Suitable for multiple-choice questions, numerical results, and fixed-format outputs.
    • Rule-based validation: For example, checking whether JSON can be parsed and whether all required fields are present.
    • Human or model-assisted scoring: Suitable for open-ended responses. A consistent scoring rubric is required, and a sample should be reviewed to measure agreement between evaluators.

    3. Coding Capability: Measure Executable Results

    Coding evaluations should focus on whether the generated code runs and passes tests, rather than whether it merely appears reasonable. Two commonly used metrics are:

    • pass@1: The proportion of tasks passed on the first generation attempt.
    • pass@k: The proportion of tasks for which at least one of k generated solutions passes, reflecting the model’s upper-bound capability.

    For tasks that require modifying an existing codebase, requirements such as fixing defects or adding an interface can be converted into test cases with unit tests. The model makes changes in a sandbox environment, and a test suite determines the result. This produces conclusions that are easier to reproduce.

    Demonstration-oriented tasks, such as generating complex visualization scenes from natural-language descriptions, often lack clear validation criteria and should be treated only as supplementary observations. Public conclusions should be based on reproducible test suites.

    4. Inference Speed: Define the Metrics First

    Speed metrics are often used inconsistently, so their definitions should be established before testing.

    • Time to first token (TTFT): The time from sending a request to receiving the first output token. This directly determines when the user perceives that the response has started.
    • Output throughput: The number of tokens generated per second during the decoding phase. It is calculated as (output tokens - 1) ÷ generation time, where generation time starts when the first token is received.
    • End-to-end latency: The total time from sending the request until generation is complete. It equals TTFT plus generation time.
    • Peak throughput: The highest throughput observed within a one-second interval. It is strongly affected by batching, caching, and transient concurrency, so it does not represent stable performance.

    For models that include a reasoning process, the reasoning and answer-generation stages should be measured separately. The report should state whether tokens generated during reasoning are included in throughput and billing. Otherwise, the figures cannot be compared meaningfully.

    Statistical results should include percentiles such as P50, P90, and P95 instead of relying only on averages. Averages can be distorted by a small number of long-tail requests, while percentiles more closely reflect real user experience.

    5. Keep the Test Environment Consistent

    A set of speed measurements is comparable only when the testing conditions are consistent. The following variables should be standardized:

    1. Hardware and deployment method: Use the same class of compute resources and the same concurrency level.
    2. Input distribution: Set prompt lengths and maximum output lengths based on real business logs rather than fixed short prompts.
    3. Sampling parameters: Keep parameters such as temperature and top-p consistent, and state whether reasoning mode is enabled.
    4. Warm-up and repetition: Send several warm-up requests first. Repeat each scenario over multiple rounds and report the resulting distribution.
    5. Network factors: Separate server-side processing time from client-side network time. When necessary, run tests within the same region.

    6. Check the Credibility of Evaluation Results

    Before accepting an evaluation report, review it against the following questions:

    • Is the sample size large enough to estimate a confidence interval?
    • Is there any benchmark contamination or deliberate tuning against the test set?
    • Was the comparison conducted blindly, and did evaluators know which model produced each response?
    • Does the report provide both quality and speed metrics, rather than highlighting only one advantage?

    7. Example Test Script Framework

    The following code demonstrates how to measure TTFT and output throughput for a single request. The caller must implement call_model, which should return text fragments through a streaming interface. This example illustrates the measurement logic only; the API details must be adapted to the actual service.

    A streaming fragment is not necessarily equivalent to a single token. Formal testing should use token-usage information returned by the server.

    1. import time
    2. import statistics
    3. def measure_once(call_model, prompt):
    4. start = time.perf_counter()
    5. first_token_at = None
    6. token_count = 0
    7. for _chunk in call_model(prompt): # Streaming fragment, approximately treated as one token
    8. now = time.perf_counter()
    9. if first_token_at is None:
    10. first_token_at = now
    11. token_count += 1
    12. end = time.perf_counter()
    13. if first_token_at is None:
    14. raise RuntimeError("No output received")
    15. decode_time = end - first_token_at
    16. return {
    17. "ttft": first_token_at - start,
    18. "tps": (token_count - 1) / decode_time if decode_time > 0 else 0.0,
    19. "total": end - start,
    20. }
    21. def summarize(samples):
    22. ttfts = sorted(s["ttft"] for s in samples)
    23. tps = [s["tps"] for s in samples]
    24. p95_index = min(len(ttfts) - 1, int(0.95 * (len(ttfts) - 1)))
    25. return {
    26. "ttft_p50": statistics.median(ttfts),
    27. "ttft_p95": ttfts[p95_index],
    28. "tps_median": statistics.median(tps),
    29. }

    The script produces metrics for each sample, which are then aggregated into percentiles by summarize. These results can be compared directly with the application’s latency budget.

    8. From Test Results to a Selection Decision

    Evaluation data must ultimately be mapped to business constraints. One practical approach is to define three gates:

    • Quality gate: The pass rate for critical tasks must not fall below the baseline.
    • Latency gate: P95 TTFT must not exceed the business budget.
    • Cost gate: The cost per request must remain within the acceptable range.

    First eliminate candidates that fail any required gate. Then compare the remaining models using the metrics that matter most for the target scenario. Different use cases prioritize different indicators. The following table provides a reference:

    Business scenario Primary metric Secondary metric
    Real-time conversation and customer service P95 TTFT Instruction-following rate
    Code completion and programming assistance Test pass rate Output throughput