LLM Selection Guide: Five Core Metrics and How to Measure Them

Why Selection Should Be Based on Measurement

When teams choose a large language model, a common approach is to read official documentation, compare parameter tables, and then go live. The problem with this approach is that parameter tables describe upper bounds under ideal conditions, while business requests have their own length distributions, concurrency pressure, and compliance requirements. Model selection should turn each metric into a measurable number and ensure that others can reproduce the results using the same method.

This article focuses on five metrics: context window, output limit, price structure, response latency, and false-block rate in safety filtering. For each metric, it explains the definition, common pitfalls, measurement method, and provides example code. The numbers in the examples are hypothetical values for demonstration only and do not represent the actual parameters of any service.

1. Context Window: Nominal Length Is Not Effective Length

The context window refers to the maximum length a model can handle in a single request, usually including both input and output, measured in tokens. A token is the basic unit by which a model splits text. A Chinese character may correspond to one or more tokens, and an English word may also correspond to one or more tokens, depending on the tokenization method. Therefore, when estimating budgets, use the same tokenizer as the target model rather than a rough conversion based on character count.

Two concepts need to be distinguished:

• Nominal window: the maximum value declared in the API documentation.
• Effective window: the range within which the model can still accurately retrieve, cite, and reason about information at that length.

In long text, when key information is located in the middle, the model’s retrieval accuracy is usually lower than when the information is at the beginning or end. Therefore, a “needle in a haystack” style test can be used to check the effective window. The specific approach is to insert a fact at different total lengths and different positions, ask the model to answer, calculate the accuracy, and finally plot the relationship between length and accuracy.

Example code for a budget check:

def check_input_budget(context_limit, reserved_output, prompt_tokens_list):
    """Calculate the available input budget for a single request and flag samples that exceed it."""
    input_budget = context_limit - reserved_output
    report = []
    for idx, tokens in enumerate(prompt_tokens_list):
        report.append({
            "sample": idx,
            "tokens": tokens,
            "fits": tokens <= input_budget,
            "headroom": input_budget - tokens,
        })
    return report

reserved_output should be set according to the longest business output; otherwise, long requests will be truncated during the generation stage.

2. Output Limit: Focus on Truncation and Structured Output

The output limit determines how much content can be generated in a single request. Business scenarios fall roughly into two categories. One is tasks that require large amounts of output, such as long reports and code generation, where the focus is whether the limit is sufficient. The other is tasks with very short output, such as classification and information extraction, where the focus is stability and format adherence.

When measuring, check three things: whether output is truncated when the limit is reached, whether there is a clear end marker after truncation, and whether structured output such as JSON can be parsed normally. Parsing failures caused by truncation often appear as intermittent errors in production and need to be counted separately; otherwise, they may be mistaken for random failures.

In practice, you can add validation and a limited number of retries for structured output. Retries increase token consumption and latency, so the number of retries should be included in the cost model and capped.

3. Price Structure: Input, Output, and Cache Are Priced Separately

Usage-based model services usually price input tokens and output tokens separately. Some services also offer a lower unit price for cached input. Output unit prices are generally higher than input unit prices, so tasks that generate long text see costs grow faster than input.

Cost estimation should be based on the request distribution of real traffic. Simply multiplying the per-request price by the number of requests ignores the difference between long and short requests. The correct approach is to accumulate the input, cache, and output token costs for each request individually. Model unit prices change frequently, so it is advisable to put unit prices in a configuration file and periodically verify them against the prices published by the service provider.

# Example values in CNY per million tokens, for demonstrating the calculation method only
PRICE_PER_MILLION = {
    "input": 8.0,
    "cached_input": 1.0,
    "output": 40.0,
}
def estimate_cost(requests, price=PRICE_PER_MILLION):
    """Each item in requests contains input_tokens (cache miss), cached_tokens, and output_tokens."""
    total = 0.0
    for req in requests:
        total += req["input_tokens"] * price["input"] / 1_000_000
        total += req["cached_tokens"] * price["cached_input"] / 1_000_000
        total += req["output_tokens"] * price["output"] / 1_000_000
    return round(total, 2)

Cost estimates should also include the cost of retries and failed requests, which are often overlooked.

4. Response Latency: Distinguish Time to First Token from Overall Time

Latency includes at least two metrics. The first is Time to First Token, which is the time from sending a request to receiving the first output fragment. The second is generation speed, which is the number of tokens output per second. Interactive products are sensitive to Time to First Token, while batch tasks care more about overall throughput.

Speeds published by service providers are usually results under ideal conditions. During model selection, you should run your own load tests in the target region, under target concurrency, and with target request lengths, and record P50, P95, and P99 percentiles. Averages hide long-tail problems; a small fraction of timed-out requests may already affect a large number of users.

import statistics
def latency_percentiles(samples_ms):
    """samples_ms needs at least 100 samples to calculate P99 reliably."""
    qs = statistics.quantiles(samples_ms, n=100)
    return {"p50": qs[49], "p95": qs[94], "p99": qs[98]}

Load tests should use the real prompt length distribution rather than fixed-length test text.

5. False-Block Rate in Safety Filtering

Content safety policies block some requests. For normal business operations, two types of errors need attention. A missed block refers to a request that should have been blocked but was allowed through. A false block refers to a normal request being incorrectly blocked. False blocks directly affect user experience and are easily underestimated. They usually do not appear in error logs; instead, they manifest as higher task failure rates and lower conversion.

The steps for building an evaluation set are as follows:

1. Collect samples by business scenario, covering normal but easily misjudged domains such as biomedicine, cybersecurity, law, and healthcare.
2. Have humans label whether each sample should be blocked.
3. Calculate the false-block rate and missed-block rate separately.

The false-block rate is calculated as the number of blocked normal samples divided by the total number of normal samples.

def false_block_rate(results):
    """Each item in results is {"should_block": bool, "blocked": bool}."""
    benign = [r for r in results if not r["should_block"]]
    if not benign:
        return 0.0
    blocked = sum(1 for r in benign if r["blocked"])
    return blocked / len(benign)