How to Quantify Performance Regressions in Large Language Model Services

1. Defining the Problem

Large language model services evolve rapidly. Users may notice that similar tasks take longer, consume more tokens, or occasionally fail to solve problems that previously worked. These impressions may reflect real changes, but they can also result from cognitive bias or inconsistent measurement.

For teams that rely on LLM services in production, “it feels worse” must be converted into a verifiable conclusion. Otherwise, it is difficult to determine whether to revise the prompt, change the calling pattern, or report a potential issue to the service provider.

This article addresses three questions:

• What causes the perception that an LLM service has degraded?
• Which metrics can objectively describe service behavior?
• How can you design reproducible comparison experiments and use the results to control costs?

2. Common Sources of Perception Bias

The gap between user perception and actual service behavior usually comes from several factors.

1. Inconsistent measurement definitions

   “Latency” may refer to time to first token or the time required to receive the complete output. If one measurement includes client-side rendering time and another does not, the two datasets cannot be meaningfully compared. Before comparing results, make sure both measurements use the same start and end points.

2. Changes in tasks or inputs

   The prompt, context length, and attached files may have changed even when the business request appears identical. If the context grows from 2,000 tokens to 200,000 tokens, the increase in latency and cost may primarily result from the larger input rather than a change in the model itself.

3. Sample bias

   People generally remember failures more vividly than successful results. A small number of memorable failures can create an exaggerated negative impression of the overall distribution. A single interaction cannot represent the performance of a service.

4. Normal service-side variation

   Queuing, traffic peaks, and differences between service nodes can all cause latency fluctuations. One timeout does not prove that the service strategy has changed.

5. Service-side configuration changes

   Model versions, inference-depth settings, request-routing strategies, and billing rules may change over time. These changes are often not fully transparent to external users and can only be observed indirectly through long-term data. Without supporting evidence, avoid drawing immediate conclusions.

The first step in identifying bias is to replace “I feel that it has become slower” with a measurable statement such as: “Under the same input conditions, median latency increased from one value to another.”

3. A Quantifiable Metrics Framework

At a minimum, collect the following metrics and preserve the raw record for every request.

• Time to first token (TTFT): The time from sending a request to receiving the first output fragment. This primarily reflects queueing and preprocessing overhead. Measuring TTFT requires a streaming interface.
• Total latency: The time from sending a request until the output is complete. Record both the median, or P50, and the 90th percentile, or P90. P90 helps expose long-tail latency.
• Input and output token counts: These directly affect billing and context usage. Use the usage fields returned by the service rather than replacing them with local estimates.
• Per-task cost: Multiply input and output tokens by their respective unit prices and add the results. Compared with token counts alone, per-task cost provides better support for business decisions.
• Task pass rate: Use automatically verifiable checks, such as regular-expression matching, unit tests, JSON schema validation, or comparison with a reference answer.
• Result stability: Measure the consistency rate or quality variance when running the same input multiple times.

These metrics should be interpreted together:

Observation	Possible cause	Priority action
Latency increases significantly while output tokens remain mostly unchanged	Queueing, network conditions, or service-side load	Examine the latency distribution and request time windows
Output tokens increase without a higher pass rate	Additional reasoning is not producing better results	Simplify prompts and split the task
Input tokens, latency, and cost all increase	Context growth	Trim conversation history and summarize context
Pass rate decreases while other metrics remain stable	A change in output quality or prompts	Compare prompt versions and sample results

4. Building a Reproducible Baseline

Subjective comparisons are easily affected by unrelated variables. A useful baseline should meet four conditions.

1. Use a fixed test set

   Prepare 50 to 200 samples covering the main business scenarios. Include easy, medium, and difficult tasks, and preserve historical failure cases for regression testing.

2. Use a fixed prompt template

   Run every test with the same template version. When the template changes, record a separate version number and identify it in the report.

3. Use fixed request parameters

   Keep parameters such as sampling temperature, maximum output length, and stop sequences consistent. Higher sampling temperatures increase the randomness of individual results, which reduces the reliability of direct comparisons.

4. Run multiple samples and perform statistical tests

   Run each sample multiple times. Compare medians and percentiles rather than relying on a single result. A non-parametric test, such as the Mann–Whitney U test, can be used to assess whether differences between two time periods are statistically significant. Because it does not require normally distributed data, it is suitable for skewed measurements such as latency.

Archive baseline data by date. When users report a change in perceived performance, the team can compare it directly with historical measurements instead of relying on memory.

5. Illustrative Evaluation Script

The following code is independent of any specific service. The call_model function must be implemented by the caller for the interface being used. It should return the generated text and the usage fields reported by the service.

This example measures total completion time. To measure TTFT, use a streaming interface and record the timestamp when the first text chunk arrives.

import time
import statistics
def measure(call_model, prompt, check, runs=20):
    """Call the same prompt multiple times and collect latency, usage, and pass results.
    call_model: A function that returns a dict containing
                text, input_tokens, and output_tokens
    check:      A validation function that accepts the output text
                and returns a boolean
    """
    records = []
    for _ in range(runs):
        start = time.perf_counter()
        resp = call_model(prompt)
        elapsed = time.perf_counter() - start
        records.append({
            "elapsed": elapsed,
            "in_tokens": resp["input_tokens"],
            "out_tokens": resp["output_tokens"],
            "passed": bool(check(resp["text"])),
        })
    return records
def summarize(records):
    latencies = [r["elapsed"] for r in records]
    return {
        "latency_p50": statistics