LLM Coding Performance Degradation: Diagnosing Issues Through Latency, Token Cost, and Task Pass Rate

1. Where the Problem Comes From

Recently, many developers have reported changes in the behavior of the same models, concentrated in three areas.

First, response times have increased. For the same prompt, the time to see the first character has noticeably grown, and the time to complete the full output is longer as well.

Second, token consumption per question has risen. Edits that previously took a single round of conversation now require multiple rounds of interaction, outputs are longer, and costs increase accordingly.

Third, the failure rate on complex tasks has gone up. For performance optimization tasks, for example, the model can produce seemingly reasonable analysis but struggles to deliver verifiable improvements, and its explanations grow longer and longer.

These impressions are valuable, but feelings alone make it hard to determine whether the problem lies in the model, the serving side, the prompting approach, or the task itself. Breaking the problem apart and measuring each component is a more reliable starting point.

2. Rule Out Common Confounding Factors First

Before concluding that “the model got worse,” you should check the following categories of causes.

1. Server load and queuing. During peak hours, providers may rate-limit or queue requests, which manifests as increased time-to-first-token. Automatic client retries after failed requests can also double end-to-end latency. Checking error codes and retry logs can quickly identify this class of problem.

2. Context length inflation. In long sessions, history messages accumulate continuously, and the input tokens for each request keep growing. The model has more information to process, attention is diluted, results may degrade, and costs rise as well. Periodically compressing the context or starting a new session often restores performance.

3. Changes to prompts and tool configuration. After system prompts, tool definitions, or project rules are modified, model behavior changes accordingly. This is especially easy to overlook when multiple automation scripts inject content into the prompt together, silently altering how the model responds.

4. Changes in task difficulty. Teams gradually move toward more complex requirements, or benchmark tasks have already been optimized by the community, both of which can make the same model appear to perform worse.

5. Internal computation of reasoning models. Some reasoning models consume internal reasoning tokens that are not shown to users. These tokens may be billed and also increase response time. For specific billing rules and reasoning parameters, refer to the provider’s documentation and verify them before use.

3. Establish Quantifiable Evaluation Metrics

Subjective evaluation is easily influenced by recent experience. The following metrics are recommended for describing model performance.

Metric	Meaning	Problem Reflected
Time to first token (TTFT)	Time from sending a request to receiving the first output character	Queuing, serving load
End-to-end latency	Total time from request start to output completion	Output length, reasoning time
Input/output token count	Token usage per call	Cost, context inflation
Cost per task	Total token cost to complete one task	Overall efficiency
First-pass rate	Proportion of task results that pass automated validation	Capability changes
Retry count	Average number of calls per task	Stability

Among these, the first-pass rate best reflects actual capability. It requires each task to have an explicit validation method, such as unit tests, output comparison against benchmark scripts, or fixed performance metric thresholds. Tasks without automated validation can hardly avoid subjective bias.

4. Build a Fixed Regression Benchmark

A benchmark set should satisfy three conditions: fixed tasks, automated validation, and reproducible results. Below is a general evaluation framework, where call_model needs to be replaced with your team’s own model invocation wrapper. The usage field names may differ across providers and need to be adapted.

import time
import statistics
from dataclasses import dataclass
@dataclass
class RunResult:
    task_id: str
    latency_s: float
    prompt_tokens: int
    completion_tokens: int
    passed: bool
def run_benchmark(tasks, call_model, check):
    """Call the model once per task, recording latency, token usage, and validation results."""
    results = []
    for task in tasks:
        start = time.perf_counter()
        resp = call_model(task["prompt"])
        latency = time.perf_counter() - start
        results.append(RunResult(
            task_id=task["id"],
            latency_s=latency,
            prompt_tokens=resp["usage"]["prompt_tokens"],
            completion_tokens=resp["usage"]["completion_tokens"],
            passed=check(task, resp["text"]),
        ))
    return results
def summarize(results):
    n = len(results)
    return {
        "pass_rate": sum(r.passed for r in results) / n,
        "median_latency_s": statistics.median(r.latency_s for r in results),
        "avg_completion_tokens": statistics.mean(r.completion_tokens for r in results),
    }

When using this framework, note the following points.

• Tasks should be diverse. The benchmark should include code modification, defect localization, performance analysis, and code review, among other types, to avoid bias from a single task category.
• Run multiple times and average. LLM outputs are stochastic, and a single run is not enough to draw conclusions. Run each task at least 3 to 5 times and record the distribution.
• Fix the environment. Identical prompt templates, tool configuration, temperature parameters, and context length are required to make two evaluations comparable.
• Validate results in a sandbox. Model-generated code should be executed in an isolated container with restricted network and file system access to avoid the risks of running untrusted code.

5. How to Determine Whether Degradation Is Real

With evaluation results in hand, you still need to judge whether the difference between two runs is credible.

Suppose the first-pass rate on the same benchmark was 70% last week and 60% this week. This 10-percentage-point gap may simply be random fluctuation, especially when the number of tasks is small. You can use bootstrapping to estimate a confidence interval for the difference:

import random
def bootstrap_diff(old_flags, new_flags, rounds=10000, seed=0):
    """Estimate the 95% confidence interval of the difference in pass rates between two groups. old_flags and new_flags are lists of 0/1 values."""
    rng = random.Random(seed)
    diffs = []
    for _ in range(rounds):
        o = [rng.choice(old_flags) for _ in old_flags]
        n = [rng.choice(new_flags) for _ in new_flags]
        diffs.append(sum(n) / len(n) - sum(o) / len(o))
    diffs.sort()
    return diffs[int(0.025 * rounds)], diffs[int(0.975 * rounds)]

Only when the confidence interval lies entirely on one side of zero can you be reasonably confident that performance has changed. If the interval crosses zero, the data is insufficient to draw a conclusion, and you should increase the number of tasks or runs.

6. Engineering Practices to Reduce Risk

Before evaluation conclusions are available, teams still need to complete work reliably. The following practices can reduce the impact of model fluctuations.

Pin model versions. If the provider supports specifying a concrete version, pin the version number in configuration rather than using an alias that updates at any time. When the version changes, validate on the benchmark set before switching production traffic.

Tier tasks. Different tasks have different requirements for model capability. Simple tasks such as code retrieval and format conversion can use lower-cost models; complex architecture design and performance optimization should be reserved for models with more stable performance, combined with human review.

Set fallback paths. When the primary model’s latency exceeds a threshold or it fails repeatedly, automatically switch to a backup model or queue a retry. Fallback logic should be logged for later analysis.

Control context. Set an upper limit for long sessions, and summarize history or start a new session when it is exceeded. For code tasks, pass only the relevant files and necessary error information rather than stuffing the entire repository into the context.

Record and audit. Save the prompt, model version, token usage, latency, and results of every call. Without historical data, you cannot determine whether “it got worse recently” is actually true.

Treat prompt injection with care. If multiple scripts or plugins automatically append content to prompts, review that content regularly. Irrelevant instructions change model output and are very hard to notice after the fact.

7. Practical Checklist

• Use a fixed benchmark set to measure first-pass rate, time to first token, token usage, and cost per task.
• Before each comparison, confirm that model version, prompt template, tool configuration, and context length are identical.
• Run each task at least 3 to 5 times, and use confidence intervals to judge whether differences are significant.
• Execute generated code in an isolated environment with restricted external access.
• Configure timeouts, retry limits, and a backup model for the primary model.
• Keep call logs and turn subjective impressions into data that can be reviewed.