Evaluating Large Language Models: From Benchmark Racing to User Experience

Whenever a new generation of large language models (LLMs) is released, opinions quickly split into two camps: one scrutinizes benchmark scores, while the other questions the actual user experience. This divide highlights a core issue—how we evaluate models has fallen behind how we actually use them. This article does not target specific products; rather, it discusses a broader technical question: in an era of benchmark inflation and endless evaluation leaderboards, how should developers establish a stable model evaluation framework without being swayed by public opinion?

1. Expectation Management: Why “Low Expectations” Lead to Higher Satisfaction

An interesting mindset when evaluating new versions is that having low expectations initially often leads to a satisfactory actual experience. This is not mere self-consolation; it is a repeatedly validated assessment strategy.

LLM iteration is currently in a phase of “incremental improvement” rather than “generational leaps.” Most version upgrades focus on engineering aspects such as training data ratios, alignment strategies, and inference efficiency. On leaderboards, this might translate to a few percentage points of improvement, but in daily use, it is often imperceptible. If you test a new model expecting a revolutionary transformation, disappointment is almost inevitable. Conversely, if your baseline expectation is just “normal progress,” any small improvement in stability, response speed, or instruction following will be perceived as a positive gain.

From an engineering management perspective, this suggests that model evaluation should establish tiered baselines. This can be broken down into three levels:

• Pass Line: Core tasks must not fail. Common tasks like code generation, text comprehension, and multi-turn dialogue should show no obvious degradation.
• Tracking Line: Benchmark scores should remain within a reasonable range compared to peer models, without falling a generation behind.
• Surprise Line: Perceptible improvements in specific scenarios (e.g., long context, tool calling, structured output).

Only when the pass line is secured and the tracking line is met is it necessary to discuss the surprise line. Many controversies actually stem from treating the surprise line as the pass line.

2. The True Value of Benchmarks: A Sieve, Not a Ruler

Benchmark scores are often misused by two types of people: those who use them as the sole basis for selection, and those who dismiss their significance entirely. Both approaches are biased.

The reasonable positioning of benchmarks is as a sieve—it excels at coarse-grained elimination. A model that clearly lags on mainstream benchmarks will likely expose shortcomings in real-world tasks. Although benchmark tests can be specifically optimized for, completely gaming them without affecting general capabilities is extremely costly. Therefore, benchmarks can quickly rule out obviously unqualified options and narrow down the candidate pool.

However, benchmarks are not a ruler—they cannot precisely measure daily usability. There are three main reasons for this:

1. Mismatch between benchmarks and business distributions: Public benchmarks mostly cover general tasks, while enterprise scenarios often focus on specific domains. The difficulty distribution between the two differs vastly.
2. Benchmarks do not reflect interactive experience: Metrics that directly impact user experience, such as first-token latency, streaming output stability, and context retention in long conversations, are rarely reflected in static leaderboards.
3. Saturation effect in benchmarks: When multiple models approach the upper limit on the same benchmark, score differences lose their discriminating power. Continuing to compare digits after the decimal point holds little significance.

A pragmatic approach is: use benchmarks for initial screening, and use tasks for final judgment. In the screening phase, focus on whether the model falls within a reasonable range. In the final judgment phase, hand the evaluation entirely over to your own business use cases.

3. Three Observable Dimensions of Daily Experience

Since benchmarks are insufficient to determine “how good a model is to use,” a set of actionable alternative metrics is required. It is recommended to observe from the following three dimensions, each of which can be verified at a low cost.

1. Stability: Reproducibility of the Same Input

Stability is the most easily overlooked dimension, yet it most impacts user trust. A model might occasionally provide a stunning answer, but if it gives an irrelevant response when the same question is phrased slightly differently, this uncertainty forces users to perform secondary checks every time, actually reducing efficiency.

The verification method is simple: prepare a set of 20 to 50 fixed tasks, invoke them multiple times under the same parameters, and calculate the compliance rate of the output structure and the consistency rate of key information. No complex evaluation framework is needed—a simple script will suffice:

# Pseudo-code: Stability sampling verification
tasks = load_fixed_tasks()          # Fixed task set
results = []
for _ in range(N):
    for task in tasks:
        out = model.invoke(task, temperature=0)
        results.append(check_schema(out))   # Verify structural compliance
consistency = sum(results) / len(results)

The focus should be on the structural compliance rate (whether it consistently returns a parsable format) and the key field hit rate (whether it consistently includes necessary information), rather than the literary quality of the answer.

2. Response Efficiency: Balancing Latency and Throughput

The impact of response speed on experience is non-linear. If first-token latency increases from 1 second to 3 seconds, the user perception is noticeable. If it increases from 3 seconds to 5 seconds, the perception dulls. Therefore, evaluation should distinguish between first-token latency and overall generation time, and determine which is more critical based on the business scenario: interactive scenarios prioritize first-token latency, while batch processing scenarios prioritize throughput.

It is worth noting that speed and quality do not always trade off against each other. Optimizations in inference scheduling by some models can improve both simultaneously. Therefore, “faster speed but lower quality” should not be accepted as a default reasonable trade-off, but rather treated as a hypothesis to be verified.

3. Instruction Following: Completion Under Complex Constraints

Prompts in real-world business scenarios often carry numerous constraints: output format, field ranges, tone requirements, and prohibitions. Models with weak instruction-following capabilities will frequently fail these constraints, leading to increased post-processing costs.

During evaluation, multi-constraint tasks can be designed. For example: “Output as JSON, include three specified fields, field values must be from a given enumeration, and no additional explanatory text is allowed.” Calculating the proportion of outputs that fully satisfy all constraints reflects engineering usability far better than single-dimensional accuracy.

4. Selection Practice: Embedding Evaluation into the Development Workflow

To implement the above ideas, the key is not to treat model evaluation as a one-time event, but to embed it into the continuous integration process.

• Establish a regression set: Sample from real business logs to construct a small regression set covering core scenarios, and run it automatically with every model version change.
• Set threshold alerts: Define acceptable ranges for stability, structural compliance rate, and average latency. Trigger alerts when boundaries are crossed, rather than relying on manual spot checks.
• Retain human review: Automated metrics are responsible for detecting degradation, while human review determines “whether the upgrade is worthwhile.” The two cannot replace each other.
• Account for migration costs: When changing model versions, the adjustment costs for prompt adaptation and output parsing logic are often underestimated. These should be factored into the total cost during evaluation.

5. Conclusion

The improvement of LLM capabilities is shifting from “visible leaps” to “invisible polishing.” At this pace, a gap between public noise and actual progress is inevitable. For developers, rather than chasing every leaderboard refresh, it is better to establish an evaluation system centered on business tasks, with stability and efficiency as observation points. Keeping up with benchmarks is sufficient; what truly determines a model’s value is whether it can perform reliably and predictably within your own pipeline. This framework does not rely on any specific product, which is exactly why it will remain applicable in the next round of version updates.