Beyond Benchmarks: A Practical Guide to LLM Evaluation and Model Selection

Every few weeks, a new model launches with impressive benchmark numbers and quickly dominates community attention. A few days later, the narrative often reverses: real-world testing falls short of the marketing claims, and the model’s reputation drops just as fast. The recurring cycle of “peak at launch, disappointment in practice” is not mysterious. It can be explained through a set of engineering factors: differences in evaluation scope, task distribution, cost models, and platform constraints all contribute to the gap between official narratives and actual user experience.

The typical gaps fall into four areas: capability attribution (demo results vs. native model ability), task structure (single-turn Q&A vs. long-horizon delivery), statistical definitions (how hallucination rates are calculated), and cost definitions (per-token price vs. per-task cost)—plus platform constraints such as quotas and rate limits. This article breaks down the technical causes behind each gap and provides a practical framework for building your own evaluation system.

1. What Demo Results Really Represent: Separating Model Capability from System Capability

Modern LLM products are no longer bare models. They are complete systems composed of a model plus tools, skills, and engineering wrappers. An impressive demo—for example, a high-quality 3D content generation—may not reflect the model’s native ability. It may come from a built-in skill that calls an external specialized service behind the scenes. Viewers see the result, not the call chain.

This creates a classic evaluation trap: treating a system score as a model score. What you think is an evaluation of the model is actually an evaluation of “model × external tools × prompt engineering.” If you disable those skills during your own usage, or if the external service becomes unavailable or irreplaceable, the experience degrades sharply—and the purchasing decision based on the demo becomes unreliable.

The core method for identifying this gap is variable control:

1. Disable all external tools and skills. Allow the model to produce only native output or intermediate representations (such as directly writing modeling code and scene descriptions), then compare it with the tool-enabled version;
2. Check call-trace logs to confirm whether requests to external endpoints occurred during generation;
3. Quantify the quality difference between the two results. The larger the gap, the more the demo depends on the toolchain.

Here is a simplified pseudocode snippet for isolating tools and logging call behavior during evaluation:

# Evaluation framework example: isolate tools and log call traces
run_case(model, task, config={"tools": "disabled"})
trace = get_call_trace(model, task)
external_calls = [c for c in trace if c.target not in allowlist]
log("native_only_quality", score(output))
log("external_dependency", len(external_calls))

If the difference between “native-only capability” and “skill-enabled output” is large, the two should be tracked separately: the model score belongs to the model, and the tool score belongs to the tool. You should also evaluate whether the external tool is replaceable and portable. Otherwise, your business may become locked into a component you cannot control.

2. Complex Generation Tasks: Evaluation Must Reflect Real Workloads

Another common gap appears in complex generation tasks. Static benchmarks mostly measure single-turn Q&A, short code generation, and similar one-step abilities. Real engineering tasks are long-horizon: given several reference images, the model is asked to reproduce an interactive 3D object model. This requires multiple iterations, self-correction, and spatial reasoning. A typical community test follows this pattern: the first output is rough and clearly under-delivered; after the user points out the problem, the model acknowledges the mistake and starts revising; just as the revision is about to succeed, the process hits a platform time window or quota limit, and the task is forcibly interrupted.

Two technical signals are worth extracting here.

First, proactive verification is a hallmark of stronger models. When performing spatial tasks, better models actively measure coordinates, check relative positions, and run code self-checks rather than producing output based on intuition. Passive, response-only models tend to keep making mistakes on such tasks even if they have more parameters. Evaluation should therefore record “whether the model proactively verifies” and “whether the verification method is correct” as process metrics. These signals predict real-world performance better than a final screenshot.

Second, iteration budget determines task success. Long tasks naturally require multiple rounds of calls. If rate limiting or quota design does not leave room for long tasks, even a highly capable model will be cut short by engineering constraints. During evaluation, you must provide enough iteration rounds and record the “completion rate within N rounds.” This is closer to real-world experience than single-turn accuracy.

In practice, it is recommended to divide the evaluation set into three levels:

• L1 Single-point capability: single-turn Q&A, short code generation. Verifies baseline intelligence;
• L2 Multi-turn iteration: tasks with feedback and correction. Verifies self-correction and planning ability;
• L3 End-to-end delivery: complete tasks from requirement description to runnable artifact. Verifies integrated engineering capability.

Business decisions should rely mainly on L2 and L3 results, because user patience is measured by the final deliverable, not by single-turn accuracy.

3. Hallucination Rate: Without a Unified Definition, Numbers Are Not Comparable

Hallucination rate is another problem area. You may have seen this split: an official announcement claims a dramatic reduction in hallucination rate, but the footnote reveals that the measurement was taken with an external fact-checking loop attached. Meanwhile, community users testing the same questions on a bare model get numbers several times higher. Both numbers are “true,” but the definitions are completely different. Comparing them directly is meaningless.

At least three definitional variables cause this divergence:

1. Different definitions: Is it the factual error rate, citation source error rate, or instruction violation rate? Numbers under different definitions can differ widely;
2. Different system configurations: A bare model, a model with retrieval augmentation, and a model with a fact-checking loop are three different systems. Retrieval and fact-checking can significantly reduce externally delivered errors, but that is a contribution of the system, not proof that the model itself has improved;
3. Different judgment methods: Automated scoring scripts and human annotation may produce inconsistent results. Question distribution, difficulty, and domain sampling strategies also significantly affect the outcome.

The correct approach is to align the definitions before comparing: the same set of questions, the same judgment script, and the same system configuration, measuring separately the “bare model hallucination rate” and the “hallucination rate with fact-checking attached.” The former represents the model’s underlying capability; the latter represents delivery quality. You need both numbers: the foundation determines the ceiling, and delivery determines the experience.

4. Cost Accounting: A Lower Unit Price Does Not Mean Cheaper Usage

In a price war, the easiest mistake is treating per-token price as per-task cost. Reasoning-oriented models perform long internal reasoning before giving an answer. In Agent mode, they also decompose subtasks, make multiple rounds of calls, and retry after failures. Actual token consumption can far exceed the length of the final answer. This can lead to a situation where Model A has a lower output price per token, but longer reasoning time per task and more calling rounds make the actual billed amount less favorable. Model B has a higher unit price, but completes the task in one pass, making total cost equal or even lower.

Cost evaluation must therefore shift from a per-token view to a per-task view. Here is a simplified estimation function:

# Per-task cost estimation example
def task_cost(calls):
    total = 0
    for r in calls:                    # r: one call record
        total += r.input_tokens * price_in
        total += r.output_tokens * price_out
        total += r.cached_tokens * price_cache
    total *= (1 + retry_rate)          # Account for retry loss
    return total / success_rate        # Amortize over successful tasks

Besides explicit fees, two hidden costs should be included. The first is the sunk cost of unfinished tasks. When a long task is cut off by quota limits, the tokens already consumed are pure loss. The second is latency cost. If single-task reasoning time doubles, it means lost human waiting time and pipeline throughput. During model selection, it is recommended to run billing comparisons on three to five real business tasks and obtain the actual billed cost per task before drawing conclusions.

5. Quotas and Rate Limits: Stress Testing Before Production Deployment

Quotas and rate limits are the final piece that is easy to overlook. Platform rate limiting usually has multiple dimensions: requests per minute, tokens per minute, concurrency, daily quota, and time-window-based quota pools. Short tasks barely notice these constraints, but long tasks requiring dozens of call rounds can be cut off by any one of them. In addition, if the Agent’s subtask decomposition logic has redundancy—such as repeated planning or ineffective retries—quota consumption can be multiplied.

Before deployment, do at least three things:

1. Stress testing: Run the longest real task through the entire process and record at which round and which dimension the limit is triggered;
2. Checkpoint and resume: Design a checkpoint mechanism for long tasks so that after interruption, the process can resume from the latest state instead of starting over;
3. Backoff and degradation: Use exponential backoff retries for rate-limit errors, and prepare a degradation path, such as switching to a smaller model to complete the final steps.

At the same time, include the transparency of rate-limit policies in the selection evaluation. Whether policies are clearly disclosed and whether adjustments come with advance notice or compensation mechanisms directly determines the stability of upper-layer applications.

6. Building Evaluation into a System: A Practical Checklist

The methods above can be consolidated into a sustainable evaluation workflow:

1. Task library construction: Sample and anonymize real business tasks. Build a private evaluation set covering L1/L2/L3, and rotate questions regularly to prevent overfitting;
2. Variable control: Fix prompt templates, sampling parameters, and tool configurations. Change only one variable at a time;
3. Dual-track scoring: Run automated metrics (pass rate, compilation pass rate, test pass rate) in parallel with blind human evaluation (scores given without knowing the model name) to avoid brand bias;
4. Process instrumentation: Record token consumption per round, call duration, retry count, tool calls, and external dependencies;
5. Composite score model: Quality score × completion rate ÷ per-task cost. Avoid the misleading effect of optimizing a single dimension;
6. Regression mechanism: Automatically run regression evaluation when a model or platform version upgrades to prevent “upgrade as regression.”

Engineering-wise, this instrumentation and statistics collection does not need to be built from scratch. Cloud platforms such as Baidu AI Cloud generally provide call logs, usage statistics, monitoring, and alerting capabilities for their LLM services. These can automate the collection of token consumption, failure rate, and latency, allowing evaluation data to accumulate continuously instead of remaining a one-off manual test.

Conclusion

Benchmarks are an entry ticket, not a report card. Demos can depend on toolchains. Hallucination rates can depend on fact-checking loops. A lower unit price can lose to longer reasoning time. Even the most impressive benchmark scores can be blocked at the delivery gate by rate limits and quotas. The essence of evaluation is to re-score a model using your tasks, your definitions, and your cost model. Once this system is in place, no matter how the market promotes a model, you will be able to answer the most basic question: does it actually work well in my business?