The “Fake Acceleration” in Model Evaluation: Starting from Input Caching

1. The Problem: Running Fast Does Not Mean Computing Fast

In model performance evaluation, a common phenomenon requires vigilance: the end-to-end latency of a benchmark test is significantly lower than historical records. The team might conclude that “inference performance has drastically improved,” only to find no improvement in throughput or latency in real-world business scenarios after deployment. This discrepancy is often not a measurement error but a structural flaw in the evaluation methodology itself—the measured “speed” might not involve any actual computation.

A typical scenario mentioned in original discussions is that when a benchmark dataset contains “a relatively fixed set of reused inputs,” the system might directly cache intermediate states or final outputs. Upon encountering the same input again, it skips actual computation. The total duration of the benchmark is significantly shortened, but this reduction has nothing to do with the general performance improvement of the GPU Kernel itself. This article abstracts this phenomenon as “evaluation escape” and analyzes it from three perspectives: mechanism, monitoring, and governance.

2. Mechanism Analysis: Why Caching Distorts Evaluations

2.1 The Value and Side Effects of Input Caching

Input caching (prompt/prefix caching) is a legitimate engineering optimization in itself. For highly repetitive prefixes like system prompts, few-shot examples, or long-document Q&A, caching their Key-Value (KV) states avoids redundant attention computation, thereby reducing Time-To-First-Token (TTFT) and increasing throughput. It is a strict necessity in production environments.

However, under the context of evaluation, this same mechanism produces side effects. To ensure reproducibility, benchmarks usually employ fixed datasets, fixed random seeds, and fixed batch sizes. If the evaluation framework or inference service applies result-level caching to “exact same inputs,” the actual execution path for the Nth run of the i-th sample might look like this:

Input x_i → Cache Hit → Directly return y_i
           (Skips forward computation, skips Kernel scheduling)

Consequently, the time consumed for the entire evaluation round is compressed to the overhead of “cache lookup + result assembly,” leaving the GPU nearly idle.

2.2 The Fundamental Difference Between Two Types of Caches

It is crucial to distinguish between two types of caches, as their implications for evaluation are entirely different:

Cache Type	Cached Object	Reduces Computation?	Implication for Evaluation
Prefix/Intermediate State Cache	KV states, activations	Partial reduction (prefix portion)	Legitimate engineering optimization, but must be explicitly declared in evaluation
Result-Level Cache	Final output	Completely skipped	Invalidates the evaluation, constitutes “escape”

If the former is enabled during evaluation, Kernels are still executing, albeit with a reduced computational load. The latter transforms the subject under test from “model inference” into a “lookup table.” The key to determining whether an evaluation is credible is checking if the tested path still includes actual forward computation.

2.3 Variants of Evaluation Escape

Beyond result caching, there are several other variants of evaluation escape that should be monitored at the engineering level:

• Implicit caching caused by sample deduplication: Duplicate samples exist within the evaluation set, or multiple sampling iterations hit the same input, triggering cache reuse.
• Fixed batch combination reuse: Consecutive evaluation rounds use identical batch partitioning, giving the scheduling layer an opportunity to reuse entire batch results.
• Evaluation script-level memoization: The evaluation code memoizes function return values. Although unrelated to the inference engine, it artificially inflates performance.
• Warm-up phase pollution: Cache hits during the warm-up round are counted into the official timing interval.

The common characteristic of these variants is that the actual number of Kernels executed during the timing interval does not match the theoretical value.

3. Monitorable vs. Unmonitorable: Where Lies the Difference

Original discussions raised a valuable question: why can some evaluation frameworks detect this type of escape while others cannot? The answer lies not in the model itself, but in the observability design of the evaluation pipeline.

3.1 Three Characteristics of a Monitorable System

1. Traceable Computation Path: The evaluation framework can log the actual operator sequence or Kernel call count executed for each sample and compare it with the theoretical computation volume. If the Kernel call count for a sample is zero while timing continues to advance, it can be identified as a cache hit.
2. Explicit Cache Toggle: Caching is exposed to the evaluator as a configurable option, defaulting to off or explicitly flagged. The evaluation report must annotate the caching strategy. This ensures the source of “acceleration” can be attributed.
3. Aligned Timing and Computation Scopes: The timing interval strictly covers forward computation. Cache lookups, result serialization, and network transmission are measured separately to prevent non-computational overhead from being mixed into performance metrics.

3.2 Typical Gaps in Unmonitorable Systems

Conversely, evaluation systems prone to “missed judgments” often exhibit the following gaps:

• Only collecting end-to-end wall-clock time without capturing Kernel-level or operator-level counts.
• Caching is enabled by default and not exposed externally, leaving evaluators unaware of whether hits occurred.
• The evaluation dataset is disconnected from production traffic distribution, with a high proportion of duplicate inputs that are not tracked.
• Lacking control experiments of “running the same input multiple times,” making it impossible to distinguish between first-time computation and subsequent hits.

In other words, the issue is not that “certain models are inherently monitorable while others are not.” Rather, it is about whether the evaluation infrastructure treats the computation process as a first-class citizen for observation. By baking observability into the evaluation framework, any model can be effectively monitored. Conversely, even the most powerful model will leave evaluation blind spots on an opaque pipeline.

3.3 A Minimal Viable Detection Approach

Below is illustrative pseudo-code demonstrating how to identify result-level cache hits within an evaluation loop:

def evaluate(model, dataset, cache_enabled=False):
    total_kernel_calls = 0
    total_time = 0.0
    for sample in dataset:
        before = model.kernel_call_counter()   # Read operator call count
        t0 = now()
        out = model.infer(sample.input)
        elapsed = now() - t0
        delta = model.kernel_call_counter() - before
        total_kernel_calls += delta
        total_time += elapsed
        if delta == 0:
            record_suspicious(sample.id, reason="zero_kernel_call")
    report = {
        "time": total_time,
        "kernel_calls": total_kernel_calls,
        "avg_kernel_per_sample": total_kernel_calls / len(dataset),
    }
    return report

The key takeaway is to record time and computation volume simultaneously. When the average Kernel calls per sample fall significantly below the theoretical baseline, the evaluation should be deemed untrustworthy, even if the latency metrics look excellent.

4. Engineering Governance: Returning Evaluation to “Computing”

4.1 Evaluation Design Level

• Disable result-level caching by default, and explicitly state the prefix caching strategy in the report.
• Construct control experiments: Run the same batch of inputs consecutively for two rounds and compare the latency and Kernel counts between them. The difference represents the cache benefit.
• Introduce deduplication and perturbation: Analyze the duplicate rate of the evaluation set, and apply harmless perturbations to inputs when necessary to break cache hit paths.
• Separate warm-up and timing: Handle the warm-up round separately and exclude it from official metrics.

4.2 Monitoring System Level

Fill the evaluation pipeline with three categories of observable data:

1. Operator-level counts: Kernel call counts, estimated FLOPs.
2. Cache metrics: Hit rate, hit source (prefix/result), proportion of hit samples.
3. Resource metrics: GPU utilization, memory bandwidth usage. GPU utilization is typically abnormally low during cache hits, serving as a cross-validation signal.

Feeding these three types of metrics into a unified monitoring, alerting, and logging service allows for the automatic generation of a “credibility score” upon completion of the evaluation task, triggering alerts for anomalously low computational density.

4.3 Reporting and Attribution Level

Performance reports should ensure “attributable acceleration”: every performance gain must be annotated with its source—whether it is Kernel optimization, parallel strategy adjustment, or a cache hit. Unattributable acceleration should be treated as suspicious by default. The wording of evaluation conclusions should also be more rigorous: distinguish between “end-to-end latency reduction” and “unit computation performance improvement.” The two must not be conflated.

5. Conclusion

“Model escape” is not a mysterious characteristic of a specific model, but an external manifestation of methodological flaws in evaluation. Input caching allows repeated inputs to bypass actual computation, creating a “fake acceleration” in benchmarks that is unrelated to Kernel performance. The ability to identify such phenomena depends on whether the evaluation system possesses three capabilities: traceable computation paths, explicitly controllable cache states, and aligned timing and computation scopes.

For developers and enterprises, the actionable steps are clear: simultaneously collect time and computation volume in the evaluation framework, disable result-level caching by default, establish control experiments and attribution reports, and integrate metrics like cache hit rates and GPU utilization into unified monitoring. Only by doing so will benchmark numbers truly correspond to “the model computes fast” rather than “the system looks up fast.”