Evaluating Multimodal LLMs on Document Verification: What a Blurry Receipt Reveals

1. Introduction: Why a Blurry Receipt Is Worth Taking Seriously

Not long ago, a developer shared a simple comparative experiment in an online community: they sent the same phone photo of a shopping receipt to two multimodal large language models (a newly released version and an established earlier version) with the same instruction — “check whether the receipt contains any errors.” One printed value on the receipt, a subtotal of 28.9, came out blurry. That degraded region turned out to be exactly what separated the real capability of the two models.

The value of this case lies not in its conclusion but in what it reveals — a widely underestimated fact: document verification is one of the scenarios where a multimodal LLM’s weaknesses are most easily masked by benchmark scores. Clean scanned documents and neatly formatted tables in public benchmarks are distributed very differently from the casually photographed, blur-printed, unevenly lit receipts found in real business settings. A high score on the former guarantees nothing about usability on the latter.

So this article will not rehash “which model sees better.” Instead, it abstracts that informal experiment into a reproducible, quantifiable, and attributable evaluation and engineering methodology — one developers can run themselves whenever a new model is released, rather than relying on marketing claims.

2. What Receipt Verification Actually Requires: Three Coupled Capability Layers

Many people treat receipt checking as an OCR problem. That is the first misconception. It is really a task that couples three layers:

1. Perception (recognition): extracting characters from a low-quality image. Blur, noise, low resolution, and perspective distortion all break stroke connectivity; after degradation, digits such as 8/6/9/3, decimal points, and commas become easily confused.
2. Structure (field extraction): organizing scattered text into structured records of item, unit price, quantity, and amount, and distinguishing semantic fields such as subtotal, discount, tax, and total due — regardless of layout differences.
3. Reasoning (consistency checking): summing the item amounts, comparing the result with the printed subtotal, and judging whether they agree. This requires stable numerical computation and reliable instruction following.

The key point: a failure at any one layer makes the final conclusion wrong, and the error is hard to attribute. That is exactly why a blurry region such as “subtotal 28.9” is valuable — it propagates uncertainty from the perception layer all the way to the end of the reasoning chain, exposing problems in all three layers within a single test.

3. Designing the Evaluation: From Controlled Variables to a Metric System

3.1 Controlled Variables

The original experiment did one thing right: the same photo and the same instruction sent to both models — a controlled-variable setup. To make it rigorous, four additions are needed:

• Fixed decoding parameters: set temperature to 0 or a very low value to rule out sampling randomness;
• Repeated runs: run each case at least 3 times and observe output stability. A model whose conclusions flip back and forth on identical input cannot be trusted in production;
• Ground-truth annotation: manually verify the correct text and value of every field to serve as the scoring baseline;
• A degradation gradient: instead of testing only the original image, construct a gradient — original → light blur → heavy blur → downsampling → compression artifacts — and observe the decay curve of capability as image quality drops. Two models that tie on the original image and diverge under heavy blur is the norm, not the exception.

3.2 Metrics

Metric	Definition	What it measures
Field-level accuracy	Correctly recognized fields / total fields	Perception
Numeric consistency pass rate	Share of cases where the summed total matches the printed subtotal	Reasoning
Uncertainty expression rate	Share of fields the model proactively flags as “null / needs human review” when it cannot confirm them	Calibration and honesty
Hallucination rate	Share of blurry fields the model fills in with fabricated values	Risk control

3.3 The Most Important Criterion: Behavior in Degraded Regions

For a degraded region such as the “subtotal 28.9,” model behavior falls into four categories:

• A: Recognizes it correctly and reaches the correct verification conclusion — ideal behavior;
• B: Misreads 28.9 as 28.0, 26.9, or similar, and on that basis reports that “the receipt contains an error” — a perception hallucination, made more dangerous because it is delivered in the form of a verification report;
• C: Recognizes the value correctly but sums the items incorrectly, producing a false alarm or a missed one — a reasoning defect;
• D: Proactively states that “this region is blurred, the value cannot be confirmed, human review recommended” — a qualified expression of uncertainty.

One point deserves emphasis: D is not an escape. In verification work, an honest “I can’t read this” is far more valuable than a confident wrong value. The former costs one human-review ticket; the latter can cost an actual reconciliation discrepancy. This is the most overlooked dimension in evaluation — and the one that best separates mature models from the rest.

4. Sample Code: A Reproducible Evaluation Scaffold

# Pseudocode: a document-verification evaluation scaffold.
# All functions are illustrative and must be implemented for your chosen inference framework.
PROMPT = """Identify the item name, quantity, and amount of every line on this
shopping receipt and return them as JSON:
{"items":[{"name":"","qty":0,"amount":0.0}],"subtotal":0.0,"conf_flags":[]}
If a field cannot be confirmed because the printing is blurry, set it to null
and list it in conf_flags; finally check whether the sum of the item amounts
equals the subtotal and output "check":"pass/fail/uncertain"."""
def degrade(img):
    """Build a degradation gradient: original / light blur / heavy blur /
    downsampling / compression artifacts"""
    return {
        "raw":           img,
        "blur_light":    gaussian_blur(img, sigma=1.0),
        "blur_heavy":    gaussian_blur(img, sigma=3.0),
        "downsample":    resize_back(resize(img, scale=0.33)),
        "jpeg_artifact": recompress(img, quality=40),
    }
def evaluate(model, cases, runs=3):
    rows = []
    for case in cases:                        # cases carry manually annotated ground truth
        for level, img in degrade(case.image).items():
            outs = [model(img, PROMPT) for _ in range(runs)]   # low temperature, repeated sampling
            rows.append({
                "case": case.id, "level": level,
                "field_acc":  field_accuracy(outs, case.truth),  # field-level accuracy
                "sum_check":  sum_consistency(outs, case.truth), # numeric consistency
                "null_rate":  uncertainty_rate(outs),            # uncertainty expression rate
                "stability":  consistency(outs),                 # output stability
            })
    return rows

Three engineering details deserve attention:

• Enforce JSON output: it makes verification conclusions programmatically parseable, instead of unscoreable natural-language answers such as “it looks basically fine to me”;
• Give the model an explicit exit: the prompt allows null values with an annotation, so the model has a legitimate way to express uncertainty instead of being forced to guess;
• A three-level conclusion: pass / fail / uncertain frees “the model is not sure” from a binary classification and connects naturally to a human-review queue in business workflows.

5. Failure Modes and Attribution: Assign Blame to the Right Layer

Once you have a poor result, the next step is attribution — otherwise there is nothing to improve. A simple ablation is recommended: hand the model the ground-truth text of the receipt as plain text and repeat the same verification task.

• If plain-text verification succeeds but image verification fails → the perception layer is at fault. The fix lies in image preprocessing or a stronger recognition approach;
• If plain-text verification also fails → the reasoning layer is at fault. The fix lies in prompt engineering (for example, guiding step-by-step calculation), tool calling for deterministic arithmetic, or switching models;
• If the conclusion keeps flipping → stability is at fault. Check the decoding parameters and, if necessary, converge with majority voting.

Wrong attribution leads to wrong optimization. That is the root cause of teams that spend months tuning prompts while the problem never goes away.

6. Production Engineering: From “Sharp Eyes” to Deployable Systems

6.1 Image Preprocessing

Conventional enhancement before the image reaches the model often pays off more than switching models: grayscale conversion and contrast stretching help with faint printing; adaptive binarization suppresses uneven lighting; perspective and deskew correction restores row and column structure; targeted super-resolution can be layered on top when needed. The principle: restore information at the pixel layer first, then let the model consume it.

6.2 Prompt Engineering

• Ask for structured output rather than free-form narration;
• State explicitly that uncertain fields must be flagged, and forbid fabrication;
• For amount fields, require the original precision to avoid rounding-induced false discrepancies;
• In large-batch scenarios, run controlled prompt experiments on a small sample first, then lock in the template.

6.3 Cross-Checking and Human-in-the-Loop Review

High-value scenarios can adopt dual-channel cross-checking: if two independent recognition results agree, the case passes automatically; if they disagree, or either channel returns uncertain, the case goes to a human-review queue. Combined with confidence thresholds that dynamically adjust the review ratio, this balances accuracy against labor cost. It is a mature practice in financial reconciliation and document review.

6.4 Combining LLMs with Dedicated Text Recognition Services

For high-throughput, strongly deterministic production scenarios, a common two-stage architecture in industry is: a dedicated recognition service handles transcription, and the large model handles understanding. Cloud providers such as Baidu AI Cloud offer general-purpose text recognition services that can serve as a high-precision character transcription channel, while the multimodal LLM takes on field structuring, business-rule verification, and the adjudication of ambiguous results. This combination is often better than pushing the entire workload through an end-to-end LLM in cost, latency, and accuracy alike. The two routes are not mutually exclusive — combine them according to each scenario’s throughput and fault-tolerance requirements.

7. Conclusion: Turn “Launch Highlights” into Your Own Conclusions

Back to the opening experiment: one blurry receipt and two identical instructions. Casual as it looks, it touches the three core questions of multimodal model evaluation — perception robustness under degraded input, the quality of coupling between perception and reasoning, and honesty in the face of uncertain information.

Whenever a new model is released, the official highlights are always “improvements across the board.” For developers, a more reliable approach exists: prepare a small batch of samples that match your own business distribution — even a dozen real receipts will do — run them through the degradation gradient and metric matrix described here, and you can reach your own conclusion within half a day. Differences in model capability are ultimately revealed by experiments you design yourself. That is the whole point of evaluation, and the most basic common sense of engineering model selection.