Evaluating Document Verification in Multimodal LLMs: What a Blurry Receipt Reveals

Why a Real-World Receipt Is a Useful Test

A common but often underestimated business task is handing a model a poorly photographed receipt and asking it to verify whether the amounts, quantities, and totals contain errors. The task seems straightforward, but it actually tests three capabilities at once: character recognition on low-quality images, table structure recovery, and numerical consistency reasoning.

After the release of a recent multimodal model, many developers immediately tested it on this kind of scenario. One typical observation was that, given the same shopping receipt photo where the subtotal of 28.9 was printed rather blurrily, different models produced different conclusions. Some could locate the blurry region and proactively flag uncertainty, while others simply returned “no errors.” The difference points to several distinct capability gaps in the document understanding pipeline.

This article does not evaluate or recommend any specific product. Instead, it treats the phenomenon as a general technical question: how should we systematically assess and use the document verification capabilities of multimodal large language models?

Breaking Down the Task: What It Takes to Verify a Receipt

When a natural language instruction like “check whether this receipt has errors” is given to a model, the following subtasks must be completed internally:

1. Text detection and recognition (OCR). Locate text lines in the image and recognize them character by character. Blur, skew, glare, and low resolution all significantly increase error rates.
2. Layout structure recovery. Identify the hierarchy of headers, item rows, and subtotal/total rows, and distinguish between “unit price x quantity” and “amount” columns. When table lines are missing, the model must rely on alignment and semantic clues.
3. Numerical extraction and normalization. Convert recognized digit strings into numeric values while handling noise such as decimal points, currency symbols, and thousands separators.
4. Consistency verification. Verify row by row that unit price x quantity equals amount, that the sum of line amounts equals the subtotal, and finally that subtotal plus tax equals the total.
5. Uncertainty expression. When character confidence is low, the model should proactively state that a region is blurry and the conclusion is uncertain, rather than forcing a definitive answer.

Step 5 is the most easily overlooked yet most telling indicator of engineering maturity. A reliable system is valuable not because it is always right, but because it knows where it might be wrong.

Why a Blurry Subtotal Is a Good Stress Test

The subtotal line is often the most critical verification anchor on a receipt for three reasons:

• It is an aggregated result. Once misread, it cascades into the total verification.
• It usually has the worst print quality. Receipt tails suffer from paper wear and faded thermal printing, making digits like 8/9/0 and 5/6 easy to confuse.
• It sits at a structural boundary between the item list and the total, so a small layout error can misplace it in the wrong row.

Testing a model with a blurry subtotal therefore tests its low-confidence handling strategy. Ideal behavior includes:

• Proactively pointing out that the digit is ambiguous.
• Offering candidate readings, such as 28.9 and 28.8, along with the verification outcome for each.
• Suggesting that the user double-check the original image or provide a clearer photo.

The failure mode is silently accepting a wrong reading and outputting “verified” based on it. Such confident errors are more dangerous in business systems than explicit failures.

A Reproducible Evaluation Methodology

If you want to establish an internal baseline for this type of capability, consider the following workflow.

Build a Layered Test Set

Level	Sample Characteristics	What It Tests
L1 Clean	Front-facing photo, even lighting	Basic OCR and structure recovery
L2 Mild degradation	Slight skew, mild blur	Robustness
L3 Severe degradation	Noticeable blur, partial occlusion, creases	Uncertainty expression
L4 Logic traps	Deliberately introduced arithmetic errors	Verification reasoning

L4 is especially important. Deliberately altering an amount on one line reveals whether the model can locate the specific row and explain the error type. OCR alone cannot pass this level.

Define Quantifiable Metrics

• Character-level accuracy. Character-by-character comparison, suitable for L1 and L2.
• Field-level accuracy. Whether key fields such as amounts and quantities are exactly correct.
• Verification conclusion accuracy. Whether the model correctly judges “error” or “no error,” and whether it can locate the error when one exists.
• Uncertainty recall. On artificially degraded samples, the proportion of cases where the model proactively declares uncertainty.
• Hallucination rate. The rate at which the model fabricates readings when the image information is genuinely missing.

Fix the Prompt and Output Schema

Controlled comparisons require consistent variables. The prompt should specify a structured output format, for example:

{
  "items": [{"name": "...", "qty": 1, "unit_price": 0.0, "amount": 0.0}],
  "subtotal": {"value": 0.0, "confidence": "high|low", "note": "..."},
  "checks": [{"rule": "qty*price==amount", "passed": true, "line": 1}],
  "conclusion": "ok|mismatch|uncertain"
}

A fixed schema enables automated comparison and avoids bias from manual reading. At the same time, the confidence and conclusion fields naturally guide the model to express uncertainty.

Prompt Engineering: Turning Verification into Executable Instructions

Most models perform inconsistently under open-ended prompts but improve noticeably with structured instructions. The following strategies are broadly applicable:

1. Make the steps explicit. Do not ask the model to judge correctness directly. Instead, require it to transcribe line by line, then verify line by line, and finally summarize. Chain-of-thought-style task decomposition greatly reduces skipped-step errors.
2. Provide the verification rules. Explicitly state formulas such as unit price x quantity = amount and sum of amounts = subtotal, turning implicit common sense into explicit constraints.
3. Require evidence annotations. Ask the model to attach the corresponding image region description or line number to each conclusion for easier manual review.
4. Define an uncertainty exit. Write in the instruction that if a field cannot be confirmed, the model should mark it uncertain and explain why. This gives the model a legitimate channel for abstention.
5. Cross-verify critical values. Require the model to check key numbers through different calculation paths, such as forward summation and reverse deduction from the total.

Engineering Integration: Putting the Model into a Pipeline

In real business scenarios, a single model call rarely handles verification on its own. A more robust architecture is a three-layer design combining models, rules, and human review:

• Layer 1: Rule engine as the safety net. Deterministic logic such as amount summation and tax calculation should be implemented in code first. The model only converts images into structured data.
• Layer 2: Model handles semantic and structural understanding. It processes cases that rules cannot easily cover, such as missing table lines, irregular field names, and multi-column layouts.
• Layer 3: Low-confidence cases go to human review. When the model outputs low confidence, or when the rule engine and the model disagree, the case automatically enters a manual review queue.

The core idea behind this division of labor is to let the model do what it is good at (perception and structuring), let code do what it is good at (precise calculation), and let humans make the final judgment. Any attempt to have the model handle everything end to end should be treated with caution in scenarios involving financial amounts.

Conclusion

Returning to the original observation: a blurry subtotal on a receipt reveals not just OCR accuracy, but a model’s attitude toward uncertainty. For developers, evaluating a multimodal capability should not be about winning a one-off comparison. It should be about building a layered, quantifiable, reproducible evaluation framework and treating honest uncertainty expression as a first-class metric.

The real basis for technology selection is never a stunning demo. It is whether the model behaves predictably on your test set, in your pipeline, and in your failure mode statistics. Solid evaluation matters more than chasing the latest release.