Evaluating Vision LLMs for Receipt Recognition: Technical Insights and Best Practices

I. A Real-World Scenario: Why Blurry Receipts Are Hard to Recognize

When evaluating the capabilities of Vision Large Language Models (VLMs), shopping receipts serve as a highly representative sample. They appear simple but simultaneously present multiple recognition challenges:

• Unstable print quality: Numbers may suffer from missing pixels, uneven ink, or overlapping strokes.
• Uncontrollable shooting conditions: Photos may be tilted, suffer from glare, shadows, or perspective distortion.
• Compact layout: Amounts, quantities, and unit prices are densely packed.
• Minimal differences between numbers: For instance, “28.9” and “28.8” are easily confused when blurry.
• Varying clarity within the same receipt: Different regions of a single receipt may have different clarity, requiring the model to possess local reasoning capabilities.

In a recent experiment, the same photo of a shopping receipt taken by a mobile phone was fed into two different VLMs for verification. The “Subtotal: 28.9” on the receipt happened to be quite blurry, becoming the critical point for distinguishing the fine-grained recognition capabilities of the models. One model performed better on the subtotal amount by cross-verifying it with the context and item details, rather than relying solely on single-pixel recognition.

This phenomenon indicates that in receipt recognition scenarios, simply comparing whether a model “supports image input” is meaningless. The true focus should be on the model’s comprehensive performance under low-quality images, blurry numbers, and contextual reasoning.

II. Five Core Dimensions for Evaluating VLM Receipt Recognition

If developers need to integrate VLMs into business systems to process receipts, forms, or vouchers, it is recommended to systematically evaluate them across the following five dimensions, rather than drawing conclusions based on one or two demos.

1. Robustness to Low-Quality Images

In real-world business operations, receipt images are rarely scanned documents; they are usually photos taken with mobile phones. Shooting angles, lighting, and focus all affect recognition accuracy. An excellent model should possess the following capabilities:

• A certain tolerance for slight blurriness, noise, and shadows;
• The ability to recognize low-contrast text, especially in numerical regions;
• The ability to avoid directly outputting incorrect results when local areas are blurry, instead attempting to reason based on context;
• The ability to distinguish between “illegible” and “non-existent,” preventing hallucinated completions.

Taking “Subtotal: 28.9” as an example, if the number “9” appears similar to an “8” due to printing or shooting issues, a model that references the cumulative result of the item details above has a chance to correct the single-point misjudgment. This cross-regional reasoning capability is more critical than simply increasing image resolution.

2. Precise Recognition of Numbers and Amounts

The core value of receipt recognition lies in numerical accuracy. Occasionally misrecognizing a character in a name or address may have limited impact, but an error in amounts, quantities, tax rates, or tracking numbers can directly cause financial or compliance risks.

When evaluating, it is recommended to construct a test set containing the following scenarios:

• A mix of numbers with one and two decimal places;
• Highly confusable character combinations like “0, 6, 8, 9”;
• Incomplete numbers caused by printhead missing pins;
• Multi-line amounts aligned together, requiring row-by-row correspondence;
• A mix of handwritten and printed numbers.

A practical approach is to measure “field-level accuracy” rather than “whole-receipt pass rate.” For example, if a receipt contains 20 numerical fields, missing 1 field is fundamentally different from missing 3. The former might be resolved via secondary verification, while the latter likely requires manual review of the entire order.

3. Structured Output and Field Alignment

Recognizing text is only the first step. Whether the recognition results can be organized into structured data directly impacts the efficiency of subsequent system integration.

Ideally, the model should output a structure similar to the following:

{
  "items": [
    {"name": "Bottled Water", "quantity": 1, "unit_price": 2.5, "amount": 2.5},
    {"name": "Bread", "quantity": 2, "unit_price": 13.2, "amount": 26.4}
  ],
  "subtotal": 28.9,
  "tax": 0.0,
  "total": 28.9
}

During evaluation, check not only whether the JSON format is valid but also:

• Whether item names are correctly aligned with their corresponding amounts;
• Whether the mathematical relationships between subtotal, discount, tax, and total are self-consistent;
• Whether field misalignment occurs in multi-page receipts or long item lists;
• Whether there are duplicate items, missing items, or incorrect merges.

If the model only outputs free text, even if the content is correct, additional parsing steps are required during engineering implementation, increasing pipeline fragility.

4. Reasoning and Error Correction Capabilities

This was the most noteworthy difference in our experiment. For blurry regions, a basic approach directly outputs the character with the highest confidence; an advanced approach incorporates global image information for logical verification.

For example:

• If the subtotal is blurry but the prices of individual items are clear, the model can verify by accumulation;
• If the discount amount is blurry but the original and discounted prices are clear, the model can calculate the discount;
• If the tax rate is blurry but the tax-inclusive and tax-exclusive prices are known, the model can compute the tax rate.

This capability of “cross-field verification within the receipt” transforms the model from a mere perception tool into a verification assistant with business common sense. When evaluating, developers can intentionally obscure or blur certain fields to observe if the model can infer reasonable results based on other fields.

5. Prompt Friendliness and Task Adherence

The same model can produce significantly different results with different prompts. During evaluation, test whether the model can:

• Strictly output in the specified field format;
• Flag uncertain items as “pending manual review” instead of forcing an answer;
• Adhere to the “extract only, do not modify” principle, refraining from arbitrarily correcting content on the receipt that the model should not judge;
• Support batch processing instructions, such as recognizing multiple receipts at once and summarizing them.

Below is a reference prompt template:

Please recognize the shopping receipt in the image and extract the item details, subtotal, discount, tax, and total.
Requirements:
1. Extract only information explicitly visible on the receipt;
2. For blurry or uncertain numbers, mark them as "uncertain" and append the most likely value;
3. Output in JSON format with fields including items, subtotal, discount, tax, total;
4. Perform arithmetic verification for each amount. If inconsistencies are found, explain them in the summary.

III. Why “Better Visual Perception” Matters in Engineering

The “slightly better visual perception” observed in the experiment is essentially not a pure pixel-level recognition difference, but a comprehensive advantage in the following aspects:

• Local perception: Maintaining optimal character candidates even in blurry regions;
• Context utilization: Treating the entire receipt as a whole rather than recognizing each field in isolation;
• Numerical reasoning: Forming a closed loop between recognition and verification, reducing basic errors;
• Task stability: Less output fluctuation under varying shooting conditions.

For engineering teams, these differences mean:

• A reduced rate of secondary manual review;
• Lowered business risks caused by single-point numerical errors;
• More stable operation of automated workflows in real-world environments;
• Improved end-to-end processing efficiency, especially in high-volume daily processing scenarios.

In actual business operations, receipt recognition is rarely an isolated single model call; it is embedded in processes like expense reimbursement, financial reconciliation, inventory management, and order verification. For every percentage point increase in front-end recognition accuracy, manual processing costs downstream may decrease by several percentage points. Therefore, differences that “look just slightly better” are significantly amplified in large-scale operations.

IV. Recommended Architecture for Production Deployment

To build a reliable receipt recognition service in a production environment, it is recommended not to rely solely on a single model call. Instead, design a pipeline that includes validation and fallback mechanisms.

Image Input
  │
  ▼
Image Preprocessing (Orientation correction, denoising, contrast enhancement)
  │
  ▼
VLM Recognition (Output structured JSON)
  │
  ▼
Rule Engine Validation (Amount reconciliation, field completeness, anomaly flagging)
  │
  ├── Validation Passed ──► Write to Business System
  │
  └── Validation Failed ──► Manual Review Queue

Here, the rule engine can be independent of the model, implementing validations such as:

• Whether the sum of item amounts equals the subtotal;
• Whether the subtotal minus discount plus tax equals the total;
• Whether mandatory fields exist;
• Whether there are values obviously outside a reasonable range;
• Whether duplicate order numbers exist within the same batch of receipts.

This combination of “model recognition + rule-based fallback + manual queue” is more realistic and easier to maintain than striving for a single model to achieve perfect accuracy.

V. Advice and Summary for Developers

Returning to the original question: What does it mean if a VLM performs better in shopping receipt recognition?

From a single test, it only shows that it has superior numerical recognition and reasoning capabilities on a specific sample. From an engineering perspective, however, it prompts us to establish a more systematic evaluation methodology. Developers are advised to follow these steps during model selection:

1. Build a test set covering real-world scenarios, including blurry, low-light, tilted, and multi-page samples;
2. Use field-level accuracy and structured output quality as primary metrics;
3. Focus on how the model handles uncertain content and whether it is suitable for integrating into manual review workflows;
4. Design necessary image preprocessing and rule validation steps outside the model;
5. Conduct regular regression testing to observe the actual impact of model version updates on business metrics.

The value of VLMs in the receipt recognition domain lies not in “appearing smart,” but in the ability to integrate into real business workflows stably, accurately, and verifiably. For developers, translating the difference in judging a single blurry number into a measurable, optimizable recognition system is the true key to landing this technology.