Multimodal LLM Evaluation Guide: Image-Text Understanding and Traceable Citations

    This article explains how to evaluate multimodal models not only by answer correctness but also by whether their answers can be traced to evidence. It defines understanding accuracy, citation traceability, and verifiability; provides a 14-point scoring rubric with deduction rules and independent double scoring; and introduces task-template conventions for chart screenshots and other business materials.

    Why Evaluate for Traceable Evidence

    Multimodal models can already read charts, screenshots, and documents that mix text and images. The real concern for business teams has become whether an answer can be traced. If a conclusion cannot be linked to a specific value, paragraph, or region, reviewers cannot verify it, and therefore cannot use it for reimbursement, compliance decisions, or external reporting.

    For this reason, evaluation should not focus only on whether the answer is correct. This article splits evaluation into two dimensions and provides a scoring sheet and task templates that teams can apply directly. Teams can adjust scores and thresholds according to their own business materials.

    1. Decision Boundaries for Two Core Concepts

    Understanding accuracy means whether the model’s reading and reasoning about the input material are consistent with the source material. The basis for judgment can only be the material itself, not common sense or external knowledge.

    Citation traceability means whether the model points to specific evidence when giving a conclusion and whether it keeps the same evidence source across multiple rounds of follow-up questions.

    Verifiability means that a reviewer does not need to read all the material to quickly locate the position on which an answer is based and confirm its content.

    When judging, distinguish two types of errors. If the answer is correct but the citation is wrong, record it as “unverifiable evidence.” If the citation is correct but the answer is misread, record it as “understanding error.” Both types weaken business credibility, so they are scored separately in the scoring sheet.

    2. Scoring Sheet and Scoring Rules

    Each question has six scoring items: three belong to understanding accuracy and three to citation traceability. The table below gives initial point values. Teams can adjust them after the first test round based on business risk.

    Scoring Item Dimension Points Scoring Criteria
    Key field extraction Understanding accuracy 0–2 2: all numbers, names, units, and time conditions are correct; 1: one item is missing or the unit is incorrect; 0: a key value is incorrect
    Distinguishing similar information Understanding accuracy 0–2 2: parallel metrics are not confused; 1: confusion occurs but is self-corrected; 0: information is attributed to the wrong item and not corrected
    Complex layout parsing Understanding accuracy 0–2 2: headings, body text, and notes are correctly distinguished; 1: individual regions are assigned incorrectly; 0: the overall structure is confused
    Citation specificity Citation traceability 0–3 3: page number or region is given with a direct quote from the source; 2: page number or region is given; 1: only “according to the information in the figure” is stated; 0: no citation
    Evidence consistency Citation traceability 0–3 3: the same source is cited across three question phrasings; 2: one switch occurs, and the source after the switch is still correct; 1: the source after the switch is incorrect; 0: all three sources differ
    Supporting evidence selection Citation traceability 0–2 2: the cited information exactly supports the conclusion; 1: the citation is relevant but insufficient; 0: a distractor is cited

    The total full score is 14. Two deduction rules are also applied:

    • Value-level errors include misreading data, incorrect digit placement, incorrect step order, omitted conditions, and introducing rules that are not in the material. Each occurrence deducts 2 points from the total score for that question.
    • Trend-level errors refer to incorrect judgment of the direction of a trend or of an approximate change. Each occurrence deducts 1 point from the total score for that question.

    The pass standard for a single question is: total score of at least 12 and no value-level errors.

    Scoring uses two independent reviewers. If the score difference between the two exceeds 1 point, a third person adjudicates and records the reason for the decision. This keeps subjective judgment within an explainable range.

    3. Task Templates for Three Types of Materials

    The example data below are fictional and are used only to illustrate how tasks should be written. Teams should replace them with their own de-identified materials before use.