Multimodal LLM Evaluation: Image-Text Understanding and Citation Traceability

    This article presents a practical evaluation framework for multimodal large language models, focusing on two capabilities: accurately understanding charts, screenshots, process diagrams, and mixed-layout documents, and linking conclusions to specific evidence in the source material. It explains how to design test cases, assess evidence consistency, identify unsupported answers, and build a repeatable regression benchmark for real-world workflows such as document processing, learning material organization, compliance review, and auditing.

    Multimodal LLM Evaluation: Image-Text Understanding and Citation Traceability

    1. Why Evaluation Should Focus on Verifiability

    Over the past year, discussions about multimodal large language models have gradually shifted from “Can the model understand images?” to “Can its conclusions be verified after it understands them?”

    In real-world scenarios such as office document processing, study material organization, and compliance review, users are often less concerned that a model may produce an imperfect answer than that it may provide an answer that sounds reasonable but has no supporting evidence. This kind of “correct-looking but untraceable” response creates significant risks in compliance, risk management, and auditing.

    This article proposes an evaluation framework centered on practical verifiability, with two primary dimensions:

    1. Image-text understanding accuracy: Can the model consistently extract key information from mixed inputs such as charts, screenshots, and process diagrams?
    2. Citation traceability: When asked to provide supporting evidence, can the model anchor its conclusions to specific locations in the source material?

    In simple terms, accurate understanding addresses whether the model gets the answer right, while citation traceability explains how the answer was derived. Both are essential.

    2. Breaking Down the Evaluation Goals: From Understanding to Evidence

    To make evaluation results comparable and reproducible, divide the goals into executable tasks and define clear scoring criteria for each task.

    1. What to Evaluate in Image-Text Understanding

    For typical materials such as charts, screenshots, annotated images, and process diagrams, focus on the following behaviors:

    • Key field extraction: Does the model correctly read numbers, names, units, and conditions? Special attention should be paid to error-prone details such as decimal points, percentages, and time ranges.
    • Distinguishing similar information: When an image contains multiple similar entries, such as data for the same metric across different years, does the model confuse them?
    • Robustness to low-quality inputs: When the resolution is low, text is dense, or the layout is tilted, does the model provide a reasonable interpretation or begin inventing content?

    For scoring, classify errors into three levels:

    • Critical field errors: Severe errors involving key information;
    • Minor omissions: Moderate errors involving secondary information;
    • Redundant wording without substantive errors: Minor issues that do not affect the correctness of the answer.

    This classification makes it easier to compare different models or model versions.

    2. What to Evaluate in Citation Traceability

    This dimension measures the quality of the evidence chain:

    • Specificity of the citation: Does the model provide only a verbal summary, or can it point to a specific paragraph, image region, or table row?
    • Consistency across follow-up questions: After additional questions are asked, does the model continue to rely on the same evidence source, or does it quietly switch to another basis?
    • Selection among multiple candidate sources: When several parts of an image could potentially support a conclusion, can the model identify the actually relevant one?
    • Refusal and uncertainty handling: When the image does not contain supporting information, does the model clearly say that the information is not provided, or does it force an unsupported inference?

    3. Testing Image-Text Understanding with Three Typical Material Types

    1. Chart Screenshots

    Select screenshots containing bar charts, line charts, legends, and axes. Example questions may ask the model to:

    • Read the overall trend;
    • Provide a key numerical range;
    • Explain a possible reason for a change, using only information shown in the image.

    In practice, mainstream multimodal models are generally more reliable when the information is concentrated and clearly organized. They can often connect “value,” “meaning,” and “conclusion” into a coherent explanation.

    However, numerical reading errors increase noticeably when chart legends use similar colors, when dual axes are present, or when many data points are densely arranged. The key scoring criterion should be whether the reported range falls within a reasonable error margin of the actual reading, rather than whether every data point is reproduced with perfect precision.

    2. Process and Rule Diagrams

    Use process diagrams with multiple nodes and explanatory text attached to each step. Focus on two questions:

    • Does the model preserve the correct order of steps?
    • Does it omit any critical branching condition?

    An effective testing technique is to add an explicit constraint: “Use only the information shown in the image. Do not make any external inferences.”

    Under this constraint, stronger models typically begin by restating the process structure—including the starting point, branches, and endpoints—and then explain the meaning of each node. Weaker models are more likely to introduce general knowledge or steps that do not appear in the diagram.

    Evaluators should treat the addition of conditions not present in the image as a substantive error. This is a typical source of answers that appear plausible but cannot be verified.

    3. Mixed-Layout Document Pages

    Examples include excerpts from policies, product documentation, and annotated contract pages. These materials place greater demands on the model’s understanding of page structure. The model must distinguish among headings, body text, notes, and footers, and understand positional meaning—for example, that a note applies only to the preceding clause.

    Scoring can begin with structural reconstruction. Ask the model to organize its response according to the hierarchy of the original text, then compare the result with a manually labeled layout structure.

    Outputs with high structural fidelity generally provide better citation quality during follow-up questions.

    4. Testing Citation Traceability: Treat Evidence as a First-Class Output

    Citation capability should be evaluated through a dedicated set of tasks rather than treated as a secondary aspect of image understanding.

    1. Location-Based Questions

    Instead of asking, “What is this image about?”, ask:

    “Which part of the image supports your conclusion? Please identify the specific location, such as the third row of the table in the upper-left corner.”

    An acceptable answer should include both:

    • The location; and
    • The relevant content.

    An answer that merely repeats the conclusion without identifying its source should be considered unqualified.

    2. Follow-Up Consistency Testing

    After obtaining a citation in the first round, ask a detailed follow-up question and check whether the model continues to rely on the same source.

    If the model silently changes its evidence in the second round without explaining the change, its evidence chain is unstable. This behavior is unacceptable in document review and similar scenarios.

    3. Adversarial or Unsupported-Claim Testing

    Deliberately ask a question that has no supporting evidence in the image. For example, ask, “What growth rate is mentioned in the image?” when the image does not contain any growth-rate metric.

    The ideal response is an explicit statement such as, “That information is not provided in the image.” If the model invents a value, this should be treated as a high-risk failure and assigned a substantial penalty in the overall score.

    5. Engineering Recommendations for Operationalizing the Evaluation

    Teams that want to integrate this type of evaluation into their regular development process can consider the following practices:

    • Build a fixed test set: Prepare 10–20 samples for each material type, together with manually annotated reference answers and evidence locations. Use the collection as a regression benchmark.
    • Use a structured scoring sheet: Make key metrics such as key-field accuracy, citation-location completeness, and unsupported-answer refusal rate central evaluation criteria. This avoids subjective scoring based only on overall impressions.
    • Optimize prompt templates: In practical use, constraints such as “Please identify the location of the supporting evidence in the image” and “Use only the information shown in the image” can significantly reduce unsupported citation generalization.
    • Integrate with general cloud services: Store evaluation materials centrally through object storage, and connect evaluation results to logging and monitoring systems to enable automated comparisons across version iterations.

    6. Conclusion

    The value of multimodal capabilities is not simply that a model can look at an image. The real value lies in whether it can understand accurately, explain clearly, and support its answer with traceable evidence.

    A robust evaluation framework should cover both understanding accuracy and citation traceability. Location-based questions, follow-up consistency tests, and unsupported-claim tests can expose answers that appear correct but cannot be verified.

    For enterprise users, turning this evaluation process into a regression benchmark is more valuable in the long term than pursuing impressive results in a single trial. In real business workflows, an answer that can be verified is an answer that can be used.