RAG Document Chart Processing: Table Parsing and Multimodal Fusion in Practice
This article examines how RAG systems can handle tables and image-based charts in documents without losing structural semantics. It covers common failure modes such as broken row-column relationships, mismatched retrieval granularity, weak generation-side understanding, and missing retrievable text for multimodal figures. The article then presents practical methods for table recognition, structured HTML conversion, and context stitching, followed by approaches for turning flowcharts and statistical charts into retrievable descriptions. It also discusses multi-granularity retrieval, prompt constraints, evaluation tasks, and an engineering checklist for building more reliable chart-aware RAG pipelines.
RAG Document Chart Processing: Table Parsing and Multimodal Fusion in Practice
In retrieval-augmented generation (RAG) systems, text passages are usually easy to chunk, vectorize, and recall. Figures and tables in documents, however, are often the parts most likely to be “downgraded” during processing. Tables are extracted row by row and lose row-column semantics; flowcharts are skipped as ordinary images; statistical charts keep only captions while losing data trends. These problems can lead to structural misinterpretation, incorrect numerical comparisons, or incomplete citations when a large language model answers questions. The core solution is not simply adding an OCR step. Instead, figures and tables should be treated as “structured knowledge objects,” with their semantics preserved across parsing, retrieval, stitching, and generation.
1. Main Challenges When Figures and Tables Enter RAG
Figures and tables in documents can be broadly divided into two categories. The first is tables, including ordinary tables, tables with merged cells, cross-page tables, and nested headers. The second is image-based charts, including flowcharts, architecture diagrams, sequence diagrams, line charts, bar charts, and pie charts. They face different but related problems in the RAG pipeline.
First, parsing distortion. If a table is extracted as “one row per record,” the data order can be disrupted when there are row spans, column spans, or merged cells. For example, if “Adjustment logic” spans two columns and “Example” spans three rows, simple row extraction may cause the model to assume that every cell belongs only to the current row, breaking the correspondence between headers and data.
Second, retrieval granularity mismatch. When an entire table is too large, its semantics are diluted after vectorization. When it is split too finely, headers and context are lost. A user may ask whether market research should be initiated for a certain scenario, which may require only a few rows from the table. However, the answer must also see the headers, merged-column descriptions, and table notes.
Third, insufficient understanding on the generation side. Even if table content is placed into the context, it is difficult for a large language model to determine column relationships, row-span relationships, and the meaning of empty cells when the table is presented as plain text separated by commas. A representation is needed that is both close to the original layout and stable for the model to understand.
Fourth, multimodal charts lack retrievable text. Flowcharts and statistical charts are images themselves. If only image links are stored, the retrieval stage cannot match them. If only captions are stored, the system cannot answer questions about node relationships or trend changes in the image. Therefore, the structure, text, data points, and relationships in images need to be converted into retrievable and citable descriptions.
2. Table Processing: Recognition, Structured Conversion, and Context Stitching
The recommended table-processing pipeline can be summarized in three steps: reading and recognition, structured conversion, and context stitching. The goal is not to “turn a table into a paragraph,” but to “turn a table into an HTML fragment that preserves row-column semantics and send it into retrieval and generation together with section context.”
2.1 Reading: First Identify Layout and Cell Relationships
The reading stage needs to identify the table region, headers, table body, table notes, and merged-cell information. For scanned documents, layout analysis and OCR are also required. For electronic documents, table objects can be parsed directly. The key output should not be merely a list of text. It should include information such as each cell’s row number, column number, row span, column span, whether it is a header, and original coordinates.
A simplified intermediate structure can be represented as:
{"table_id": "t-001","caption": "Monthly price adjustment logic example","cells": [{"r": 0, "c": 0, "text": "Scenario", "rowspan": 1, "colspan": 1, "header": true},{"r": 0, "c": 3, "text": "Adjustment logic", "rowspan": 1, "colspan": 2, "header": true},{"r": 1, "c": 4, "text": "Do not conduct market research; use the online monthly average range", "rowspan": 1, "colspan": 2}]}
The focus of this step is to “preserve structure,” not to rush into generating natural language. Merged cells must record their coverage relationships; otherwise, the correspondence between headers and data cannot be restored later.
2.2 Conversion: Use HTML to Express Row-Column Semantics
Converting tables to HTML is a relatively reliable approach in practice. HTML’s table, thead, th, td, rowspan, and colspan can express table structure fairly completely, and large language models also have good understanding of HTML tables. Compared with comma-separated text, HTML can explicitly tell the model which parts are headers, which cells span columns, and which rows belong to the same group.
An illustrative HTML example is as follows:
<table><thead><tr><th>Scenario</th><th>Online monthly average range</th><th>Example</th><th colspan="2">Adjustment logic</th><th>Settlement adjustment range example</th></tr></thead><tbody><tr><td></td><td>Range exceeds 5.00%</td><td>5.50%</td><td colspan="2">Do not conduct market research; use the online monthly average range</td><td>5.50%</td></tr><tr><td></td><td>Range does not exceed ±5.00%</td><td>±3.00%</td><td colspan="3">Do not conduct market research; do not adjust the execution price</td></tr></tbody></table>
The conversion algorithm can use a “grid placement” approach: first create a two-dimensional matrix based on the maximum number of rows and columns, then place cells one by one. When row spans or column spans are encountered, mark the covered positions in the matrix. When generating HTML, skip covered positions and output the corresponding rowspan and colspan. Pseudocode is as follows:
def table_to_html(cells, rows, cols):grid = [[None for _ in range(cols)] for _ in range(rows)]for cell in cells:r, c = cell.r, cell.cwhile grid[r][c] is not None:c += 1if c >= cols:c = 0r += 1for i in range(cell.rowspan):for j in range(cell.colspan):grid[r + i][c + j] = cellhtml = ["<table>"]for r in range(rows):html.append("<tr>")for c in range(cols):cell = grid[r][c]if cell is None or cell.is_covered(r, c):continuetag = "th" if cell.header else "td"attrs = []if cell.rowspan > 1:attrs.append(f'rowspan="{cell.rowspan}"')if cell.colspan > 1:attrs.append(f'colspan="{cell.colspan}"')html.append(f"<{tag} {' '.join(attrs)}>{cell.text}</{tag}>")html.append("</tr>")html.append("</table>")return "".join(html)
In actual engineering, it is also necessary to handle empty cells, multi-row headers, cross-page tables, footnotes, unit notes, and inherited merged headers. For cross-page tables, multiple fragments under the same table caption should first be merged, and then converted uniformly.
2.3 Stitching: Put the Table Back into Section Context
A table should not enter the vector store in isolation. It is recommended to stitch the section title, table caption, table notes, unit notes, and necessary explanatory paragraphs before and after the table HTML to form a “self-contained context block.” For example:
Section: 4.2.1 Annual used commodity price adjustmentTable caption: Monthly price adjustment logic exampleTable note: When the range exceeds 5.00%, do not conduct market research and use the online monthly average range; when it does not exceed ±5.00%, do not adjust the execution price.Structured content:<table>...</table>Reading hint: In the table, "Adjustment logic" spans columns, and "Example" and "Settlement adjustment range example" need to be understood together with the headers.
After this stitching, retrieval matches not just a table fragment but a knowledge block with semantic boundaries. During generation, the model can clearly identify which business rule the current table belongs to, reducing the problem of “seeing only local rows without knowing column meanings.”
3. Multimodal Charts: From Images to Retrievable Descriptions
For flowcharts, architecture diagrams, and statistical charts, it is recommended to generate multimodal descriptions through “image understanding + structure extraction + caption association.” The description should not merely say “this is a flowchart.” It should include chart type, title, legend, axes, key nodes, edge relationships, data trends, and answerable questions.
For example, a statistical chart can be converted into:
Chart type: Line chartTitle: Monthly change trend of a certain metricHorizontal axis: Month; vertical axis: Range percentageKey data points: January 3.0%, February 5.5%, March -2.0%Trend: Reaches a peak in February and turns negative in MarchAnswerable questions: Which month has the highest range; whether the change exceeds the threshold; what the peak and trough values are
A flowchart can be converted into a list of nodes and edges:
Node A: Submit applicationNode B: System validationNode C: Manual reviewEdges: A->B; B->C when passed; B->End when rejected
These descriptions, together with the original caption, section, and surrounding paragraphs, form a multimodal knowledge block. Image vectors and text vectors can be stored separately and recalled together during retrieval. After an image description is matched, the corresponding image reference and structured description can be sent into the generation context. It is important to preserve uncertainty markers in multimodal descriptions so that low-confidence recognition results are not treated as confirmed facts by the model.
4. Retrieval and Generation: Multi-Granularity Blocks and Prompt Constraints
For tables and charts, a multi-granularity retrieval design is recommended:
- Table whole block: contains the table caption, table notes, and complete HTML, used to answer rule-based and overall comparison questions.
- Row-group block: grouped by business semantics and retaining a copy of the header, used for fine-grained recall.
- Chart description block: contains chart type, data points, trends, and answerable questions.
- Parent-child block: child blocks are used for retrieval matching, and parent blocks are used to return complete context, avoiding overly fragmented context.
Generation prompts should explicitly constrain the model: when answering table questions, it must rely on the row-column relationships in the HTML; when merged cells are involved, it should first explain the headers; values that cannot be confirmed from the table should not be inferred; and the answer should end with a section or table caption citation. The following template can be used:
You will answer based on the given context. If the context contains an HTML table, first identify the headers, row-span and column-span relationships, and table notes before performing numerical comparisons.Requirements:1. Use only information that appears in the context;2. For columns with merged cells, explain their coverage scope;3. If something cannot be determined, answer "not explicitly stated in the context";4. After the answer, list the table caption or section used as evidence.
5. Evaluation and Engineering Checklist
Chart-aware RAG cannot be judged only by subjective experience. It is recommended to build an evaluation set covering five task types: table question answering, row-span and column-span understanding, numerical comparison, chart trend questions, and citation consistency. Metrics should include structural fidelity, retrieval hit rate, answer correctness, and citation consistency. Do not promise a fixed accuracy rate; instead, continuously regress on the distribution of your own documents.
The implementation checklist includes: preserve cell coordinates and merge information during parsing; use a unified HTML representation during conversion; retain section, table caption, and table notes during stitching; use parent-child blocks and hybrid recall during retrieval; add structural understanding constraints during generation; and cover merged cells and statistical chart trends during evaluation. Only by upgrading figures and tables from “attachments” to “structured knowledge objects” can a RAG system handle complex documents more reliably.
