How Text-Only LLMs Understand Images: Engineering Visual Extensions
A text-only language model cannot process pixels directly, but a surrounding system can add visual capabilities through image captioning, OCR, structured extraction, or tool calling. This article explains the differences between model and system capabilities, compares common architectures, presents a proxy-layer implementation pattern, and discusses information loss, quality control, cost, latency, security, and deployment considerations.
1. Model Capability vs. System Capability
Many people interpret “can understand images” as meaning that the model itself has native vision capabilities. In practice, whether a system can process images depends on two separate layers.
The first is model capability. Native multimodal models align an image encoder with a language model during training. After visual encoding, an image is represented as vectors in a semantic space shared with text tokens, allowing the model to perform attention over both types of input.
The second is system capability. Even when the underlying model accepts text only, a service can place preprocessing modules in front of it. These modules convert an image into text or structured information and then pass the result to the language model. From the user’s perspective, the model answered a question about an image. In reality, multiple components collaborated to produce the answer.
This distinction is important when diagnosing problems such as image-upload errors or inaccurate answers. The first step is to determine whether the issue originates in the model layer or the surrounding system.
2. Text-Only APIs vs. Multimodal APIs
A text-only model accepts a sequence of discrete text tokens. Its API generally defines text fields only. If a client sends image data, the server will typically reject the request during parameter validation rather than attempting to interpret the pixels.
A multimodal API defines image fields in addition to text. The server must then:
- Validate the image format and size.
- Call a vision encoder to generate image features.
- Combine those features with text embeddings and pass them to the inference process.
When the same business application integrates with both types of models, it should check the model’s declared capabilities before making a request. If capability metadata is missing, the application should reject image input and return a clear message. This avoids silent failures or answers that are unrelated to the image.
3. Three Common Architectures for Adding Vision
3.1 Caption First, Then Reason
This is the most straightforward approach. A vision model first generates a natural-language description of the image. The description is combined with the user’s question and sent to a text model for reasoning.
Image → Vision captioning model → Text descriptionText description + User question → Text reasoning model → Final answer
The main advantage is simplicity: the existing text model can continue to be used. The limitation is that the captioning step defines the information ceiling. Details lost during image-to-text conversion cannot be recovered by subsequent reasoning.
3.2 OCR and Structured Extraction
For screenshots, scanned documents, tables, and receipts, text is usually the primary source of information. These scenarios are better handled by using optical character recognition to extract text, followed by layout analysis to reconstruct paragraphs, tables, and heading hierarchies. The output can be formatted as Markdown or JSON.
Structured output preserves row and column relationships as well as numerical values. For reports and contract clauses, this often produces more accurate results than a free-form image description. The main challenge is reconstruction quality when the layout is complex, so the output should be validated.
3.3 Vision as a Tool Call
In the third architecture, image analysis is exposed as a tool. During reasoning, the model decides whether it needs to inspect an image. If so, it calls the tool. An independent service analyzes the image, writes the result back into the context, and the model continues reasoning.
The advantage is on-demand processing. The system does not need to analyze every image, which makes cost and latency more controllable. The trade-off is an additional round trip. Whether the model chooses the right time to call the tool can directly affect the final result.
Comparison of the Three Approaches
| Approach | Information fidelity | Latency | Typical use cases |
|---|---|---|---|
| Caption first, then reason | Medium; limited by caption quality | Two sequential calls add latency | General image question answering |
| OCR and structured extraction | High for text and tables | Depends on layout complexity | Screenshots, reports, contracts |
| Vision through tool calling | Depends on the tool output | Adds a round trip when needed | Occasional image-analysis tasks |
4. Implementing a Proxy Layer
In production systems, a proxy layer is commonly placed between the business application and the text model. This layer centralizes image preprocessing, prompt construction, and fallback behavior. The following example shows the core structure; the vision service and model client should be replaced according to the deployment environment.
from dataclasses import dataclass@dataclassclass ImageInput:content: bytesmime_type: strsource_id: str # Used for tracing and cachingdef describe_image(img: ImageInput) -> str:"""Call a vision captioning or OCR service and return structured text."""raise NotImplementedErrordef build_prompt(question: str, descriptions: list[str]) -> str:header = ("The following content consists of text descriptions of images. ""Answer only according to these descriptions. ""If the descriptions do not contain the required information, ""state clearly that it cannot be determined. Do not guess.")blocks = [header]for idx, desc in enumerate(descriptions, start=1):blocks.append(f"[Image {idx}]\n{desc}")blocks.append(f"User question: {question}")return "\n\n".join(blocks)def answer_with_images(question: str, images: list[ImageInput], text_model) -> str:descriptions = [describe_image(img) for img in images]prompt = build_prompt(question, descriptions)return text_model.chat(prompt)
Three design choices in this example are particularly important.
First, the prompt explicitly limits the model to the available descriptions and requires it to identify unknown information. This is a basic measure for reducing hallucinations.
Second, each image has an identifier. This helps the model cite the source and makes human review easier.
Third, description results can be cached using a hash of the image content. When the same image appears again, the system does not need to call the vision service a second time.
5. Information Loss and Quality Control
Information loss is the central risk of an external vision pipeline. If the description stage omits a detail, the text model cannot reconstruct it from nothing. Commonly lost information includes:
- Small text
- Exact color values
- Spatial relationships
- Chart scales and legends
- Watermarks and labels
Several measures can reduce this risk.
Use Multiple Levels of Description
For the same image, retain an overall summary, the raw OCR output, and descriptions of important local regions. This gives the reasoning stage multiple information sources.
Validate Numerical Values
For reports, totals should be checked against the sum of their line items. Units and decimal places should also match the original image. If validation fails, the system should ask the user to review the result rather than present an unqualified conclusion.
Require Confidence or Evidence Labels
Prompts can require the model to distinguish between information that is clearly shown in the image and information that is inferred from context. This allows downstream systems to apply different levels of handling or review.
Defend Against Prompt Injection in Image Text
Images may contain text such as “ignore the previous instructions.” Text extracted from an image should be treated as data to analyze, not as an instruction.
In practice, place image text inside a clearly marked data block and state in the system prompt that content inside that block has no instructional authority.
6. When External Vision Is and Is Not Suitable
External vision works well in scenarios such as:
- Extracting text from screenshots
- Interpreting values in charts and reports
- Describing the structure of interface design mockups
- Analyzing well-formatted scanned documents
These tasks mainly involve information that can be represented in text.
It is less suitable for:
- Design reviews that require pixel-level color judgments
- Detecting or counting very small objects
- Video or audio content
- Tasks that depend on precise spatial localization
- Fine-grained aesthetic evaluation
The conversion process loses information in these cases, and later text reasoning cannot compensate for that loss.
A simple decision rule is to ask: If the complete content of the image were written as text, would important information be missing?
If the answer is no, an external vision pipeline is usually sufficient. If the answer is yes, use a native multimodal model or retain a human review step.
7. Cost and Latency
An external vision pipeline usually involves two model calls: one for visual processing and one for text reasoning. Cost and latency are therefore the sum of both stages. It should not be assumed that the total cost will automatically be lower than that of a single multimodal call.
Cost estimation should account for:
- How the vision service charges, such as by image count, resolution, or tokens
- The length of the text-model input, since descriptions can be long
- Cache hit rates for repeated images
Use representative production samples for load testing and calculate the cost per task according to the actual billing model rather than relying on assumptions.
For latency, description tasks for multiple images can be processed in parallel. If the application requires fast responses, image descriptions can also be generated asynchronously when users upload images, allowing the system to retrieve the results immediately when a question arrives.
8. Selection and Deployment Recommendations
Before implementation, evaluate the following questions in order:
- Can the essential information in the task be represented as text? If not, prioritize a native multimodal model.
- Does the image contain a high proportion of text? If so, prioritize OCR and structured extraction.
- Are the same images processed frequently? If so, build a cache for description results.
- Do the images contain personal or sensitive information? If so, assess the compliance requirements for sending images to an external vision service. A locally deployed vision component may be necessary.
- Should the model decide when to inspect an image? If so, use a tool-calling design with timeouts and failure fallbacks.
Two additional engineering practices are important.
First, check model capability declarations during startup or configuration. If an endpoint does not support images, reject the request immediately with a clear error code instead of forwarding the image and waiting for it to fail.
Second, log every stage of the process, including the visual description, prompt, and model version. This makes it possible to determine which component introduced an error when the final answer is incorrect.
The value of external vision lies in extending an existing text model through a controllable combination of components. Separating visual captioning, structured extraction, and validation logic helps maintain a practical balance between accuracy, cost, and maintainability.
