How to Give Text-Only LLMs Image Recognition Capabilities with MCP

    This article explains a widely underestimated pain point: most text-centric conversational LLMs simply cannot "see" images. Rather than forcing teams to replace an existing text model with a heavier, native multimodal one, it shows how to decouple visual understanding into a standalone capability exposed as a standard tool via the Model Context Protocol (MCP). You'll find a four-layer architecture, an end-to-end request flow, a concrete tool schema and call-chain example, plus production guidance on caching, error handling and fallback, security and privacy, cost control, and observability—along with common pitfalls and advice on when to choose composition over a bundled multimodal model.

    Introduction: Why “LLMs can’t read images” is an underestimated pain point

    In real-world development and deployment, we run into this scenario all the time: a chat thread has an error screenshot pasted into it, a business group shares a line chart, a support ticket comes with a photo of an order attached. Users naturally expect the model to “understand” the image and give a judgment, but most text-centric conversational models only accept string input and are effectively blind to the image itself.

    Two things are behind this mismatch. The first is architectural: a text-only model’s input interface accepts only token sequences, and an image must first pass through a dedicated vision encoder that converts it into a representation the language model can understand—not every model ships with that capability built in. The second is an engineering trade-off: models with vision capability are typically larger, with higher inference cost and latency, so teams chasing lightweight, low-latency, high-throughput scenarios often deliberately pick text-only models.

    This raises a genuinely valuable question: how do you let a text model that is already live gain image-reading ability on demand, instead of being forced into a wholesale model replacement? The answer this article offers is to decouple “reading an image” from inside the model and turn it into an independently callable capability.

    Pinning down the problem: where the capability gap actually is

    To solve a problem, you first have to describe it precisely. The gap here is not that “the model isn’t smart enough,” but an interface-level mismatch:

    • Input mismatch: an image is a two-dimensional pixel matrix, while the model’s entry point is a symbol sequence. A translation layer is missing in between.
    • Responsibility mismatch: text models are good at reasoning, induction, writing, and structured output, but not at low-level visual perception.
    • Scheduling mismatch: even if the organization has a model that can do visual work, the text model has no mechanism to “reach out and use it.”

    Put together, the requirement is clear: you need a mechanism that lets the text model, when it encounters image input, delegate “describe this image” to something else and then feed the result back into its own reasoning flow. This is exactly the territory that the idea of tool use (Tool Use / Function Calling) is meant to cover.

    Solution approach: composition over bundling

    Faced with a capability gap, there are usually two paths. The first is “bundled”: switch directly to a native multimodal model that can both see images and hold a conversation. This route is simple, but the cost is pushing every request to a heavier, more expensive model, and getting locked into a single model’s versions, rate limits, and availability.

    The second is “composed”: split different capabilities into independent units and wire them together on demand through a standard protocol. Its core advantages are:

    • Controllable cost: the vast majority of text-only requests (Q&A, writing, code, structured extraction) still go to a lightweight text model, and visual capability is called extra only when an image actually shows up.
    • Loose coupling: the vision model can be upgraded and scaled independently, or even swapped for a different vendor at any time, without affecting the main conversation path.
    • Reusability: the same visual capability can serve multiple upstream applications, rather than being integrated repeatedly into every model.

    This article chooses the second path and uses MCP (Model Context Protocol) to implement it. To be clear, MCP is an open, general-purpose protocol that lets AI applications expose “tools, resources, and prompts” to models in a standardized way. It is not a proprietary product from any single vendor, but an integration paradigm widely adopted across the industry. You can think of it as the “USB port” of the AI world: a model on one side, a capability provider on the other, connected through a unified slot in the middle.

    Solution design: plugging visual capability into a conversation with MCP

    The overall architecture can be abstracted into four layers:

    1. Conversation entry layer: receives user messages and detects whether they contain an image (attachments, URLs, Base64, and so on).
    2. Main model layer: a text LLM at the core, responsible for intent understanding and the final response, with tool-calling capability.
    3. Tool protocol layer (MCP): declares and exposes the “image understanding” tool in a standard way, describing its inputs and outputs.
    4. Capability execution layer: the model or vision interface that actually does visual perception, responsible for turning an image into a text description, structured fields, or tags.

    A typical interaction flows like this:

    • The user sends a message that contains an image.
    • The main model recognizes “this requires looking at an image,” so instead of answering directly, it issues a tool call, passing the image data to the “image understanding” tool.
    • The tool protocol layer routes the request to the capability execution layer, the vision model performs recognition, and a textual result comes back (a description, OCR text, chart readings, structured JSON, and so on).
    • The result returns to the main model, which combines the image description with the user’s original question to complete its reasoning and produce the final response.

    To the end user, the whole process is seamless: they simply feel that “this text model can read images now, too.”

    Implementation details: defining the tool and the call chain

    In practice, the key is to design “image understanding” as a tool with clear semantics and stable inputs and outputs. A tool description typically includes a name, a purpose statement, and a structured parameter definition (expressed here as an illustrative schema, not tied to any specific interface):

    1. {
    2. "name": "describe_image",
    3. "description": "Analyze an image and return a textual result, used for OCR, chart reading, scene description, and structured extraction.",
    4. "input_schema": {
    5. "type": "object",
    6. "properties": {
    7. "image": {
    8. "type": "string",
    9. "description": "The image URL or Base64-encoded string"
    10. },
    11. "task": {
    12. "type": "string",
    13. "enum": ["ocr", "caption", "chart_reading", "extract_json"],
    14. "description": "The type of visual task you want the model to perform"
    15. },
    16. "prompt": {
    17. "type": "string",
    18. "description": "Supplementary instructions for the specific task, such as the fields to focus on"
    19. }
    20. },
    21. "required": ["image", "task"]
    22. }
    23. }

    On the server side, the tool handler can roughly be written as the following illustrative logic (language-agnostic, pseudocode style):

    1. def handle_describe_image(args):
    2. # 1. Validate and normalize the input
    3. image = normalize_input(args["image"]) # accepts URL / Base64
    4. task = args["task"]
    5. prompt = args.get("prompt", default_prompt_for(task))
    6. # 2. Call the vision capability execution layer (replace with the actual vision model interface)
    7. visual_model = load_visual_model()
    8. raw = visual_model.infer(image=image, task=task, instruction=prompt)
    9. # 3. Normalize the output so the main model can parse it reliably
    10. return {
    11. "task": task,
    12. "description": raw.text,
    13. "structured": raw.json if task == "extract_json" else None,
    14. "confidence": raw.confidence # optional confidence score for fallback decisions upstream
    15. }

    On the main-model side, the call is often triggered automatically by the model’s tool-selection ability, but it can also be orchestrated explicitly at the application layer:

    1. def answer(user_message):
    2. if has_image(user_message):
    3. visual = call_mcp_tool(
    4. name="describe_image",
    5. arguments={
    6. "image": user_message.image_ref,
    7. "task": infer_task(user_message), # infer whether it's OCR or description based on the question
    8. "prompt": build_prompt(user_message.text)
    9. }
    10. )
    11. context = f"[Image info] {visual['description']}"
    12. else:
    13. context = ""
    14. return main_text_model.chat(
    15. system="You are a rigorous analysis assistant who can answer using image information.",
    16. user=f"{context}\n{user_message.text}"
    17. )

    The benefit of this approach is that the responsibility boundaries are very clean: the main model only handles “when to use the tool and how to reason afterward,” the vision model only handles “seeing the image clearly,” and the two are decoupled through a unified protocol, so any layer can evolve independently.

    Production engineering: key optimization points

    To actually run this in production, you need to polish it across several dimensions—otherwise it’s easy to trip over cost, stability, or compliance.

    Caching and deduplication

    Image recognition is the step most prone to repeated calls. You can use “image content hash + task type + instruction” as the cache key and store results in a cache service, returning immediately on a hit. For the scenario where the same image keeps reappearing across a long conversation, caching can significantly reduce call volume and latency.

    Error handling and fallback

    A vision call is an external dependency, and you must assume it will fail. Sensible measures include: setting timeouts and retry limits; on failure, returning a structured “recognition failed” marker to the main model so it can choose to ask a follow-up question, ask the user to describe it in text, or clearly state that it cannot recognize the image, rather than staying silent and fabricating. Never let the text model “make things up” about the image content when recognition fails—that produces serious hallucinations.

    Security and privacy

    Images may contain sensitive information (ID cards, contracts, faces). In practice you need to: encrypt the transport path; redact sensitive images or apply access controls; avoid retaining raw images and recognition results in logs in plaintext; and clearly define data retention periods and boundaries of use. When personal information is involved, follow the principle of data minimization and keep traceable records.

    Cost control

    Compress and crop images to a reasonable size to avoid pointless transmission of large files; route by task complexity, sending simple OCR and complex chart interpretation to capability tiers of different specs; and combine caching with batch processing to amortize per-call cost. The core idea is “save where you can, and pay only when needed.”

    Observability

    Add instrumentation to every vision call: call volume, success rate, average latency, token/duration consumption, and the distribution of failure reasons. This data supports both capacity planning and diagnosing whether “poor image recognition” stems from input quality or from instruction design.

    Common pitfalls and selection advice

    A few pitfalls are worth watching out for in practice:

    • Pitfall one: routing all requests to a multimodal model. This brings unnecessary cost and latency; the correct approach is composition on demand.
    • Pitfall two: treating the visual result as absolute fact. Vision models also misread things, especially with blurry, occluded, or complex charts. For critical decisions, keep a confidence assessment and a human review channel.
    • Pitfall three: neglecting instruction design. With the same vision model, “describe this image” and “extract the amount, date, and title from this invoice” produce very different results—instructions need to be specific to the task goal.

    As for selection: if you want a one-stop experience and are not sensitive to cost, a native multimodal model is the easier choice; if you already have a mature text-model stack and value cost, controllability, and replaceability, then the combination of “text model + MCP vision tool” tends to fit engineering reality better.

    Conclusion

    LLMs “not understanding images” is not an insurmountable gulf, but an integration problem that can be solved elegantly with engineering. By decoupling visual capability from inside the model and wrapping it as a standard tool through an open protocol like MCP, we can give existing text models the ability to read images on demand, without sacrificing cost or stability. This idea of “pluggable capability” is the key paradigm for building resilient, scalable AI applications—first understand the boundaries of the problem, then respond with composition rather than bundling.