Multimodal LLMs for Coding: Practical Experience, Pitfalls, and Workarounds

    This article reviews the practical use of multimodal large language models in coding workflows. It covers basic evaluation criteria, observed capability boundaries, common pitfalls, and workflow-based strategies for selecting and deploying these tools. It emphasizes that human review, structured constraints, plan-first validation, and secondary verification are essential for reliable delivery.

    1. Background: Visual Capabilities Are Entering the Coding Workflow

    Over the past year, large language models have continued to expand their capabilities, moving from pure text understanding toward multimodal systems that can process images, UI screenshots, and documents. When these models are integrated into coding tools, developers can provide requirement documents, design mockups, and error screenshots to the model and use it in scenarios such as frontend restoration, UI debugging, and failure localization. These capabilities have brought measurable efficiency gains in real workflows.

    However, there is still a clear gap between “usable” and “effective.” Based on a large-scale, long-term record of real-world usage, this article provides a systematic review of the capability boundaries, common pitfalls, and practical strategies for using multimodal models in coding scenarios. It is intended as a reference for practitioners evaluating and adopting these tools.

    2. Baseline Experience: Evaluate the Hard Metrics First

    Before evaluating any coding-assistant model, it is useful to establish a set of measurable baseline metrics:

    • Output stability: Does the model produce garbled text, formatting errors, or truncated output?
    • Frontend and UI capability: How well does it understand and reproduce layout, styles, and component structure?
    • Response speed: Are average first-token latency and generation speed stable enough to support long sessions?
    • Context retention: In multi-turn conversations, does the model forget earlier agreements or constraints?

    In testing, the model performed reasonably well in several areas, including output stability, UI capability, and response speed. Quota consumption also remained acceptable during extended continuous use. This suggests that basic engineering capability is no longer the main bottleneck. The real differences show up in deeper reasoning and instruction-following capability.

    One additional point is worth noting: speed fluctuations across different time periods, such as peak hours and late-night usage, are common. Evaluation should cover multiple load scenarios rather than relying on experience from a single time window.

    3. Typical Problems: Where Are the Capability Boundaries?

    1. Insufficient Reasoning Depth: The Model Only Handles What Is Immediately in Front of It

    One of the most direct observations is that the model can feel “not sharp enough.” It can complete a single-step task, but it often lacks planning and backtracking across the broader objective.

    For example, when asked to summarize changes to a component over several recent versions, the model may only aggregate the latest change and fail to actively cover all versions within the specified range. A model with stronger reasoning capability would typically summarize the entire interval proactively. This “task-at-hand” tendency reduces the completeness of complex tasks and forces users to provide repeated clarification.

    2. Weak Adherence to Rules and Instructions

    When project rules explicitly constrain how tasks should be executed—for example, services must be started through a containerized environment with complete steps provided—the model may still switch to a local runtime environment and ignore the existing constraints.

    A more subtle issue is the loss of requirement-level constraints. For example, a requirement may distinguish between events where notifications are enabled by default and events where they are disabled by default. However, in the final implementation, this distinction may not be preserved.

    These problems are not necessarily caused by insufficient capability. Instead, they reflect insufficient attention to constraints contained in long text.

    3. Divided Attention and Subtask Scope Creep

    In coding tools that support multi-agent or sub-agent architectures, this problem becomes more visible. Sub-agents may perform the wrong task or take on too much work, drifting beyond the intended scope. This issue is especially noticeable in complex, multi-turn collaboration.

    In addition, when requirements are slightly shifted or contain omissions, the model may misinterpret them. In other words, the more convoluted an instruction is, the higher the demand on the model’s ability to recover the user’s true intent.

    4. Visual Detail Capture and Output Quality

    Visual capability is an advantage, but fine-grained detail remains a weakness.

    In website redesign tasks, issues such as slight color deviations on homepage buttons can recur and require multiple rounds of fixing before they are correctly identified. When style conflicts are difficult to resolve at the source-code level, the model may degrade into writing inline styles directly on the page as a workaround.

    This shows that visual understanding can help ensure that the general direction is correct, but pixel-level and detail-level consistency still requires additional verification.

    4. Practical Strategies

    Simply switching to another model will not fundamentally solve these problems. A more realistic approach is to use workflow design to compensate for the model’s capability boundaries.

    1. Introduce Human Review Gates

    All plans, scripts, and proposals generated by the model should undergo human review and follow-up discussion before actual execution.

    This is especially important for outputs such as load-testing scripts, which may run successfully but still be meaningless. Users need to confirm that the logic truly points to the intended goal, rather than merely being syntactically correct and executable.

    2. Require “Explain the Plan First, Then Implement”

    A low-cost and effective technique is to ask the model to explain its implementation approach and expected outcome after it produces a plan. This acts as a pre-check before execution:

    1. Your plan has been received.
    2. Before you start implementing, please explain:
    3. 1) How many steps you plan to take, and what the input and output are for each step;
    4. 2) What the final deliverable will look like and how to verify whether the goal is achieved;
    5. 3) What constraints or boundary conditions need to be followed.

    This step can clearly expose misunderstandings in requirement interpretation and intercept errors before coding begins, rather than discovering that the direction is wrong only during acceptance testing.

    3. Move Specifications and Constraints to the Front

    Fine-grained constraints, such as whether notifications should be enabled or disabled by default, should be written explicitly in requirement documents or system rules. The same applies to runtime environment requirements, naming conventions, and style guidelines.

    These items should be expressed in a structured, checklist-like format and placed early in the workflow. At key checkpoints, the model should be asked to restate the constraints to reduce attention decay across long text.

    The more the constraints resemble a checkable list, the less likely they are to be missed.

    4. Apply Secondary Verification to Visual and Styling Tasks

    For UI restoration tasks, users should not rely only on the model’s claim that the work is “done.” Secondary verification should be performed through methods such as screenshot comparison and style review.

    When style conflicts are difficult to resolve at the source-code level, human intervention should be used to clarify style priority and inheritance relationships. Letting the model repeatedly patch problems with inline styles can create hidden maintenance risks.

    5. Break Down Tasks and Constrain Boundaries

    For complex tasks, it is better to decompose the work into smaller subtasks with clear responsibilities. For each sub-agent, specify what it should do, what it should not do, and what it must deliver.

    Clear task boundaries can significantly reduce the probability of sub-agents overstepping their scope or duplicating work. In many cases, clear boundaries improve overall output quality more effectively than simply using a stronger model.

    5. Conclusion

    Multimodal large language models are entering coding workflows and bringing new possibilities in areas such as UI restoration and document understanding. At the same time, their basic engineering metrics are becoming increasingly mature.

    However, in the current stage, there are still clear limitations in reasoning depth, constraint adherence, attention focus, and detail control.

    The rational approach is not to pursue full automation blindly, but to position these models as collaborators: efficient assistants that still require experienced human oversight. Through engineering practices such as human review, plan-first explanation, structured constraints, and secondary verification, teams can bring the uncertainty of model behavior under control.

    Only in this way can organizations benefit from efficiency gains while still maintaining a reliable baseline for delivery quality.