Multimodal LLM-Assisted Coding: Practical Experience and Pitfalls

    This article shares practical lessons from using multimodal large language models for coding tasks during an intensive one-month period that consumed nearly one billion tokens. It covers baseline strengths such as stable output and strong frontend support, along with recurring problems such as response-time fluctuations, limited reasoning depth, imprecise intent handling, incomplete adherence to rule documents, weak subtask boundary control, and missed visual details. It also presents practical strategies: adding human approval gates, making constraints explicit, placing strong rules at the top of prompts, breaking down tasks, and applying layered verification for logic, rules, and visuals.

    Introduction

    Multimodal large language models are entering coding workflows, and developers are both excited and cautious. These models can read UI screenshots, understand requirement documents, and generate runnable code, so they may look like a full-stack assistant in theory. However, after one month of intensive use with nearly one billion tokens consumed, the distance between these models and the ideal of fully automated coding is still clear. Based on a real deep-use record, this article reviews the baseline experience, analyzes the root causes of typical problems, and summarizes practical mitigation strategies for engineering teams.

    1. Baseline experience: stable, with both surprises and letdowns

    Overall, the basic capabilities of these models are solid:

    • Output is clean and well formatted, with no garbled text.
    • Frontend and interface-related capabilities are relatively strong, and the model can quickly scaffold page structures.
    • Response is stable, and a single session can handle a relatively large token throughput.

    During one trial cycle, three hours of continuous work consumed only about 86% of the usage allowance. This leaves a fairly comfortable margin between output per unit time and usage consumption, making the model suitable for high-frequency, long-duration coding collaboration. The reliability of this baseline layer provides a foundation for more advanced engineering use.

    2. Typical problems and root-cause analysis

    What really affects the experience is not whether the model can be used, but whether it can be used well. The following issues recurred during long-term use.

    2.1 Response speed fluctuates significantly

    Response speed drops noticeably during daytime high-load periods and improves significantly during lower-load periods at night. Such fluctuations are often related to scheduling policies for shared compute resources. It is advisable to design asynchronous handling and timeouts, and to avoid making critical paths depend strongly on real-time responses.

    2.2 A gap between reasoning depth and perceived intelligence

    One of the deepest impressions is that the model often only handles what is immediately in front of it. Its reasoning depth is insufficient, and it may fail to consider the problem comprehensively, leading to errors. For example, when asked to check what changed in a new release, it may only capture the latest version instead of systematically organizing all changes. More capable models tend to provide a full summary. This suggests that complex tasks need to be actively decomposed, with additional context supplied at each step.

    2.3 Plans must be reviewed by humans

    All plans generated by the model need human review and re-alignment before they can be implemented. A typical example is asking the model to generate a load-testing script: the program runs, but it creates no actual pressure and does not reflect the intended load-testing goal. Another example is a requirement that clearly states some events should enable notifications by default and some should disable them, but the final implementation does not make this distinction. The root cause is that the model does not grasp intent and constraints precisely enough.

    2.4 Instruction understanding and logic can still miss the point

    Even if an instruction is slightly complex, a mature assistant should be able to infer the intended meaning. In practice, misunderstandings can occur. For example, when trying to ask for changes between a starting version and the current date, the model may understand it as asking for the current installed version number. This requires us to explicitly provide key information such as time ranges and reference points when issuing complex instructions.

    2.5 Limited adherence to rule documents

    A rules document may clearly state that the task must be run in a containerized way and provide the steps, but the model still tries to execute it directly in a local Python environment. This shows that key constraints in long rule documents can easily be drowned out. Strong constraints should be placed at the top and highlighted.

    2.6 Poor attention and boundary control in subtasks

    In multi-agent coding tools that support sub-agents, subtasks often do the wrong work or extra work: necessary work is not done, while unnecessary work is performed. In contrast, more capable models tend to have better boundary awareness in such collaborative tasks. This reminds us to set strict acceptance standards for sub-agent outputs.

    2.7 Output quality and missed visual details

    In one website redesign, a button color problem on the homepage went unnoticed for a long time and only after repeated revisions was it resolved. Since the issue could not be located through source-code analysis, it was ultimately handled by hardcoding the value in the styles. Insufficient attention to visual details is a relatively common weakness of these models.

    3. Practical efficiency strategies

    In response to the issues above, the following engineering practices are useful:

    1. Add human gates. After a plan is generated, do not execute it immediately. Add a request such as “First explain how you plan to do it and what the final result will look like.” This effectively inserts a human confirmation step into the automated process.
    2. Make constraints and reference points explicit. When issuing instructions, include key information such as time ranges, version ranges, and boundary conditions to reduce the model’s room for free interpretation.
    3. Put strong constraints first. Place rules that must be followed at the very top of the prompt and highlight them, so they are not drowned out by long documents.
    4. Break down tasks and control boundaries. Split complex requirements into small tasks with single responsibilities, and define clear inputs, outputs, and acceptance criteria for sub-agents.
    5. Use layered verification. Design separate checklists for logic, rules, and visual issues. In particular, visual details should be checked item by item.

    A simple human gate and acceptance template:

    1. [Before execution]
    2. 1. Restate the requirement and key constraints (time range / version range / default switches)
    3. 2. Explain the implementation approach and expected result
    4. 3. Start coding only after confirmation
    5. [After delivery]
    6. - Logic: whether boundary conditions are covered
    7. - Rules: whether strong constraints such as containerized execution are followed
    8. - Visuals: whether styles such as button colors and spacing are checked item by item

    4. Conclusion: leverage strengths, keep humans at key points

    The value of multimodal coding models is clear, but their capability boundaries are also clear: they are stable at the basic level and strong in frontend work, while reasoning depth, intent understanding, rule adherence, and visual detail still require human oversight. A truly efficient approach is not to hand the entire task to the model, but to place developers at key checkpoints for requirement clarification, plan review, and result acceptance. By understanding the model’s boundaries and designing the collaboration workflow well, human-AI collaboration can be focused on the most critical points.