Multimodal Large Models in R&D Workflows: Six Real-World Failure Modes and Practical Fixes
Drawing on a month-long, high-intensity evaluation, this article examines where multimodal large models hold up in software development and where they break down. These models have cleared the basic-usability bar—strong on output completeness, front-end generation, and cost-efficient long runs—but they fall short in judgment and execution across long task chains. The article breaks down six concrete failure modes (response-rate fluctuation, weak planning without a human gate, rule-reading failures, uncontrolled sub-agent delegation, shallow logical and temporal reasoning, and insufficient visual-detail discrimination) and pairs each with low-cost engineering countermeasures such as time-window scheduling, contractual rules with runtime checks, explicit scope boundaries, absolute time semantics, and value-first visual assertions. It ends with a tiered guide for matching models to task types and a checklist for hardening the process so models deliver reliably regardless of their capability.
1. The Bottom Line: One Trial Isn’t Enough
Connecting a large model to a development pipeline—“can it be used?”—is too vague a question. What actually matters is whether it can reliably produce deliverable engineering results: where does it save time, and where does it create extra work?
I kept a high-intensity, real-world usage log over roughly a month: continuous coding and refactoring sessions running three hours without interruption, with cumulative token consumption reaching the billion scale. This covered several typical scenarios—front-end interface refactoring, version-change tracking, stress-test script generation, and environment troubleshooting against a rules document. From these records, here is the headline judgment:
These models have passed the basic-usability bar, but they did not comprehensively outperform the previous generation of general-purpose flagship models on these tasks. They behave more like a strengthened “long-chain executor”—and long-chain execution is exactly where intelligent decay shows up most easily in engineering.
By “long chain” I mean completing a full loop in a single session: read the rules → make a plan → edit multiple files → self-test → correct. The model performs well on short problems (no garbled text, clean front-end code structure, stable responses), but as the chain gets longer, problems start to appear systematically. Below, I break them down by failure type.
2. What Works: Output Quality and Front-End Generation
It would be misleading to talk only about problems. The model genuinely solved some things during this period:
- Good output completeness. No garbled text, truncation, or formatting chaos. Code blocks and prose had clear boundaries. This is a valuable baseline quality for batch processing—many failures aren’t “thinking wrong” but “writing wrong.”
- Solid front-end generation. Building pages, writing component structure, and tuning basic styles produced usable results that didn’t need to be rebuilt from scratch.
- Friendly quota consumption on long runs. After three hours of continuous high-intensity use, only 86% of the paid-plan quota was consumed. For long-running batch or refactoring tasks, this consumption curve means the per-unit cost is acceptable.
Taken together, these three points show that the model isn’t short on the ability to “write” or “produce pages.” The problem is in judgment and execution.
3. Failure 1: Noticeable Response-Rate Fluctuation
This is the single most direct factor affecting the experience. During peak hours (workday business hours), responses are noticeably slower; late at night or during off-peak periods, speed improves considerably. This fluctuation gets amplified in batch tasks—a single refactoring may require dozens of tool-call rounds, and a few extra seconds per round can double the total time.
Engineering countermeasures:
- Schedule tasks by time window. Run latency-insensitive heavy work (batch refactoring, dependency mapping, test completion) during off-peak hours; keep interactive debugging in peak hours, since human attention is present then and a slower pace is acceptable.
- Shorten the chain. Don’t cram too many goals into a single round. Change “fix it all at once” into “locate first, then modify, then verify.” This increases the number of rounds, but each round waits less and failed rollbacks cost less.
- Keep a fallback path. On critical paths, keep a faster model in reserve, especially when rapid trial-and-error is needed.
4. Failure 2: Weakened “Intelligence” in Planning—Missing a Human Gate
This is the most striking issue in practice. Generated plans look complete and well-structured, but the proposed solution is often merely “runnable,” not “useful.”
A concrete case: I asked it to generate a stress-test script. The program did run and exited with a normal exit code, but the script showed no actual pressure—the concurrency model was poorly designed and couldn’t produce an effective load. The danger of this kind of failure is that it doesn’t error out; it’s “silently ineffective.” If you just run it and look at the results, you may not discover until just before launch that you tested nothing.
Later I settled on a very low-cost but highly effective habit: after generating a plan, add one line—“first walk me through how you intend to do it and what the end result should look like.”
Its value is that it forces the model to make its implicit reasoning explicit. Once it must describe in natural language “how I’ll apply pressure, what the target concurrency is, what the expected output is,” the gaps in the plan (missing target metrics, an untenable load model) often expose themselves and can be corrected in advance. This amounts to adding a human gate at the cheapest stage.
Another example makes the same point: the requirements document already clearly distinguished “which events have notifications on by default and which are off,” but in the final implementation this distinction was not carried through at all—all events were handled uniformly. This shows that the model read the description but didn’t convert it into executable constraints. This closely resembles how a certain previous-generation model behaved in a specific tool environment: incomplete understanding, inaccurate analysis, and output that needed rework—so that every plan had to be manually reviewed and re-communicated before it was usable.
5. Failure 3: Rule-Reading Failures
The rules document already stated plainly that “this project runs in a container,” even with complete steps, yet the model still tried to run the host’s Python interpreter directly. This isn’t a case of “not reading it”; it’s that the rule text wasn’t converted into a runtime constraint.
The reason is that the priority of natural language is unstable in the model’s attention: the relative weights of the rules file, the code context, and the user’s current-turn instruction shift with context length. The longer the file, the more easily later rules get diluted.
Engineering countermeasure: make rules contractual, and back them with runtime checks.
# .rules/00-precedence.md1. All scripts in this repository must run inside a container; calling the host interpreter is forbidden.2. When any environment execution is involved, run scripts/check_env.sh first and proceed only after it passes.3. If the check above fails, stop modifying code and report to me first.
#!/usr/bin/env bash# scripts/check_env.sh — replace the model's self-discipline with machine verificationset -euo pipefailif [ ! -f /.dockerenv ]; thenecho "[ABORT] Not in a container environment; building and testing commands are forbidden" >&2exit 1fiecho "[OK] Container environment confirmed"
The key is a three-layer progression: the rules file declares priority (contract), the script makes a machine judgment (verification), and failure forces an escalation (fallback path). The model may still “forget” the rules, but the moment it tries to run tests, the script stops it immediately. Replacing “hoping it complies” with “non-compliance means it can’t run” is far more effective than repeatedly emphasizing the rules.
6. Failure 4: Attention Drift—Sub-Agent Delegation Out of Control
This problem is most obvious when using an AI coding environment with sub-agents: sub-agents repeatedly “do the wrong task” or “do extra work”—problems that didn’t appear with the previous-generation model on the same tasks.
Typical manifestations fall into three categories:
- Drifting: a sub-task modifies files outside its scope;
- Overstepping: it does things no one asked for (adding logs, renaming things, refactoring unrelated functions);
- Omission: it declares completion after handling only part of the main task’s sub-items.
The root cause is that a sub-agent inherits only part of the context, and the task boundary isn’t passed explicitly. The parent knows “change only file A,” but the sub-agent may not.
Engineering countermeasure: state “what not to do” explicitly in the task description, and attach an output contract.
Task: Refactor the token-validation logic in the auth moduleInput: src/auth/token.tsScope (do only): token expiry checks, signature validationForbidden: modifying caller files, changing function signatures, adding dependencies, altering test casesDone criteria:1. Output a list of changed files (itemized)2. Explain which requirement each change maps to3. The count of unchanged files must be 0; if the scope is exceeded, stop and ask first
Pair this with a machine check using git status / a diff allowlist: if a path outside the allowlist appears in the change list, judge it a failure and rerun. Replacing “the model’s self-discipline” with “auditable diffs” is the most reliable way to converge on these problems.
7. Failure 5: Logical Understanding and Time-Semantic Parsing
The hallmark of this class of problem is that the instruction is a bit convoluted but a mature model should be able to “work around it.” In practice, it often can’t.
A typical example: what I wanted to ask was “what version changes occurred between 4.5 and 4.21.” The model interpreted it as “the currently installed version is 4.5—what version is that?” It treated a date range as a version number, and only checked the currently installed version, without mapping out all the version changes within the interval. On the same question, the previous-generation model’s approach was to list all versions in the interval and summarize each change item by item.
A related problem: when I asked it to check “what changed in the new version,” it looked at only the latest single version before delivering a conclusion. This “answer only the immediate step” behavior shows insufficient reasoning depth—it didn’t understand “changes” as an action requiring cumulative comparison across versions.
Engineering countermeasure: absolute baselines for time semantics + closed-interval declarations.
- For any time-related task, explicitly declare the current date first (e.g., “today is month X, day X”), converting relative expressions into absolute ones;
- Write interval requirements as “from A to B, inclusive of A and B, N versions in total, please list them one by one”;
- Replace vague expressions like “the new version” with “the most recent N versions,” with N fixed.
These three rules essentially turn the model’s implicit inference into explicit input. The closer instructions are to a machine-readable structure, the smaller the parsing deviation.
8. Failure 6: Insufficient Visual-Detail Discrimination
This is the most on-point capability for a multimodal model, and also where the gap is largest.
During a website overhaul, the color of the homepage button kept recurring; the model went through several rounds of changes without detecting the problem. After a lot of back-and-forth, the issue was finally resolved by my hard-coding the style directly in the HTML style attribute. In hindsight, the problem wasn’t that it “couldn’t see”; it was that it didn’t look finely enough: multimodal models can recognize layout and structure, but their judgment of fine-grained attributes like color values, contrast, and spacing isn’t stable enough; their ability to analyze source code (especially compiled style chains and cascade overrides) is also weak, so fixes stay incomplete.
Engineering countermeasure: for visual checks, “compute first, then look.”
Don’t treat a visual problem as “let the model look at a screenshot.” Treat it as “establish executable acceptance conditions”:
// Playwright example: compute the style first, then lookconst box = await page.locator('button.submit').boundingBox();const style = await page.locator('button.submit').evaluate(el => {const s = getComputedStyle(el);return { bg: s.backgroundColor, fg: s.color, radius: s.borderRadius };});// Assert: matches the design specexpect(style.bg).toBe('rgb(22, 93, 255)');// Assert: contrast meets readability requirements (illustrative only; compute against the actual standard)
The concrete approach:
- First translate visual requirements into machine-checkable values—color values, font sizes, spacing, border radius, contrast;
- Write assertions that fail, rather than letting the model “try changing it again”;
- Then let the model look at the screenshot for an overall review, at which point it takes on the coarse-grained judgment of “does the overall look make sense,” not precise color-value verification;
- Avoid inline-style fallbacks. Hard-coding styles directly in the HTML bypasses the problem rather than solving it and leaves hard-to-maintain technical debt—always go back and put it in the right place in the style system afterward.
9. Which Model to Use When: A Tiered Guide
Synthesizing a month of records, I can give a rough tiered profile:
- General-purpose flagship models: suited to logical understanding, multi-version comparison, rule-following, and long-chain tasks that need reasoning depth. They’re the default first choice.
- Long-chain-execution models: suited to batch, clearly bounded, machine-verifiable tasks—batch refactoring, boilerplate code, script generation to clear specs, large-scale document organization. In these scenarios, their execution stability is an advantage.
- Multimodal capability: suited to “coarse-grained visual understanding + structured reconstruction,” but must be paired with numeric assertions for acceptance. Don’t expect it to do pixel-level proofreading.
The key to choosing isn’t the benchmark; it’s whether the task requires cross-step reasoning. If it does, choose the stronger reasoning model; if it doesn’t, write the acceptance criteria clearly and trade process for results.
10. Putting the Engineering Process in Place
In this final section, I compress the above experience into an executable process checklist:
- Add a gate at the planning stage: after a plan is generated, force it to “first explain the approach and expected result,” and release only after human confirmation.
- Make rules contractual at the rules stage: write rules as a document with numbered priorities, add machine-check scripts, and use runtime interception instead of verbal emphasis.
- Add boundaries at the execution stage: state “what not to do” explicitly in the task description, backed by machine audits via a diff allowlist.
- Make time semantics absolute: declare the baseline date, close the interval, specify the version count.
- Compute values at the acceptance stage: turn visual and style issues into assertions first, then let the model do an overall review.
- Schedule by time window: put heavy work off-peak, keep interactions in peak hours, and split long tasks into short chains.
The core idea is one sentence: Turn every instance of the model’s “uncertainty” into a “verifiable” step in the process. The capability boundaries of multimodal models are still blurry today, but their entry into the development process is irreversible. Rather than debating whether they’re smart enough, it’s better to design engineering constraints that are solid enough that they can deliver reliably whether or not they’re smart.
