Evaluating Multimodal Code Models: Capability Levels, Boundaries, and Governance

    Multimodal code models can understand images and generate code, but their practical value depends on task scope, validation, permissions, and cost. This article presents a five-level capability framework, reviews common strengths and failure modes, and outlines engineering controls for adopting these models as bounded, semi-autonomous development assistants.

    1. The Question Is Not “How Strong Is It?” but “Where Should It Be Used?”

    Discussions about multimodal code models often fall into two extremes. Some demonstrations create the impression that these models can independently handle software development tasks. A few failure cases, on the other hand, may lead teams to conclude that the models are useful only for code completion.

    Actual engineering practice usually falls between these positions. The more useful questions are:

    1. Which tasks have reached a deliverable level?
    2. Which tasks should remain assistive?
    3. What engineering gates are required to keep uncertainty within an acceptable range?
    4. Is the total cost per accepted task lower than the cost of human work or conventional text-only models?

    This article does not recommend or rank any specific vendor product. Instead, it abstracts common patterns observed during an anonymized pilot. The pilot covered a relatively long period and included page development, refactoring, script generation, version-change queries, rule execution, and agent collaboration. These observations are therefore better understood through a combination of capability levels and engineering governance.

    2. A Practical Capability Framework

    To avoid vague judgments such as “it feels powerful,” code-model participation in the development process can be divided into five levels.

    Level Capability Human involvement Typical tasks
    L1 Code completion, snippet explanation, and simple error localization Human-led, model-assisted Function completion, comment generation, single-point questions
    L2 Single-file or single-component generation, with localized changes completed under clear specifications Human provides detailed specifications and performs acceptance testing UI components, test scaffolding, localized refactoring
    L3 Multi-file task decomposition, validation in an isolated environment, and evidence-based output Human reviews the plan and changes Module refactoring, front-end/back-end integration assistance, test completion
    L4 Autonomous planning across modules or services, with anomaly detection and plan adjustment Human defines goals and boundaries, then samples results Complex migrations, architecture-level refactoring, incident remediation
    L5 Continuous production autonomy with accountability for business outcomes Human handles exceptions only Not currently appropriate as a default target

    The pilot showed relatively strong performance in page-structure understanding, single-component generation, and localized modifications under clear specifications. In some conditions, these capabilities approached L3. However, the models still required substantial human controls for cross-constraint reasoning, rule compliance, long-chain agent collaboration, and visual-detail validation.

    In production terms, this class of model is better described as:

    A semi-autonomous development assistant with strong point-generation capabilities, weak cross-module engineering capabilities, and a mandatory need for engineering guardrails.

    This does not mean treating the model as an independently deliverable senior engineer. Nor should it be limited to low-value completion tasks. A more effective approach is to reduce task size, clarify specifications, automate validation, restrict permissions, and use workflows to amplify the model’s strengths.

    3. Positive Results and Boundaries Observed in the Pilot

    3.1 Strong Basic Generation and Page-Development Capabilities

    The pilot showed several clear strengths:

    • Basic generation was generally fluent when Chinese and code were mixed, with relatively few garbled or unreadable outputs.
    • The model understood page structure, layout intent, and visual elements better than text-only models.
    • When screenshots, design descriptions, and existing code were provided together, it could efficiently generate page skeletons and propose styling adjustments.
    • Response stability was acceptable when workloads remained steady and the context did not grow excessively.

    These results suggest that multimodal input is more than a visual gimmick. In page development, information about spacing, alignment, hierarchy, color, and component relationships can be expensive to describe in text. Screenshots and design files can substantially reduce communication overhead.

    However, generating a page that “looks similar” does not mean consistently delivering an implementation that conforms to the design system, remains maintainable, supports accessibility, and passes regression testing. Visual regression, computed-style validation, and code review are still required.

    3.2 Latency and Quotas Are Engineering Concerns

    Two patterns were visible during the pilot: long-running sessions consumed quotas quickly, and complex multimodal inputs combined with long execution chains produced significantly higher end-to-end latency than ordinary text tasks. Latency also varied across different periods.

    The causes are generally predictable:

    • Images, screenshots, and long contexts increase input-processing costs.
    • Agent-style execution includes multiple planning rounds, tool calls, result reads, and corrections. Its end-to-end latency is therefore much higher than that of a single generation.
    • Concurrent tasks create resource queues, while averages can hide long-tail latency.
    • Failed retries, repeated reads, and context growth further increase cost.

    Capacity planning should therefore not rely only on average response time or estimate usage from the remaining quota after a single session. Teams should record:

    • Input and output tokens per task;
    • Tool-call count per task;
    • Time to first runnable result and time to final acceptance;
    • p50, p95, and p99 latency;
    • Total cost per accepted task;
    • Additional cost caused by failed retries.

    Non-real-time tasks can be handled in batches. Tasks such as page generation and screenshot understanding may require multimodal input, while log analysis, version queries, and text refactoring can usually be routed to conventional text models or lighter workflows.

    The central principle is:

    Choose the input modality by task instead of attaching screenshots and the full context to every task.

    3.3 Insufficient Planning Depth Is the Most Important Warning Sign

    The most serious issue observed in the pilot was not that the model could not write code. It was that a seemingly reasonable plan could produce an engineering result with little practical value.

    For example, a generated load-testing script might run successfully but lack a realistic concurrency model, bottleneck metrics, failure thresholds, or result analysis. The output would be an executable program, but it would not answer the actual testing objective.

    The root causes are usually not syntax-related:

    • The model does not ask enough questions or clarify the objective.
    • Business acceptance conditions are not translated into executable checks.
    • The difference between “runnable” and “effective” is not properly addressed.
    • Coding begins too early in a multi-step task, before the plan is reviewed.
    • Hard constraints in the context are not extracted into a checklist.

    A useful control is a planning gate. Before writing code, the model must provide a plan, file list, acceptance evidence, and risk analysis. Human approval is required before execution begins.

    The lightweight instruction “Explain what you plan to do and what the result should look like” is useful, but it should be structured rather than treated as an informal request.

    An example task contract is:

    1. task_contract = {
    2. objective: "Refactor a module and pass the specified acceptance checks",
    3. scope: ["Modify only the module directory", "Do not change public interfaces"],
    4. invariants: [
    5. "The default notification policy must match the policy table",
    6. "Runtime execution must use an isolated container environment",
    7. "Do not bypass the design system with hard-coded inline styles"
    8. ],
    9. acceptance: [
    10. "Unit and integration tests pass",
    11. "Key pages pass visual regression checks",
    12. "Date-range queries cover all boundary conditions",
    13. "Provide a change summary and rollback notes"
    14. ],
    15. evidence: ["Test report", "Visual-diff report", "Execution logs", "Known risks"],
    16. budget: {"max_files": 8, "max_steps": 30, "retry_limit": 2}
    17. }

    Without a task contract, the model tends to focus only on the immediate action. With a contract but no human gate, it may execute an incorrect plan to completion. Both controls are necessary.

    4. Common Failure Modes and Governance Measures

    4.1 Rules Are Read but Do Not Become Hard Constraints

    One observed failure involved a rule document that explicitly required execution inside a container and provided the relevant steps. The model still attempted to use the local interpreter. This was not simply a memory problem. The rule remained natural-language context instead of becoming a pre-execution check, and the runtime environment did not enforce it.

    Governance should operate at three layers:

    1. Extraction layer: Before work begins, require the model to list hard constraints, their sources, validation methods, and the response to a violation.
    2. Execution layer: Route sensitive operations through wrappers that permit execution only in an isolated environment. Prevent direct use of the host interpreter or unrestricted access to host directories.
    3. Audit layer: Log every tool call, working directory, environment variable, and exit code. Abort immediately when a boundary is crossed.

    A prompt saying “Please follow the rules” is not sufficient. Rules must exist simultaneously in the task contract, execution tools, and audit logs.

    4.2 Time, Version, and Scope Semantics Are Easy to Misinterpret

    A request such as “query all version changes between two dates” may be misread as “query the currently installed version.” This type of error is common in natural-language requirements because date ranges, open and closed intervals, version ordering, and data sources may all be ambiguous.

    A safer interaction pattern is to require the model to convert natural language into structured conditions first:

    1. query = {
    2. entity: "version change records",
    3. range: {
    4. start: "April 5 00:00",
    5. end: "April 21 23:59",
    6. inclusive: "both endpoints included"
    7. },
    8. scope: "all versions released in this interval, not only the latest version",
    9. output: "summarize changes in ascending time order and cite the source",
    10. validation: "if records are missing or the range is ambiguous, ask first and do not guess"
    11. }

    Templates should explicitly distinguish terms such as “latest version,” “most recent versions,” “installed version,” “target version,” and “release-date range.” Requirements involving dates, amounts, permissions, defaults, or compatibility should always include boundary cases.

    4.3 Defaults and State Matrices Are Not Properly Implemented

    Another common failure occurs when a requirement specifies that some events should have notifications enabled by default while others should be disabled, but the implementation does not distinguish between them. This usually means the model understood the main flow but did not convert conditional rules into a testable state table.

    Implicit rules should be made explicit:

    Event type Notification default User-configurable Acceptance method
    Type A Enabled Yes Unit tests cover the default and disabling then re-enabling
    Type B Disabled Yes Unit tests cover the default and manual enabling
    Type C Enabled and cannot be disabled No Validate at both the API and UI layers

    When a requirement contains phrases such as “some events” and “other events,” it should not remain a paragraph of prose. The model should first produce a state matrix, exception branches, and test cases before coding begins. This converts an attention problem into verifiable engineering artifacts.

    4.4 Agent Collaboration Can Exceed Boundaries and Duplicate Work

    In agent mode, common problems include sub-agents misunderstanding their responsibilities, making extra changes, repeatedly reading the same information, or retrying unnecessarily.

    Typical causes include:

    • The main agent provides a vague goal without defining the subtask boundary.
    • The sub-agent has no independent acceptance criteria.
    • Tool permissions are too broad, allowing any executor to modify the entire repository.
    • There are no limits on steps, files, or retries.
    • No global change ledger exists, resulting in duplicate modifications.

    A sound rule is:

    One executor, one responsibility, one deliverable, and one body of evidence.

    The main workflow should decompose, review, and integrate tasks. Subtasks must not expand their own scope. A simplified process might look like this:

    1. run_task(contract):
    2. constraints = extract_constraints(contract, docs)
    3. plan = draft_plan(contract, constraints)
    4. require_human_approval(plan)
    5. for step in plan.steps:
    6. assert step.scope <= contract.scope
    7. result = execute_in_sandbox(step)
    8. evidence = verify(result, step.acceptance)
    9. if not evidence.pass:
    10. rollback()
    11. return ask_human(evidence)
    12. return assemble_evidence(plan, results)

    The goal is not to make the process unnecessarily complex. Every step should be verifiable, interruptible, and reversible.

    4.5 Runnable Does Not Mean Effective

    Load-testing scripts are a typical example. A script that runs successfully is not necessarily capable of generating meaningful load, and printed output does not necessarily identify a bottleneck.

    For performance, capacity, and reliability tasks, acceptance criteria should cover at least:

    • The system path under test and its boundaries;
    • The concurrency model, request distribution, and data volume;
    • Success and failure definitions;
    • Metrics such as latency, throughput, and error rate;
    • Logs and monitoring points required for bottleneck analysis;
    • The result report and conditions required to reproduce the experiment.

    If the model delivers only a script without an experiment design, the task should be considered incomplete. Templates can require the model to state its testing assumptions, metrics, load model, and expected outputs before generating the script.

    4.6 Visual Details Need an Independent Acceptance Path

    During page refactoring, details such as button colors, spacing, and hierarchy often require repeated adjustments. A model may understand the overall page structure but fail to consistently detect that a color does not match a design token, that contrast is insufficient, or that styles shift at different breakpoints. Hard-coding the final adjustment as inline CSS only bypasses the problem and creates additional maintenance debt.

    A three-layer visual acceptance process is recommended:

    1. Screenshot baseline comparison: Store baselines for key pages and breakpoints, then check pixel and layout differences after changes.
    2. Style-source validation: Verify that colors, font sizes, and spacing come from design variables. Disallow unexplained inline styles.
    3. Accessibility and interaction checks: Validate contrast, focus states, keyboard operation, and error messages.

    Visual regression should be part of task acceptance rather than a final manual check. Otherwise, the model may repeatedly apply local patches until the result becomes a difficult-to-maintain collection of style debt.

    5. Evaluation Metrics: From “Feels Intelligent” to Measurable Delivery

    Model capability should not be judged solely by demonstrations or isolated failures. Teams should establish metrics by task type, including:

    • First-pass acceptance rate: The percentage of tasks accepted without human rewriting;
    • Plan rework rate: The percentage of plans requiring human modification or another clarification cycle;
    • Rule-violation rate: The number of path, permission, or environment violations per 100 tool calls;
    • Defect escape rate: The percentage of functional, visual, or security issues discovered only during review;
    • Human intervention count: The average number of clarification and correction rounds per task;
    • Unit task cost: The sum of input/output token costs, tool execution time, and human review time divided by the number of accepted tasks;
    • p95 completion latency: A more realistic indicator than average latency;
    • Rollback and recovery time: How quickly the system can recover after an error.

    Weights should differ by task type. Page generation should place greater weight on visual regression and first-pass acceptance. Backend refactoring should emphasize test coverage, interface compatibility, and rule violations. Agent-collaboration tasks should focus on unauthorized execution, retry counts, and evidence completeness.

    6. Adoption Path: From Read-Only Assistance to Bounded Autonomy

    A cautious adoption path can proceed in stages:

    1. Shadow stage: The model analyzes and recommends but does not write code. Collect data on plan quality, misunderstanding patterns, and completeness of acceptance evidence.
    2. Suggestion stage: The model outputs patch suggestions, which humans apply. Measure the mergeability of those suggestions.
    3. Sandbox write stage: The model may modify code and run tests in an isolated environment, but human approval remains mandatory before merging. Focus on rule violations and scope expansion.
    4. Bounded autonomy stage: Allow automatic execution only for low-risk, easily reversible tasks with stable acceptance checks, such as documentation updates, test completion, and changes to non-critical page components.
    5. Core production paths: Do not delegate directly until stable rollback, auditing, and human takeover mechanisms are available.

    Each stage should have exit criteria: no serious rule violations across multiple iterations, complete acceptance evidence, lower cost per accepted task than the baseline, and an acceptable human review burden. Thresholds should be based on the team’s historical data rather than arbitrary assumptions.

    7. Suitable and Unsuitable Use Cases

    Tasks suitable for early pilots include:

    • Page skeleton and localized component generation;
    • Initial conversion from design files to page structures;
    • Clearly defined single-file or small-scope refactoring;
    • Test cases, test scaffolding, and documentation drafts;
    • Log summaries, change descriptions, and code explanations;
    • Repetitive refactoring supported by automated tests.

    Tasks requiring caution or mandatory human gates include:

    • Architecture changes spanning modules or services;
    • Security, authorization, billing, and default-policy code;
    • Direct implementation of highly ambiguous requirements;
    • Automatic production incident remediation;
    • Large-scale dependency upgrades and data migrations;
    • Page refactoring with strong visual and interaction requirements but no regression baselines;
    • Execution tasks that must strictly follow runtime-environment or compliance rules.

    8. Conclusion: Treat It as a Semi-Autonomous Development Assistant

    What capability level do these multimodal code models represent?

    The most accurate answer is not a single absolute score but a scenario-based assessment:

    • They provide strong assistance, and sometimes semi-autonomous capability, in page understanding, localized generation, and code modification under clear specifications.
    • They can perform part of a multi-file engineering task when supported by a task contract, plan review, sandbox execution, and automated testing.
    • They still require human gates for cross-module reasoning, rule compliance, agent collaboration, visual-detail validation, and long execution chains.
    • For the general task structure represented by this pilot, they are best characterized as semi-autonomous development assistants between L2+ and L3, rather than independently deliverable L4 or L5 engineers.

    Production value is determined not by how impressive a single generation looks, but by whether the team can engineer task specifications, acceptance evidence, permission controls, rollback mechanisms, and cost measurement.

    Model capabilities will continue to improve. Without these guardrails, however, longer execution chains will only amplify misunderstanding, rework, and technical debt. The most practical strategy for development teams is to use clear task contracts to select suitable scenarios, use automated validation and auditing to contain risk within the workflow, and then expand the model’s autonomous scope gradually according to the cost per accepted task.