Native Multimodal Agents: From Sketch to Frontend Code

    Text-only language models struggle with visual tasks, from turning design mockups into frontend code to operating apps on a device. This article breaks down native multimodal agents, which treat vision as a first-class capability and unify perception, reasoning, and execution in one loop. It walks through the architectural trade-offs of native multimodal design, the training stages that build visual-grounded behavior, the tooling layer that connects model to environment, and a practical sketch-to-code workflow built on an interface intermediate representation. It also covers production concerns such as sandboxing, observability, and cost and latency control, along with how to evaluate these systems and what they mean for frontend development.

    1. The Starting Point: Why “Bolt-On Vision” Isn’t Enough

    Modern software products depend heavily on visual communication. Design mockups, app screenshots, screen recordings, PDF documents, and monitoring feeds are among the most common information carriers in engineering collaboration. Text-only large language models, with their strong language understanding and code generation, have become the workhorses of AI-assisted programming. But real-world tasks quickly expose two recurring difficulties:

    • Interface implementation: a design mockup needs to become frontend code; a product manager hands over requirements as a screenshot; a tester files a bug with an attached UI screenshot.
    • Device operation: a multi-step task has to be completed inside a mobile app, such as wiring a money transfer or comparing prices and placing an order on an e-commerce app, where the only available information is what appears on screen.

    The common industry approach is to “bolt on” vision alongside a text model, roughly like this:

    1. Image → [Standalone vision model 1: read text] → text description
    2. → [Standalone vision model 2: detect widgets] → coordinates and labels
    3. → [Standalone vision model 3: describe image] → natural language
    4. → Text model (concatenates the above) → code / action instructions

    This pipeline looks workable, but in practice it surfaces three structural problems.

    1. Semantic loss: lossy compression of critical information

    An interface is highly structured two-dimensional information. Layout regions, component hierarchy, alignment, spacing ratios, color systems, and widget states often cannot be fully expressed in a paragraph of text. When an image is verbalized, the model receives only a compressed approximation. It has to “guess” at visual details, so the reconstruction can end up with tangled hierarchy, missing components, or distorted styling. The problem grows more pronounced on interfaces that are text-dense or built from tiny elements.

    2. Error accumulation: errors amplify at every stage

    In a serial pipeline, every stage carries inherent error: text recognition may miss or misread characters, widget detection may miss or misdetect elements or drift off coordinates, and image description may drop key details. Downstream modules have no access to the original image to double-check, so they can only build on the results handed down from upstream, and the errors stack up. Worse, mistakes are hard to localize: when a reconstruction goes wrong, it’s difficult to tell whether the fault lies in recognition, understanding, or generation.

    3. Missing closed loop: it can generate, but it can’t execute and verify

    When the vision modules and the reasoning module are split apart, the model can’t directly perceive the outcome of its own output. In device-operation scenarios, that means after issuing a click instruction the model can’t confirm whether the click hit its target or whether the page changed as expected. Once an action goes off track, the following steps fail in sequence.

    The so-called “native multimodal agent” is a response to exactly these problems: treat vision as a first-class capability of the model, and unify perception, reasoning, and execution inside a single loop. “Native” shows up on at least three levels:

    • Architecture-native: vision and language share a representation space and attention mechanism inside the model, rather than being stitched together across an interface between two models.
    • Training-native: visual data takes part deeply in both pretraining and post-training, so the model learns to “talk about images” and “act on images” from the start.
    • Tool-native: visual capability can directly drive tools, so the model can observe execution results, judge deviations, and correct itself.

    Major model vendors are moving in this direction today. The specific implementation and effectiveness vary by technical route and data accumulation, so selection should be based on actual evaluation.

    2. Architecture: Making “Seeing” and “Reasoning” Share One Representation

    At the architecture level, native multimodal design essentially has to solve three problems: how images enter the language model, how visual detail is preserved, and how vision and language reason together. Existing industry explorations fall into roughly these categories:

    Fusion approach Basic method Main advantages Main challenges
    Late fusion (projection) A vision encoder extracts features, a projection layer maps them into visual tokens, which are concatenated with text tokens and fed to the language model Mature engineering, low implementation cost Both ends are trained separately; fine-grained alignment depends on data quality
    Mid fusion (cross-attention) The language backbone keeps its text-only ability, querying visual features on demand through cross-attention layers Text-only ability is well preserved; compute can be allocated dynamically Complex layer structure; heavy training and inference cost
    Early fusion (unified tokens) Images are split directly into patches and mapped to tokens that enter the same attention computation as text tokens Ample cross-modal interaction; strong localization ability High visual-token count at high resolution; long-context efficiency must be solved

    Whatever the route, interface and device understanding place three shared demands on the vision side.

    1. Fine-grained, high resolution: small targets determine reconstruction quality

    What actually carries meaning on an interface is often small text, icons, input-field borders, and dividers. If the image is over-compressed to keep token counts down, this information is lost at the encoding stage, and no amount of downstream reasoning can recover it. In practice, dynamic or native resolution encoding is common: the number of visual tokens adapts to image content. Simple pages can be covered with fewer tokens, while information-dense pages are allocated more, balancing precision against cost.

    2. Two-dimensional position awareness: let the model “state a location”

    Device operation requires the model to output not a vague “click the button” but a precise “click the button at coordinates (x, y).” This calls for two capabilities:

    • 2D positional awareness: inject two-dimensional position information when encoding visual tokens so that attention reflects the image’s spatial structure;
    • Localization-style output: explicitly add “referential grounding” tasks during training, for example “find the login entry in the top-right corner of the page,” so that the output corresponds to a target region.

    This localization ability matters especially for long screenshots: the model needs to judge whether a given piece of text belongs to the first screen or the bottom of the page, otherwise scrolling and clicking will miss.

    3. Joint image-text reasoning: visual evidence and language context check each other

    The value of native multimodality isn’t just being able to “see,” but to “see accurately.” Faced with a design mockup, the model has to understand three things at once:

    • Overall visual intent: what kind of page this is, and what layout and color scheme it uses;
    • Local detail: how many lines of text a given card holds, what color the primary button is;
    • Semantic constraints: whether the heading hierarchy follows conventions, whether spacing follows the rules of the design system.

    A joint attention mechanism lets the model look back at the image directly while generating code or action instructions, using visual evidence to check its linguistic reasoning. For example, when the text context suggests “this should be a selected tab,” the model can confirm its highlighted state in the image rather than assuming.

    3. Training: From Image-Text Alignment to Interaction Feedback

    Architecture sets the ceiling on capability; data and training decide whether that capability is realized. Training a native multimodal agent usually proceeds through several progressive stages.

    1. Image-text pretraining: building a cross-modal semantic foundation

    Image-text pretraining solves the “can it understand” problem, and its data spans three layers:

    • Image-text semantic alignment: pairing images with descriptions to build correspondences between visual and linguistic concepts;
    • Text and document understanding: covering text-heavy layouts such as web pages, documents, and charts, sharpening sensitivity to layout;
    • Localization and reference: questions that point to specific regions of an image, cultivating spatial localization.

    2. Instruction tuning: aligning general understanding with specific tasks

    General visual understanding doesn’t automatically equal task ability. The instruction-tuning stage needs high-quality samples close to the target scenario:

    • Interface samples: design-mockup/code pairs, screenshot/structured-description pairs, and different states of the same interface (default, hover, selected, disabled);
    • Operation samples: complete “observe-think-act” interaction traces, including both successful paths and post-failure correction paths;
    • Negative samples: wrong clicks, misread states, and repeated ineffective actions, to help the model learn to avoid common errors.

    3. Interactive training: letting the model learn from execution results

    An agent’s ability isn’t “answering correctly once,” but “noticing errors and correcting them.” Interactive training places the model in a real or high-fidelity simulated environment:

    1. The model observes the current interface and chooses an action;
    2. The environment executes the action and returns a new observation;
    3. A reward signal is given based on whether the task is complete;
    4. The reward signal is used to optimize the model’s policy.

    This training builds on the “observe-act-reward” loop from reinforcement learning. It gradually teaches the model to recognize failure signals, such as a page not changing after a click or an unexpected dialog appearing, and to adjust its subsequent strategy. It’s precisely the capability a bolt-on approach struggles to obtain: an individual vision module on the pipeline knows neither the ultimate goal of the task nor the feedback after execution.

    4. Tooling and Execution: Closing the Perception-Reasoning-Execution Loop

    For a native multimodal agent to actually do work, it also needs a tooling layer that connects the model to the environment. That layer usually contains five modules:

    • Perceiver: captures environment state, including screen captures, the widget tree or page structure tree, and video frames. In an interface environment, the element types and hierarchy from the widget tree can effectively offset the uncertainty of pure visual judgment.
    • Executor: translates the model’s high-level intent into concrete actions such as clicking, swiping, typing, or switching apps. Before execution, coordinates should be boundary-checked to avoid out-of-range or accidental system-level actions.
    • Verifier: compares state before and after execution to judge whether an action took effect or produced side effects.
    • Memory: stores historical traces and task state, and compresses or summarizes when the context grows too long.
    • Safety gate: adds a second confirmation or human approval for high-risk actions such as transfers, deletions, and authorizations.

    A typical agent execution loop can be described with the following pseudocode:

    1. # Generic pseudocode: illustrates only the structure of the agent execution loop
    2. def run_agent(task, env, model, max_steps=20):
    3. history = []
    4. for step in range(max_steps):
    5. # 1. Perceive: get the current interface state
    6. screenshot = env.capture() # screenshot (image modality)
    7. ui_tree = env.get_ui_tree() # widget tree / page structure (text modality)
    8. # 2. Reason: combine the goal and history to output the next action
    9. action = model.decide(task, screenshot, ui_tree, history)
    10. # 3. Safety: high-risk actions need human confirmation
    11. if action.risk_level == "high" and not user_confirmed(action):
    12. return {"status": "awaiting_confirmation", "action": action}
    13. # 4. Execute: hand the action to the environment
    14. before = env.state_fingerprint()
    15. env.perform(action)
    16. after = env.state_fingerprint()
    17. # 5. Verify: if state didn't change, retry or adjust the strategy
    18. effective = before != after
    19. history.append({"step": step, "action": action, "effective": effective})
    20. # 6. Reflect: decide whether the task is done
    21. if model.is_task_done(task, env.capture()):
    22. return {"status": "done", "history": history}
    23. return {"status": "failed", "history": history}

    The loop above has three key design points:

    • Complementary dual-modality perception: images provide global visual semantics while the widget tree provides precise structure and clickable elements; the two corroborate each other and significantly lower the misjudgment rate.
    • State-fingerprint verification: whether an action took effect isn’t judged by the model’s subjective opinion, but by comparing state before and after execution.
    • Diagnosable failure: a complete trace lets every failure be pinned to a specific step, making retrospection and continuous optimization easier.

    5. From Sketch to Frontend: Crossing from “Looks Right” to “Ready to Ship”

    “Generate a frontend interface from a sketch” is one of the most watched applications of native multimodal agents; the industry usually calls this kind of task “design-to-code.” Developers need a clear standard for judgment: what problems does sketch-to-code actually have to solve?

    1. Four Levels of Task Capability

    Level Capability required How to check it
    Visual reconstruction Accurately identify layout, components, copy, and colors Visual similarity between the generated page and the reference image
    Semantic understanding Understand the page’s purpose and information hierarchy; choose appropriate components and structure Whether semantic tags and component reuse are reasonable
    Engineering usability Responsive layout, interaction logic, accessibility Whether multi-device rendering, build, and lint checks pass
    Maintainability Clear code organization, unified styling system, easy to modify later Human review and the cost of later iterations

    2. The Key Is an “Interface Intermediate Representation”

    If the model generates code directly from a sketch, then whenever the result falls short, the developer can only throw it all away and redo it. A more controllable approach introduces a framework-agnostic “interface intermediate representation.” The model first outputs a structured description of the interface, and the code-generation stage then translates it into code for a specific framework:

    1. {
    2. "page": {
    3. "type": "login",
    4. "theme": { "primary": "#2F6BFF", "radius": 8 },
    5. "regions": [
    6. { "id": "header", "layout": "flex", "children": ["logo", "title"] },
    7. {
    8. "id": "form",
    9. "layout": "column",
    10. "gap": 16,
    11. "children": [
    12. { "type": "input", "label": "Username", "required": true },
    13. { "type": "input", "label": "Password", "inputType": "password", "required": true },
    14. { "type": "button", "label": "Log in", "variant": "primary", "block": true }
    15. ]
    16. }
    17. ]
    18. }
    19. }

    The intermediate representation brings three kinds of value:

    • Controllable: developers can review whether the structure matches the design intent before code is generated;
    • Reusable: the same intermediate representation can be output as implementations for different frameworks and platforms;
    • Traceable: when a reconstruction goes wrong, you can tell whether the problem is in “reading the image” or “writing the code.”

    3. Render Verification and Auto-Repair

    “Looks like it” must have an objective standard. A practical verification loop looks like this:

    1. Sketch / mockup → visual understanding → interface intermediate representation → code generation
    2. → render in an isolated environment → screenshot and compare with the reference → locate differences
    3. → local repair (adjust only the deviating region) → verification passes → final human review

    Rendering the generated page in an isolated container lets you run build checks, visual comparison, and interaction verification without touching the development environment. Repairing only the deviating region significantly reduces wasted changes and regression risk.

    4. The Gap from “It Runs” to “We Dare to Ship It”

    It’s important to view this capability’s current, typical shortcomings objectively:

    • Interaction logic: behavioral logic such as login validation, form interdependencies, and access control is hard to infer from visual information alone;
    • Data binding: the correspondence between API fields, state management, and real business data needs extra convention;
    • Long-tail styling: complex animations, special font rendering, and precise spacing and alignment are often where differences concentrate most;
    • Design-system consistency: whether the generated code follows the team’s design system and component library needs extra constraints.

    So a more pragmatic positioning is a “fast generator of high-quality prototypes and basic pages.” Developers still need to do substantive work at the data, interaction, and design-system levels.

    6. Engineering: Four Key Concerns from Demo to Production

    1. Isolation and sandboxing

    An agent really operates interfaces, and generated code really gets executed and rendered. All operations should happen in a controlled environment: use dedicated test accounts and test data, achieve environment isolation through containers, and set boundaries on filesystem and network access. High-risk actions should force human confirmation. This isn’t optional; it’s a precondition for an agent entering the production flow.

    2. Observability

    Agent behavior is multi-step and non-deterministic, so traditional logs struggle to reconstruct the full picture of a problem. It’s advisable to build a complete trace for each task:

    1. {
    2. "task_id": "t-20250612-0001",
    3. "steps": [
    4. {
    5. "step": 0,
    6. "observation_ref": "screenshot://t-20250612-0001/0.png",
    7. "thought": "Currently on the home screen; need to go to the login page",
    8. "action": { "type": "click", "target": "element:login_btn", "point": [980, 120] },
    9. "state_changed": true,
    10. "latency_ms": 860
    11. },
    12. { "step": 1, "...": "..." }
    13. ],
    14. "result": "done",
    15. "total_tokens": 18240
    16. }

    Files such as screenshots and assets can be stored in object storage and linked to trace records by reference. Combined with log services and monitoring alerts, you can set thresholds on metrics such as task success rate, average steps, latency distribution, and failure-reason share, so any fluctuation is caught in time. An often-overlooked metric is average steps: when it keeps rising while tasks still succeed, it usually means the model is “taking the scenic route,” an early signal of policy degradation.

    3. Engineering for long-running tasks

    A single task involves multiple rounds of reasoning and needs to handle queuing, concurrency, and timeouts. Common engineering measures include:

    • Decoupling task submission from execution with a message queue to smooth peak load;
    • Tiered scheduling by model size: simple pages to a lightweight model, complex pages to a more capable one;
    • Caching screenshots, intermediate representations, and generated code, and reusing them on a cache hit.

    Containerized services that carry the inference and rendering workload can scale on demand with a cloud platform’s elastic-scaling ability. General capabilities from cloud providers such as Baidu AI Cloud, including container services, object storage, log services, and monitoring alerts, can serve as infrastructure options for this kind of agent.

    4. Cost and latency control

    Optimization How it works
    Tiered task routing Choose a model of different size based on task complexity
    Context compression Summarize historical traces to avoid unbounded context growth
    Prompt prefix caching Reuse fixed system prompts and tool descriptions
    Async execution Turn long tasks into background jobs, returning results via callbacks or message notifications

    7. Measuring Results: Three Principles for Evaluation Design

    Faced with various vendors’ capability claims, developers should return to the reproducible evaluation method itself and focus on three points.

    Principle 1: Distinguish capability dimensions; avoid summarizing with a single score

    Different evaluations examine very different capabilities. For example, interface-replication tasks examine the ability to reconstruct from vision to code, while mobile-operation tasks examine the ability to complete multi-step tasks in a real device environment. The two kinds of scores can’t substitute for each other. A model might excel at interface replication yet fall short at device operation that requires long-horizon planning.

    Principle 2: Focus on stability, not a single best run

    Agents are stochastic, so multiple runs of the same task may differ. A more meaningful metric is “success rate over n runs,” not “the best single performance.” For online business, also pay attention to failure modes: is it a hang, an infinite loop, or a wrong action? Different failure modes carry very different risk and repair costs.

    Principle 3: Build an evaluation set that fits your own business

    General evaluation sets struggle to reflect a company’s own interface style, component conventions, and business flows. It’s advisable to distill an evaluation set from real needs:

    1. Suggested evaluation-set composition
    2. ├── Interface replication: standard pages / chart pages / form-dense pages / multilingual interfaces
    3. ├── Device operation: single-app single-step / single-app multi-step / cross-app flows / with error dialogs
    4. └── Text regression: code generation / code understanding / technical Q&A (ensuring text ability doesn't regress after visual ability improves)

    In addition, pay attention to the often-overlooked dimension of “keeping text ability intact.” Introducing multimodal capability shouldn’t come at the cost of text reasoning and coding ability; regression testing should become a fixed part of evaluation.

    8. Will This Change the Programming Paradigm?

    Back to the original question: will this kind of capability trigger a programming paradigm shift? The answer is conditional, staged, and step-dependent.

    1. Interface form is changing: from “describing the implementation” to “describing the intent”

    The input to traditional interface development is written requirements and verbal description, and the developer has to make three conversions: “linguistic intent → visual structure → code.” Native multimodal agents move the input forward to a sketch or mockup, so the developer only needs to focus on “what’s wanted,” while the model takes on a large part of “how to write it.” This is similar to high-level languages replacing assembly, or visual tools replacing hand-written layout: the level of abstraction rises, but programming itself isn’t eliminated.

    2. The value center is migrating: from “writing code” to “defining and verifying”

    When the marginal cost of interface implementation drops sharply, the developer’s focus shifts to three areas:

    • Defining specs: business rules, boundary conditions, exception flows, and data contracts;
    • Verifying results: judging whether the generated result meets standards on visuals, interaction, performance, and security;
    • Architecture design: component systems, state management, performance budgets, and engineering conventions.

    Much as unit tests relate to functional code, “specs and verification” will become the core production material for agents.

    3. Constraints remain hard: which parts are hard to replace

    • Complex business logic: multi-party collaboration, permission models, and domain rules need deep business understanding;
    • Performance and engineering: first-screen loading, rendering performance, build systems, and code governance still need professional judgment;
    • Responsibility and compliance: intellectual-property, security-vulnerability, and data-compliance responsibility for generated code ultimately rests with humans.

    4. The realistic path of change: assist first, reconstruct later

    A more likely evolution is as follows:

    1. Assist stage: the agent handles component generation, page assembly, and styling adjustments, while the developer reviews and integrates;
    2. Collaboration stage: design and engineering share the same intermediate representation, communication loss between prototype and implementation drops sharply, and the review target shifts from “mockup plus code” to “a directly runnable, interactive prototype”;
    3. Reconstruction stage: some teams’ delivery flows evolve toward “spec-driven,” visual assets become a first-class input, and test coverage extends to agent-generated code.

    Whether this path works out depends on two preconditions:

    • Reliability: whether the agent’s success rate and predictability in critical business scenarios can meet production requirements;
    • Governability: whether generated code can be continuously maintained, rather than becoming a hard-to-modify “one-off artifact.”

    Only when these two conditions are met will the programming paradigm change in substance.

    9. Conclusion

    From sketch to frontend, native multimodal agents demonstrate a new way of engineering collaboration: vision is no longer an “external input” that needs extra translation, but “native information” that the model can directly understand and act on.

    For developers, the right way to view this capability rationally is to treat it as a new tool that needs to be handled with engineering discipline:

    • Understand the principles: knowing the basic mechanisms of unified representation, training strategy, and closed-loop execution is what lets you judge capability boundaries;
    • Build evaluations: quantify results with a task set that fits your own business, rather than being persuaded by a single demo;
    • Hold the line: isolation sandboxes, observability, human review, and regression testing are the basic guarantees for an agent entering the production flow.

    The change in the programming paradigm won’t happen overnight, but it has already quietly started at the point of “turning vision into executable context.” For R&D teams, the real opportunity isn’t whether they’ll be replaced, but whether they can design a “human-machine collaboration” workflow earlier.