Native Multimodal LLMs Reshape the AI Coding Loop and Cost Structure
Native multimodal large language models (LLMs), combined with sparse-activation (MoE) architectures and falling inference costs, are changing how AI-assisted programming works. This article explains why capability and price are decoupling, how sparse architectures separate model capacity from compute, and how "native" multimodality differs from bolt-on vision setups. It describes a visual feedback loop that lets a model render code, screenshot the result, find problems, and patch them on its own—the key step in moving AI coding from generating code to delivering usable products. It also reframes cost around completed tasks rather than per-token prices, compares open-weight and hosted-service options, and provides a practical engineering checklist.
Introduction: A New Cost-Performance Line Is Forming
A recently released open-weight model is pushing on three fronts at once—general capability, multimodal capability, and inference cost—with coding performance close to the top international tier while priced an order of magnitude lower. Its technical approach has several distinctive traits:
- A new architecture with a limited parameter count, where only part of the parameters are activated on each inference pass;
- Vision capability built in natively during pretraining rather than added on afterward;
- Weights released under a permissive license;
- A vendor announcement that all inference traffic runs on domestic (China-based) chip clusters.
Community comments included more than a few variations on “GLM has finally come to life.” For developers, the point that matters is not the score on any single leaderboard, but the visual feedback loop that emerges once these changes stack up: after writing UI code, the model can screenshot the result, find its own mistakes, and fix them. This loop is becoming a critical link in taking AI coding from “generating code” to “delivering usable products.”
1. Why the Cost-Performance Inflection Is Happening: Capability and Cost Are Decoupling
For a long time, the industry assumed “frontier capability = high price.” That equation held because capability gains came mainly from scaling up dense parameter counts, and every forward pass had to use all of the parameters. Inference cost rose roughly linearly with size and eventually showed up in price.
That assumption breaks when three technical threads advance together:
- Architecture: sparse activation separates total parameters (which determine knowledge capacity) from activated parameters (which determine computation);
- Engineering: operator optimization, quantization, parallelism strategies, and batch scheduling in inference frameworks keep driving down the cost per unit of compute;
- Compute supply: maturing software ecosystems for domestic (China-based) AI accelerators create more options for inference supply.
The result is that capability can climb while price falls. Developers should stop using “more capable always means more expensive” as a budgeting heuristic. Instead, treat the model as schedulable compute and allocate it dynamically according to the value of each task.
2. Sparse Activation Architecture: Decoupling Capacity from Compute, and Its Engineering Cost
2.1 Two Key Parameter Curves
Sparse-activation architectures such as Mixture of Experts (MoE) split model parameters into multiple “experts,” where each token is processed by only the few experts selected by a routing network. Taking a 300B total-parameter model that activates only about 10% per pass as an example:
- Knowledge capacity depends on the total parameter count: more parameters can hold more knowledge and skills;
- Per-token compute depends on the activated parameter count: it roughly determines the number of multiply-add operations per token, and it is the basis on which vendors compare against larger models in terms of “equivalent compute.”
So under the same per-token compute budget, a sparse model can hold more parameters. This is the technical root of the “small activation, big capability” marketing claim. But the advantage has clear boundaries:
- Memory footprint is set by total parameters: even for MoE models, all expert weights usually have to stay resident in memory, otherwise weights get shuttled back and forth as tokens switch between experts. The hardware bar does not drop in proportion to the activated parameters;
- Communication overhead rises: expert parallelism introduces heavy inter-node communication, demanding more of network bandwidth and interconnect topology;
- Scheduling complexity rises: load imbalance can leave some devices idle, so the inference framework needs finer-grained batching and routing strategies.
2.2 The Core Training Constraint
The central challenge in training is load balancing: if routing consistently favors a few experts, the rest are undertrained and model capacity is wasted. Common approaches include:
- Adding a balancing auxiliary loss to the training objective;
- Setting expert capacity limits and handling overflow tokens;
- Using noise perturbation to encourage exploration.
To improve communication efficiency in large-scale training, the industry has also developed parallel approaches such as fine-grained expert partitioning. Engineering teams need to understand these mechanisms to accurately estimate inference cost and avoid being misled by the impression that “activated parameters are small.”
2.3 Practical Recommendations
- Base capacity planning on total parameters and measured memory footprint, not just activated parameters;
- Cover typical input/output lengths in load tests, along with throughput, time-to-first-token (TTFT), and overall latency distribution under high concurrency;
- For self-hosted deployments, pay close attention to routing balance and the communication efficiency of expert parallelism;
- For hosted services, favor access methods that bill per token and support batch inference and caching.
3. Native Multimodality: From “Can Look at Images” to “Born to See”
AI coding has long had a structural flaw: language models interact with the world through code as a medium, but for a huge amount of software—frontends, mobile apps—correctness is first and foremost visual: layout alignment, color harmony, responsive behavior, and clarity of information hierarchy. A text-only coding model is like an engineer who cannot see the finished product and can only infer the outcome from a written specification.
The industry has pursued two main paths to close this gap:
- Bolted-on approaches: attach a vision encoder outside the language model, mapping image features into the language space through a projection layer, or have the model call a separate vision capability to handle image tasks;
- Native multimodal approaches: train images and text jointly as a unified sequence during pretraining, so vision and language capabilities co-evolve within the same set of parameters.
The differences show up in three areas:
- Cross-modal instruction following: UI review often needs compound instructions like “only check the first screen,” “check button contrast,” or “find overlaps.” Long-term training on mixed image-text data makes native models better at mapping natural-language constraints precisely onto visual details;
- Fine-grained grounding and association: reasoning over an interface screenshot together with the DOM structure, requirements, and edit history in a single context, native models more easily link “a particular button” to its code location;
- Context consistency: visual information shares the same attention mechanism with code and logs, with no need to shuttle context back and forth between separate models.
It is worth stressing that “native” is not free of cost:
- Images get split into large numbers of visual tokens, which is especially pronounced for high-resolution screenshots and can accumulate quickly in long conversations;
- Text and color values in high-fidelity screenshots may be lost at low resolution, and repeatedly downsampling during generation further erodes detail;
- The data mixing ratio for joint image-text training is harder to get right.
So native multimodality is the direction with the higher capability ceiling, but the final result depends on the vision-input strategy used at inference time, which the next section addresses concretely.
4. The Visual Loop: Let AI See the Result, Find Problems, and Fix the Code Itself
4.1 The Basic Loop
A visual feedback loop is essentially a “perceive–decide–act–verify” cycle, broken down into these steps:
- Render: build the page or app under a fixed viewport and device configuration;
- Capture: gather visual evidence of key states, including screenshots at each viewport, DOM snapshots, console errors, and resource-loading status;
- Review: a multimodal model checks item by item against the requirements and UI specifications, producing a list of issues with locations and severity levels;
- Fix: hand the issue list along with screenshots and relevant source code to the model to generate a patch;
- Re-verify: rebuild, re-render, and re-screenshot to confirm issues are closed and no regressions were introduced;
- Converge or escalate: if it passes, stop; if the iteration cap is exceeded or problems recur, output a list for human confirmation.
4.2 An Illustrative Implementation
The pseudocode below shows a minimal working loop. Parameters such as screenshot resolution, DOM scope, and iteration cap need to be tuned to the actual scenario, and the exact return format of the visual review model must be adapted to whichever service you choose.
# Visual feedback loop sketch: screenshot -> structured checks -> visual review -> fix -> re-verifyVIEWPORTS = [(1440, 900), (768, 1024), (375, 812)] # desktop / tablet / phoneMAX_ITER = 3 # iteration cap to prevent runaway costCHECKPOINTS = ["/", "/pricing", "/settings"] # key pages or routesdef build_with_visual_loop(spec, browser, model, reporter):"""spec describes the requirements and UI specifications; browser wraps a headless browser'sscreenshot and DOM reading; model is a model with native multimodal capability; reporterproduces the final report."""for i in range(1, MAX_ITER + 1):patch = model.generate_code(spec, feedback=spec.feedback)browser.apply(patch)browser.build()issues, visual_tokens = [], 0for route in CHECKPOINTS:for w, h in VIEWPORTS:browser.goto(route).viewport(w, h)shot = browser.screenshot(full_page=(h > w)) # full-page screenshot for tall mobile pagesdom = {"console_errors": browser.console_errors(),"overflow": browser.detect_overflow(), # elements with horizontal overflow"missing_assets": browser.broken_assets(), # failed image/font loads"labels": browser.control_texts(), # empty buttons, duplicate text, etc.}verdict = model.review(image=shot, dom=dom, requirement=spec.text, route=route)visual_tokens += verdict.input_tokens # visual tokens are billed and must count toward total costissues.extend(verdict.issues)if not issues:reporter.success(attempt=i, tokens=visual_tokens)returnspec.feedback = issues # the issue list feeds back as input context for the next roundreporter.needs_human(spec) # iteration cap exceeded; hand off to human confirmation
In this sketch, reporter corresponds to logging and observability in practice; a combination of a logging service plus monitoring and alerting is recommended. Recording the number of issues, token consumption, and elapsed time for each round helps pinpoint where the loop stalls and gives early warning when cost goes off the rails.
4.3 Reducing the Instability of Visual Review
Visual review is not as simple as “hand a screenshot to the model.” A few factors strongly affect the quality of its judgments:
- Resolution and clarity: too-low screenshot resolution loses text and fine-line detail; capture at the target device pixel ratio (e.g., 2x), and prefer full-page screenshots on mobile to avoid the fragmented information a fixed viewport produces;
- State coverage: capturing only the first screen is usually not enough—empty states, loading states, error states, long lists, and modals should all have screenshots;
- Specification constraints: putting design specs, mockups, or component-library screenshots in the context as reference answers significantly reduces “subjective taste” misjudgments;
- Context bloat: visual tokens accumulate across rounds, so keep only the latest screenshots and a structured summary each round, compressing historical issues into a numbered list;
- Decision criteria: give explicit thresholds for “acceptable” versus “needs fixing,” for example not flagging spacing deviations under 2px;
- Handling uncertainty: issues the model marks as “uncertain” should be escalated for human confirmation, not passed by default.
5. Cost per Task: The Real Measure of Cost-Performance
Traditionally, people compared cost-performance using “price per million tokens.” But in a visual-loop scenario, that metric is far from enough. What developers really care about is “how much it costs to complete one usable task.” Roughly:
task cost = cumulative input tokens x input price + cumulative output tokens x output price+ number of iterations x per-iteration overhead (render, screenshot, visual review)+ human intervention cost
The key point is that a visual loop multiplies token consumption:
- each iteration re-inputs the requirements, code context, and historical issue list;
- each screenshot is a substantial visual input;
- each fix patch produces new output tokens.
So a model that is “close in capability but an order of magnitude cheaper” delivers even more value across multiple visual iterations, making automated acceptance workflows that were once too expensive to be practical economically viable.
On the calling side, a few general techniques can push down cost per task:
- Tiered routing: route simple, repetitive tasks to a lightweight model or rules engine, and reserve the parts that need visual reasoning for the multimodal model;
- Cache reuse: reuse prior results directly for unchanged pages or viewports, and cache stable system prompts and design specifications with techniques such as prompt caching;
- Context compression: replace full screenshot history with an issue list;
- Convergence control: set iteration caps and timeout budgets.
For specific billing rules, defer to the latest documentation of the service you choose.
6. Domestic Compute Adaptation: A New Variable in Inference Supply
Some vendors have announced migrating all inference traffic onto domestic (China-based) AI accelerator chip clusters. For developers, this means at least three things:
- Supply resilience: inference compute is no longer solely dependent on a few hardware lines, which helps reduce service risk from supply volatility;
- Cost headroom: domestic chips are becoming competitive on per-unit compute cost for inference workloads, especially batch and throughput-oriented loads;
- Compliance convenience: for industries with requirements around in-country data processing and private deployment, local compute more easily satisfies regulatory requirements.
At the same time, stay rational: hardware is only the foundation; actual performance still depends on operator coverage, compiler optimization, parallelism strategies, and cluster scheduling. The same model can show markedly different throughput and latency on different compute platforms. Base selection on measured data under your business traffic, not on hardware spec sheets.
7. Open Weights vs. Cloud Services: Trade-offs Between Two Routes
Open weights under a permissive license give enterprises more options:
- Private deployment: suited to data-sensitive scenarios needing deep customization and continuous iteration, but you bear the cost of compute procurement, operations, and upgrades yourself;
- Hosted cloud calls: no operations burden, good elasticity, fast iteration—suited to quick validation and businesses with volatile traffic;
- Hybrid architecture: privatize the core path while calling cloud services for elastic overflow, a compromise many mid-to-large teams choose.
Whichever route you take, build your own evaluation set: pick 50–100 real business tasks covering typical pages, devices, and boundary conditions, run regression tests regularly, and observe how version iterations affect results. Public leaderboard scores are only a reference, because the question distribution and prompting style of leaderboards often differ substantially from real business.
8. Engineering Implementation Checklist
- Define the loop’s boundaries: pilot in high-value, automatically renderable scenarios first (such as marketing pages and admin systems), then expand to complex interactions;
- Standardize inputs: give requirement descriptions, design specs, and component conventions structured templates to reduce the model’s misreading;
- Prefer automated verification: do not hand visual review problems that deterministic methods—DOM checks, layout assertions, accessibility scans, performance metrics—can catch;
- Human-machine collaboration: visual review gives suggestions, but merge requests still need human review, especially for sensitive areas like payments and permissions;
- Security and privacy: screenshots and logs may contain user data or internal information; desensitize them and encrypt them in storage and transit;
- Full-chain observability: record each iteration’s token consumption, issue types, and elapsed time, forming a cost dashboard and alerts.
Taking Baidu AI Cloud as an example, you can use a container platform to host build and render environments, object storage to archive screenshots and design assets centrally, and a message queue to decouple the build, screenshot, and review stages, then combine a logging service with monitoring and alerting for full-chain observability. If you choose a hosted model service, you can bring multimodal calls into a unified authentication, rate-limiting, and metering system using a general-purpose LLM platform’s capabilities.
9. A Rational View: The Distance Between Leaderboard Scores and Business Results
General-capability leaderboards can reflect a model’s relative position across broad tasks, but a few caveats apply:
- Uneven capability dimensions: coding, reasoning, visual understanding, and long context do not always improve together; leading in one direction does not mean leading in all;
- Evaluation environment differences: sampling parameters, prompt templates, and tool-calling permissions all affect scores;
- Business distribution differences: the UI complexity and domain-knowledge demands of real business often differ from public leaderboards.
The more reliable approach is “use leaderboards to set direction, use your own evaluation set to make the selection.” Measure “task success rate × speed × cost” on real tasks, then decide the model orchestration strategy.
Conclusion
The maturing of native multimodal capability gives AI coding, for the first time, the ability to “see its own finished product.” Combined with the cost reductions from sparse-activation architectures and domestic compute, the visual loop is moving from the demo stage into daily productivity. For developers, what is needed is no longer a single judgment of “which model is strongest,” but an engineering method oriented toward task value: use automated verification to reduce uncertainty, use the visual loop to extend the boundary of capability, and use an observable cost system to support long-term operation. Whoever makes this loop more stable, cheaper, and more maintainable will benefit first in this round of efficiency transformation.
