Sparse MoE Native Multimodal LLMs: Architecture Analysis and Engineering Practice
A practical breakdown of the native-multimodal, sparse-MoE, and hybrid-attention design now showing up in a new class of large language models. Using a recently open-sourced model with roughly 320 billion total parameters that activates only about 18 billion per inference as a case study, this article explains how sparse Mixture-of-Experts routing, hybrid sparse/linear attention, and native multimodal pretraining balance capability, efficiency, and cost. It covers visual coding and office-document workflows, resource estimation and serving optimization for sparse architectures, disciplined context-budget management for million-token windows, and evaluation and risk-control practices—giving developers a reference for technology selection and deployment.
Late in August 2026, a Chinese model developer launched and open-sourced a native multimodal model. The model debuted with a sparse architecture of roughly 320 billion total parameters while activating only about 18 billion parameters per inference. It first opened for real usage on a limited, anonymous channel, and only later revealed its identity and full capabilities. Before launch, it appeared in anonymous form on an aggregator platform, quickly climbed to the top of that platform’s call-volume ranking, and became one of the most frequently called models at the time; it also set a single-platform call-volume record on a smart coding tool. It scored 57 on a third-party public benchmark, on par with mainstream international flagship models, while its usage price was only one-tenth that of the text-only flagship in the same series. This article does not focus on the specific product itself. Instead, it uses this model as a sample to systematically review the technical essentials and engineering practices of the “native multimodal + sparse MoE + hybrid attention” approach, as a reference for developers making technology-selection and implementation decisions.
1. Technical Background: Cost per Unit of Intelligence Becomes the New Battleground
Over the past two years, gains in large-model capability have advanced mainly along two paths: scaling up dense model parameters, and extending the context window. Both paths come at the cost of inference cost that grows nearly linearly or even quadratically. Once a model is embedded in high-frequency workloads such as smart coding tools, office assistants, and customer-service bots, the cost of a single request directly determines whether the business can scale.
At the same time, real business requirements have shifted from “being able to read text” to “being able to see images.” Tasks like design-mockup reproduction, report reconciliation, slide layout, and video-content understanding require the model to understand information visually.
Sparsification, long context, and native multimodality have therefore become the three main technical routes that balance capability, efficiency, and cost. The model analyzed here makes explicit choices in all three directions, and is the first version in its series to natively support multimodality.
Its launch cadence is itself instructive. On August 20, the model was first released anonymously and for free on an aggregator platform, where developers—unaware of its identity—freely switched between it and several mainstream models to compare them; on the evening of August 26, the vendor officially revealed its identity and open-sourced it globally. This “anonymous gray release first, formal launch second” approach carries threefold value:
- Real-workload validation: During the anonymous phase, the model topped the aggregator platform’s weekly call-volume ranking and set a single-platform call-volume record on a smart coding tool, which processed tens of trillions of tokens within six days. Massive, diverse, and unpredictable traffic is the best test set for a model’s stability and long-tail capability.
- Neutral reputation: Callers chose the model without knowing its origin, so the choice itself was a vote on effectiveness and cost—more persuasive than an official benchmark.
- Engineering stress test: During the gray release, all request traffic was carried by domestically produced accelerators, showing that the model had already been validated in large-scale production for hardware–software co-optimization and heterogeneous compute adaptation. This matters especially for enterprises that want to reduce compute supply-chain risk and pursue self-controlled deployment.
2. Architecture: Sparse, Long Context, and Native Multimodal
2.1 Sparse MoE: Balancing Capability and Cost with “Large Total, Small Active”
The model uses a Mixture-of-Experts (MoE) architecture with about 320 billion total parameters, but at each layer each token is routed to only a few experts. Each inference actually involves about 18 billion parameters in computation—roughly 5.6% of the total.
The rationale can be broken into three layers:
- Decoupling knowledge capacity from compute. Vast knowledge is stored across a large number of experts, and inference activates only the parts most relevant to the current input. Knowledge capacity therefore grows with total parameters, while per-inference compute grows only with active parameters.
- Stronger capability at the same activation scale. Compared with a dense model of similar active-parameter size, a sparse model can hold several to dozens of times more parameters at nearly the same compute cost, a clear advantage on knowledge-intensive tasks.
- Substantially lower inference compute. Ignoring the attention portion, per-inference compute is roughly proportional to the active-parameter count—about 2 × 18B = 36B floating-point operations. That is roughly on the order of a single forward pass of a ~3B-parameter dense model. In other words, it pairs “the knowledge base of a large model” with “the compute bill of a small model.”
There is also an easily overlooked engineering point: memory footprint is determined by total parameters, while compute is determined by active parameters. Deployment requires fitting the weights of all 320 billion parameters into memory (or accommodating them via other means)—a completely different order-of-magnitude problem from deploying an 18-billion-parameter dense model (see Section 4.1).
Another notable design choice is depth. The model reduces the previous text-only generation in the same series from 92 layers to 45 layers—exactly half. Fewer layers shorten the critical path of the forward pass and reduce inter-layer communication and launch overhead; the width carried per layer increases accordingly, concentrating more parameters in experts and feed-forward modules. This “fewer layers, wider layers” design directs the parameter budget toward expert capacity rather than stacked depth, a common architectural tradeoff for sparse models.
2.2 Hybrid Sparse and Linear Attention: Breaking the Quadratic Bottleneck of Long Context
The compute and memory footprint of standard self-attention grow quadratically with sequence length, becoming nearly unmanageable when context reaches the million-token scale. The two mainstream responses in the industry are:
- Sparse attention: Attention computes relationships only among a subset of positions. Common approaches include sliding windows (attend only to nearby tokens), block sparsity (aggregate and compute by blocks), and selective attention (routing decides which historical blocks to attend to). Its advantage is that it preserves direct retrieval over arbitrary positions; its drawback is that the relationship between key patterns and sequence length depends on the implementation.
- Linear attention: Using kernel approximations or recurrent state compression, it reduces complexity to a linear relationship with sequence length. At decoding time it only needs to maintain a fixed-size state, similar to a recurrent neural network, which makes it naturally suited to very long sequences—but it is relatively weaker at precisely recalling distant tokens.
This model mixes the two mechanisms: linear-complexity mechanisms handle most of the sequence’s compute, while a small amount of attention compute handles global retrieval. This “linear foundation, sparse completion” combination makes million-token context computationally feasible. For developers, two points deserve attention:
- The longer the context, the lower the unit cost. The cost advantage of linear mechanisms amplifies as length grows, whereas the cost of standard attention rises sharply.
- Long context does not mean perfect utilization. Hybrid attention typically performs well on long-context retrieval tests, but in engineering practice you should still follow the principle of “relevance first, shorter over longer” when organizing context, rather than indiscriminately stuffing an entire document into the window.
2.3 Native Multimodal: Unifying Representations from the Pretraining Stage
“Native multimodal” is fundamentally different from “later-fusion multimodal”:
| Dimension | Later-fusion multimodal | Native multimodal |
|---|---|---|
| Visual integration | Separate vision encoder + projection layer feeding into the text model | Text, images, etc. unified as tokens in the same sequence, jointly modeled from pretraining |
| Training data | Mostly text, with visual data largely from post-training alignment | Large-scale mixed multimodal corpus, jointly trained |
| Cross-modal reasoning | Limited by alignment quality, weaker deep reasoning | Visual information shares parameters with language capability, shorter reasoning path |
| Engineering maintenance | Vision and language components maintained separately | A single model handles everything |
According to public information, the model’s pretraining corpus was a mixed multimodal dataset of about 30 trillion tokens. The value is that visual understanding is no longer a “plug-in” bolted onto a text model, but a foundational capability that grows together with language capability—laying the groundwork for closed-loop tasks such as “generate, look, and revise” simultaneously.
On the input side, a native multimodal model typically accepts text, images, video, and files (such as PDFs and spreadsheets), while output remains primarily text, later rendered or assembled into final artifacts at the application layer. When integrating, it is advisable to:
- For long documents and spreadsheet-like files, complete parsing and structuring at the application layer first, feeding the model clean, compact text;
- For images and video, reasonably control resolution, sampled frame count, and segmentation granularity, balancing information completeness against the number of visual tokens.
3. Capabilities: When Vision Enters Coding and Office Workflows
3.1 Visual Coding: Letting the Model “See” Its Own Output
Traditional code-generation feedback loops have only two channels: compilation/run results and unit tests, both of which are nearly “blind” to layout, styling, and visual-effect problems. For example, whether a button is occluded, whether contrast meets standards, or whether text overflows its container—all require a human to inspect with their eyes. Visual Coding brings this “looking” step into the feedback loop, covering three typical scenarios:
- Frontend page development: After generating page code, the model feeds the rendered result back in as an image, checks spacing, alignment, color scheme, and responsive layout against the design mockup, and revises the code accordingly.
- Game feature building: The model runs the game and captures key frames, spotting misplaced colliders, animation anomalies, and logic bugs.
- 3D scene construction: The model renders scene screenshots from multiple viewpoints, checking object placement, proportions, lighting, and consistency across viewpoints to avoid problems like “looks fine from one angle but clips through geometry from another.”
Notably, the team designed a dedicated data-synthesis pipeline for visual coding. Its core idea is to programmatically construct “code–render result–defect annotation” triplets: automatically generating code with specific defects along with the corresponding rendered images, so the model learns to recognize these defects. Defect types can cover misalignment, overflow, color conflicts, missing states, and more—effectively writing a set of automatically executable “visual test cases” for the model, training its self-visual-judgment and test-time-improvement abilities. This approach is equally useful as a reference for training “generate–verify” models in other domains.
In practice, it is advisable to enforce a render–inspect–revise loop on the model’s output rather than expecting it to “get it right in one shot.” A simplified version of the flow looks like this:
# Visual coding self-check loop (illustrative pseudocode)CHECKLIST = ["Layout alignment", "Text overflow and truncation", "Color contrast","Interaction state completeness", "Responsive adaptation"]def visual_coding_loop(spec_text, design_image=None, max_rounds=3):messages = [{"role": "user", "content": [{"type": "text", "text": f"Requirements:\n{spec_text}"},# Provide the design mockup when available, as a reference for visual comparison*([{"type": "image", "data": design_image}] if design_image else []),]}]for round_idx in range(max_rounds):# 1) Generate code: the application layer writes files, installs dependencies,# builds, and starts a preview servercode = chat(messages, temperature=0.2, max_output_tokens=8192)# 2) Render multi-state screenshots: desktop/mobile viewports,# default/hover/error statesshots = render_and_capture(code, viewports=["desktop", "mobile"])# 3) Visual self-check: send the rendered results back to the model# along with the checklistverdict = chat(messages + [{"role": "assistant", "content": code},{"role": "user", "content": [{"type": "text", "text": f"Check the following rendered results against the checklist item by item, and list only issues that genuinely exist:\n{CHECKLIST}"},*shots,]},])if verdict.get("issues"):# 4) Iterate revisions using the issue listmessages += [{"role": "assistant", "content": code},{"role": "user", "content": verdict["issues"]}]continuereturn code # Passed checks; return the final output# Reached the limit without passing: return the last version# and flag items for manual reviewreturn code
Three implementation details are worth noting:
- An explicit checklist: Writing specific questions like “is it aligned?” and “does it overflow?” into the prompt yields more actionable feedback than vaguely asking “are there any problems?”
- Multi-viewpoint, multi-state rendering: A single screenshot easily misses issues; multi-viewport, multi-state screenshots expose responsive and interaction-state defects.
- A maximum iteration count: This prevents the model from getting stuck in an endless revise loop on visual judgment.
3.2 Office and Professional Scenarios: From “Can Generate” to “Has Taste”
For tasks involving presentations, PDFs, word-processing documents, and spreadsheets, the value of visual capability is equally prominent. The model can first generate an outline and content, then produce a first-draft file; the application layer renders the document into page images, and the model can then “review” the layout and give concrete aesthetic and compliance judgments:
- Whether the heading hierarchy is clear and the emphasis is prominent;
- Whether the image-to-text ratio is balanced, whether whitespace is reasonable, and whether content is crowded or overflows;
- Whether fonts, colors, and alignment are consistent within the same document;
- Whether chart annotations are complete and tables are easy to read.
This capability moves document generation from “can produce” to “deliverable output,” which is exactly what finance research, consulting reports, and teaching materials value most.
4. Inference Serving and Deployment Practices
4.1 Resource Estimation: Memory Follows Total Parameters, Compute Follows Active Parameters
The most common mistake in planning sparse-model deployment is estimating memory needs from the active-parameter count. A rough estimation method follows (weights only; actual deployment must also reserve memory for KV/state caches, activations, and framework overhead):
| Weight precision | Bytes per parameter | ~320B-param weight footprint | Use case |
|---|---|---|---|
| FP16/BF16 | 2 | ~640 GB | Quality first; needs multi-GPU tensor parallelism |
| 8-bit quantization | 1 | ~320 GB | Balance between quality and cost |
| 4-bit quantization | 0.5 | ~160 GB | Cost-sensitive, resource-constrained scenarios |
Typically, an 8-GPU node with 80GB-class accelerators provides 640 GB of memory in total, so in theory a single 8-GPU server can hold the FP16 weights. But you must leave room for KV/state caches, framework overhead, and the communication buffers of expert parallelism, leaving little headroom on a fully loaded single machine. More common in production:
- Deploy across two or more 8-GPU servers with tensor parallelism and expert parallelism;
- Run load tests based on concurrency and context length to determine the number of replicas.
If a team lacks a large-scale inference cluster, it can also use the model-inference services offered by cloud platforms. Such services generally provide standard interfaces, elastic scaling, and per-token billing, and are well suited to rapid validation and small-to-medium-scale workloads.
4.2 Serving Optimization: Unlocking the “Large Total, Small Active” Performance Advantage
The characteristics of the sparse-MoE and hybrid-attention architecture mean that a general-purpose inference framework’s default configuration may not deliver the model’s full performance. Several directions deserve attention:
- Kernels and memory management tuned for hybrid attention: Traditional KV Cache is organized by “layers × heads × head dimension,” but in a hybrid architecture the cache shape saved by different layers is not the same. The inference framework must handle them differently to maximize memory utilization and, where possible, reuse the cache of shared prefixes.
- Batching and scheduling: Continuous batching significantly improves throughput, but visual-token counts vary widely across multimodal requests. When long and short requests are queued together, short requests can be “held back” by long ones, causing jitter in time-to-first-token. In engineering, it helps to schedule in pools by input length and set up priority queues for interactive requests.
- Quantization and parallelism strategies: MoE weights can be quantized, but the impact of different precisions on expert routing and output quality must be measured on business datasets; the combination of tensor parallelism and expert parallelism determines cross-GPU communication overhead and should be tuned against node interconnect bandwidth.
- Fast/slow model collaboration: Use this kind of “large total, small active” model as the main “fast model,” paired with a stronger but more expensive “slow model” in a routing layer. Simple, latency-sensitive requests are handled directly by the fast model, while complex tasks are escalated to the slow model or trigger multi-turn reasoning—guaranteeing quality while keeping overall cost under control.
4.3 Platform Integration: Standard Interfaces, Content Safety, and Cost Observability
When integrating such a model on a cloud platform, focus on these engineering points:
- A unified API gateway: Consolidate multiple models and versions behind one interface, and improve availability with routing, fallback, and retry policies. For example, when the primary model is rate-limited or times out, traffic can automatically switch to a backup model.
- Input-side content safety and file parsing: Multimodal input includes images and user-uploaded files, so content moderation and security checks should be completed at the gateway layer; document-parsing capabilities should also convert files like PDFs and spreadsheets into structured text.
- Storage support for long context and caching: Conversation histories with very long contexts, rendered artifacts, and multi-turn intermediate results are well suited to object storage; store session metadata and retrieval indexes in a database, and set sensible lifecycle policies.
- Observability and cost attribution: Build monitoring and alerts for time-to-first-token, output rate, timeout rate, and per-business-line token consumption, and factor visual tokens and tool-call counts into cost accounting—avoiding the situation where an anomaly only shows up on the bill at month’s end.
5. Context Engineering: Using the Million-Token Window Correctly
Native support for a context window of about 1M tokens (1,048,576) does not mean you should stuff an entire code repository or a full document set into it. For developers, a more practical approach is a method of “context budgeting.”
Taking a 1M-token window as an example, a typical agent task can allocate its budget as follows (values are illustrative and must be adjusted for the actual task):
| Component | Recommended share | Typical content | Management strategy |
|---|---|---|---|
| System prompt & tool definitions | ~2% | Role, guidelines, available tools and call formats | Trim redundancy; load tools dynamically per task |
| Conversation history | ~15% | Prior turns, completed edits | Periodic summarization/compression; drop low-value fragments |
| Multimodal & reference material | ~30% | Design mockups, rendered screenshots, retrieved docs and code snippets | Relevance first; control resolution and count |
| Output reservation | ~5% | Code, document body, reasoning process | Chunked output; run long tasks in segments |
| Safety margin | ~48% | Handling context spikes and multi-turn accumulation | Keep elasticity; trigger threshold alerts and auto-compression |
A few principles apply in practice:
- Relevance over completeness: Prioritize injecting a small amount of high-value information directly relevant to the current task, rather than “stuffing in everything.”
- Summarize tool results: Large amounts of raw content returned by retrieval or tools should be summarized and deduplicated before entering the context.
- Load multimodal inputs on demand: Capture rendered screenshots as needed; avoid re-sending all historical screenshots every turn.
- Visualize context usage: During development and debugging, displaying the context composition quickly reveals “which step is quietly eating up the window.”
6. Selection, Evaluation, and Risk Control
6.1 Build Your Own Evaluation Set
Public leaderboards reflect a model’s overall capability but cannot directly answer “how does it perform in my business?” It is advisable to sample real business traffic to construct an evaluation set covering these categories:
- Visual understanding: design-mockup reproduction, reading values from reports, reviewing document layouts;
- Code generation and repair: fixing defects in real repositories, generating frontend pages, writing scripts;
- Long context: cross-document retrieval, locating issues in long logs, understanding large codebases;
- Tool calling and instruction following: multi-step tool-chain calls, stable output under format constraints;
- Hallucination control: the ability to recognize non-existent facts and non-existent APIs.
During evaluation, record both quality metrics (such as task pass rate and human ratings) and cost metrics (such as average tokens consumed per task and average retry count)—this is the only way to truly reflect “cost per unit of intelligence.”
6.2 Risks and Mitigations
No model is omnipotent, and adopting this kind of architecture requires identifying the following risks in advance:
- Confirmation bias in visual self-checks: The model may “see” defects that don’t exist or overlook obvious ones; keep a manual-review step for critical artifacts.
- Very long context is not complete memory: Place key information in prominent positions within the context, or inject it dynamically via retrieval augmentation.
- Routing jitter: MoE expert routing depends on the input distribution, so different domain prompts may activate different experts, causing the same model to vary widely across tasks—confirm this through per-domain evaluation.
- Supply chain and compliance: Before commercial use, verify the license terms of an open-source model; for scenarios involving sensitive data, prefer options that support on-premises or dedicated deployment.
7. Conclusion
Using the model analyzed here as an example, the new generation of large models is shifting from a “parameter race” to a “cost-per-unit-of-intelligence race.” Its technical route can be summarized in three points:
- Sparse MoE: use large total parameters to carry knowledge, and small active parameters to control compute;
- Hybrid attention: use linear-complexity mechanisms to support million-token context;
- Native multimodality with a visual self-check loop: move the model from “can generate” to “can inspect and improve.”
For developers, what ultimately determines real-world results is often not the model’s parameters themselves, but the engineering system built around the model:
- Sound resource planning and serving optimization;
- Disciplined context-budget management;
- An evaluation system close to the business;
- Comprehensive monitoring and cost control.
On this foundation, cloud services such as model inference, document parsing, object storage, and monitoring/alerting—for example those offered by Baidu AI Cloud—can help you move faster from prototype validation to large-scale launch. Only by deeply understanding the architectural tradeoffs and getting the engineering details right can you obtain stable, predictable intelligence at a reasonable cost.
