End-to-End Training of a Multimodal Image Generation Model on Domestic AI Chips: A Complete Technical Breakdown
This article offers a deep technical analysis of a newly released and open-sourced multimodal image generation model whose autoregressive foundation was trained end-to-end on domestic AI accelerator chips. It breaks down the hybrid "autoregressive backbone + diffusion decoder" architecture, the full training pipeline (from data preprocessing and large-scale pretraining to SFT and reinforcement learning) and its optimizations on domestic hardware, the approach to Chinese text rendering, support for arbitrary resolutions, and practical guidance on inference cost, memory estimation, deployment, and open-source adoption.
Introduction: What End-to-End Multimodal Training on Domestic AI Chips Means
Multimodal image generation models are being adopted quickly across creative design, e-commerce, marketing, and content platforms, but the compute barrier to training them remains high. As domestic AI accelerator chips continue to mature in both performance and software ecosystem, the industry has begun exploring how to move the full training pipeline of large models onto domestic compute.
A multimodal image generation model was recently released and open-sourced. According to its official announcement, its autoregressive foundation was trained end-to-end on domestic AI accelerator cards, from data preprocessing to large-scale pretraining, making it one of the earlier multimodal generation models to attempt full-pipeline training on domestic chips. Published comparison results show that it ranked first among open-source models in two benchmarks: complex visual text generation and long-text rendering.
From a technical standpoint, this release carries three layers of value:
- Proving feasibility: Multimodal models must simultaneously handle several compute patterns, including autoregressive, diffusion, and visual encoding. Completing the full pipeline demonstrates that the domestic software stack can now support this kind of heterogeneous, mixed workload.
- Offering a reusable engineering reference: The complete pipeline from data preprocessing to post-training is publicly available, so other teams can learn from it.
- Targeting practical scenarios: The model focuses on text rendering and non-standard resolution generation, exactly the areas where traditional image generation models are weakest.
The following sections break down its technical approach across five dimensions: architecture, training optimization, text rendering, resolution strategy, and engineering practice.
1. Architecture: How the Autoregressive Backbone and Diffusion Decoder Divide the Work
1.1 Choosing Between Two Generation Paradigms
Mainstream image generation models broadly follow one of two routes:
- Autoregressive route: Borrowing the “next-token prediction” paradigm from large language models, it discretizes an image into a token sequence and generates it token by token.
- Advantages: naturally compatible with text conditioning, able to leverage a language model’s knowledge and reasoning, with an interpretable and controllable generation process.
- Cost: every step passes through the entire backbone, so generation speed is strongly tied to sequence length.
- Diffusion route: Repeatedly denoises in a continuous latent space, updating the whole image at each step.
- Advantages: fast, detail-rich, and the most mature route in terms of community ecosystem.
- Weakness: weaker handling of global semantics and long-range layout; controllability depends on additional conditioning mechanisms.
The pure autoregressive route faces two more problems. First, images have a large number of tokens and long sequences, so the compute and memory cost of self-attention grows quickly with length. Second, details tend to become blurry, because discretization itself loses information.
The pure diffusion route, meanwhile, lacks an explicit “thinking” process. Faced with instructions involving counting and spatial relationships, such as “a three-tier cake with five candles on the top tier,” the model often loses track of one constraint while satisfying another.
1.2 Module Breakdown: The Backbone “Understands,” the Decoder “Renders”
The model adopts a hybrid architecture of “autoregressive backbone + diffusion decoder,” with the two modules each doing their own job:
- Autoregressive backbone (~9B): Handles semantic understanding and global planning.
- Input: text prompts, reference images, layout, and other conditions.
- Output: high-level semantic representations, including subject, style, and composition, as well as text content, position, and hierarchy.
- Multimodal semantic understanding: coreference and spatial relationships common in complex instructions, such as “the title is in the top-left and the product is in the bottom-right,” require the model to infer using both linguistic common sense and visual priors.
- Diffusion decoder (~7B, based on a DiT structure): Denoises in parallel in latent space, reconstructing high-frequency details such as texture, lighting, and text strokes.
- Glyph encoding module: Injected into the decoder as a conditioning path, providing stroke-level structural constraints for text rendering.
In effect, this writes the “figure it out first, then draw it” creative workflow into the model’s structure. The autoregressive backbone can draw on a language model’s reasoning ability and naturally fits reasoning-enhanced paradigms such as chain-of-thought and self-correction.
According to the official statement, the autoregressive backbone portion was trained end-to-end on domestic chips; the original text does not spell out the specific training environment for the decoder and text-rendering portions, one by one.
1.3 Breaking Down the Training Stages
Going from “data to model” typically involves four stages:
- Data preprocessing: Deduplicate, clean, and quality-score image-text pairs; encode images with a vision encoder and normalize text; separately annotate and validate text regions in scenarios such as posters and UIs.
- Large-scale pretraining: Align the backbone and decoder on massive image-text data. The sequence-packing strategy for visual and text tokens, and the ratio between the autoregressive and diffusion losses, directly affect convergence.
- Supervised fine-tuning (SFT): Improve instruction-following and layout control using high-quality, instruction-based samples (e.g., “generate a poster in a certain style”).
- Reinforcement learning (RL): Further optimize using automated reward signals. For example, an aesthetic scoring model constrains visual quality, an OCR model checks whether the rendering is accurate, and an image-text matching model verifies semantic consistency.
It is worth emphasizing that the original text specifically notes that the “autoregressive foundation” was completed end-to-end on domestic chips, a precise phrasing. Many so-called “domestic” solutions today only move inference or partial fine-tuning onto domestic cards while the backbone training still depends on the existing software and hardware stack. The former is merely “migrating inference”; the latter is truly “full-pipeline training.”
| Dimension | Pure autoregressive | Pure diffusion | Autoregressive backbone + diffusion decoder |
|---|---|---|---|
| Global semantics and long-range layout | Strong | Weaker | Strong (handled by the backbone) |
| Detail and texture | Weaker | Strong | Strong (handled by the decoder) |
| Generation speed | Slow, strongly tied to sequence length | Fast | One backbone forward pass + parallel decoder denoising |
| Reasoning and controllability | Strong | Depends on additional conditioning | Strong |
| Chinese text rendering | Depends on glyph modeling | Depends on data scale, prone to errors | Enhanced via a dedicated glyph encoding module |
| Main engineering cost | Long-sequence memory and latency | Global consistency | Two architectures; more complex training and inference pipelines |
2. Training on Domestic Compute: Full-Pipeline Optimization from Data Preprocessing to RL
Training a multimodal model on domestic chips is difficult not just because of the chip itself, but because of the entire toolchain from framework to operators to parallelism strategy. The official solution mainly includes the following layers.
2.1 Framework and Training Suite
Training relies on a domestic deep learning framework (MindSpore). The team built an in-house end-to-end training suite covering data preprocessing, pretraining, SFT, and RL, consolidating key steps that were previously scattered into a unified toolchain. This offers three benefits:
- Consistent interfaces and precision strategies: Different stages can produce numerical discrepancies due to framework versions or operator implementations; unifying them makes problems easier to locate and reproduce.
- Unified data formats: Samples produced by preprocessing can feed directly into each stage, avoiding repeated conversions.
- A same-stack post-training pipeline: The RL stage requires both forward and backward passes plus frequent weight synchronization, which is easier to optimize within a single framework.
2.2 Multi-Level Pipeline Dispatch for Dynamic Graphs
Root of the problem: Dynamic graphs are flexible to develop and easy to debug, but at execution time operators are dispatched to the device one by one. If the host cannot dispatch fast enough to keep up with device execution, the device “idles.” Autoregressive training traverses the whole model at every step, and diffusion training stacks multiple denoising steps on top, so the dispatch overhead is amplified several times over.
Optimization approach: The official solution uses multi-level pipeline dispatch for dynamic graphs, splitting the compute graph into multiple pipeline levels so that dispatch and execution are decoupled and dispatch happens ahead of time. To use an analogy, it upgrades a factory from “deliver one process step, perform one process step” to “the upstream process feeds material while the downstream process works simultaneously,” filling the device’s idle gaps. This mode is typically combined with graph-mode compilation.
2.3 Multi-Stream Parallelism and Computation-Communication Overlap
The communication overhead in large-model training comes mainly from three sources:
- Data parallelism: gradient synchronization via AllReduce;
- Tensor parallelism: AllReduce or AllGather in the forward and backward passes;
- Pipeline parallelism: point-to-point transfer between adjacent pipeline stages.
Conventional approach: communication and computation run serially, with communication time added directly to the per-step time.
Multi-stream parallelism: communication operations are placed on separate compute streams and overlap with the compute stream. Combined with fine-grained compute-graph splitting, a single operator’s computation can also be broken into multiple segments, alternating between compute and communication in nested fashion. Two engineering considerations apply:
- Synchronization: data dependencies exist between streams, so events are needed to guarantee ordering;
- Memory: simultaneously resident buffers increase memory pressure, so buffer sizes must be balanced.
On domestic compute, multi-stream parallelism works together with high-speed interconnects. In multi-machine, multi-card scenarios, pipeline parallelism is more sensitive to bandwidth, and the gains from overlapping communication and computation are usually more pronounced.
# Pseudocode: overlapping communication and computation under multi-stream parallelism (illustrative)import torchdef forward_with_overlap(model, x, comm_stream):"""Distribute per-layer communication onto the communication stream so itoverlaps with the main compute stream."""# Make the communication stream wait for the main stream to finish preparing inputcomm_stream.wait_stream(torch.cuda.current_stream())h = xwith torch.cuda.stream(comm_stream):# Independently compute each layer's communication volume and dispatchshapes, dtypes = model.prepare_comm(h)handles = model.comm_stream.launch_all_gather(shapes, dtypes)# The main compute stream continues with later operators without waiting for communicationfor layer in model.layers:h = layer.forward_without_allgather(h)# Synchronize only right before the communication result is actually neededwith torch.cuda.stream(comm_stream):h = torch.cat(handles.wait(), dim=model.tensor_parallel_dim)torch.cuda.current_stream().wait_stream(comm_stream)return h
2.4 Fused Operators Tailored to Domestic Chips
Beyond scheduling, the official work also implements several types of fused operators for Ascend hardware characteristics, typically in three categories:
- AdamW EMA fusion: The optimizer update and exponential moving average are two independent element-wise operations that each read and write the parameters in full. Fusing them into a single read-write pass saves memory bandwidth, which is substantial for large-model training. EMA weights are usually smoother than the raw weights, so using them for evaluation and release yields better and more stable results.
- Normalization operator fusion: Normalization operations such as RMS Norm are merged with adjacent element-wise operators before and after them. Tiling is also applied to match the memory hierarchy of the domestic chip, keeping intermediate results in on-chip high-speed cache as much as possible.
- Other high-performance operators: Core operators such as matrix multiplication and attention are implemented against the Ascend architecture, reducing memory access and improving overall compute efficiency.
import torchfrom torch.optim import AdamWclass FusedAdamWEMA(AdamW):"""Optimizer fusing the AdamW update with EMA: parameters, gradients, andEMA state are read and written in a single pass."""def __init__(self, params, *, ema_decay=0.9999, **kwargs):super().__init__(params, **kwargs)self.ema_decay = ema_decay# Maintain an EMA shadow weight for each parameterself.ema_params = [torch.zeros_like(p, memory_format=torch.preserve_format)for group in self.param_groups for p in group['params']]@torch.no_grad()def step(self, closure=None):super().step(closure)d = self.ema_decayfor p, ema in zip((p for g in self.param_groups for p in g['params']), self.ema_params):# Intended to complete within a single fused operator: one memory access# for both the EMA and the parameter updateema.mul_(d).add_(p.detach(), alpha=1 - d)
2.5 Engineering Takeaways
For teams preparing to migrate training to domestic compute, the following suggestions apply:
- Find the bottleneck first: Use profiling tools to distinguish “compute-bound” from “dispatch/communication-bound,” rather than switching parallelism strategies up front.
- Prioritize fusing high-frequency stages: The optimizer, normalization, and attention are hot spots executed every step.
- Guarantee numerical consistency: Use the same precision strategy for training and evaluation, avoiding one set of weights for training and another for production.
- Plan fault tolerance: The larger the training scale, the more unavoidable failures become. Periodically saving checkpoints and supporting resumable training are basic requirements.
3. Chinese Text Rendering: The Longstanding Hard Problem in Image Generation
3.1 Why Models Keep “Forgetting How to Write”
Many teams have had this experience: ask an image generation model for a poster, and the visuals, color scheme, and layout look fine, but the title text is either missing strokes or turned into garbled characters. Chinese rendering is especially difficult, mainly for three reasons:
- Complex glyph structure: Chinese characters have many strokes and intricate rules for combining components; missing one dot or one horizontal stroke turns it into a different character, leaving almost no margin for error.
- Limited token space: In image-text token-based models, the text encoder outputs a limited number of tokens, making it hard to fully preserve character-level information for long text; long titles and multi-line subtitles are especially error-prone.
- Sparse training distribution: In web images, “text on image” accounts for a small proportion, and clean, readable, high-quality rendering samples are even scarcer, so the model has weak priors for text.
3.2 Technical Approach
Based on public information, the model treats text rendering as a dedicated technical pipeline rather than hoping the model will “learn it naturally”:
- Dedicated glyph encoding module: Provides stroke-level structural information to the decoder, supplying the structural prior needed to “write characters correctly.”
- Data and training strategy: Beyond massive image-text data, it strengthens training on poster and layout samples. Text regions are handled separately during data preprocessing, and OCR and other automated methods are used afterward to clean and filter for samples with reliable text quality.
- Post-training reinforcement: Text-accuracy-related reward signals are introduced during SFT and RL, encouraging the model to “write correctly” during generation.
3.3 Evaluation Performance
Benchmarks fall into roughly two categories:
- Complex visual text generation benchmark: assesses consistency between on-image text and instructions, and whether layout and style follow requirements.
- Long-text rendering benchmark: specifically evaluates rendering of long titles and multi-line text.
According to the official comparison results, the model ranked first among open-source models on both categories.
3.4 Prompting Practice
Model capability is only the foundation; how you write the prompt matters just as much:
- List text content explicitly: Use quotation marks to mark the title, subtitle, and button copy, preventing the model from improvising.
- Specify the hierarchy: State which line is largest, which comes next, and where each is positioned, e.g., “title centered at the top, subtitle directly below.”
- Control the total amount of text: The more text there is, the higher the chance of rendering errors. Split long copy into a main title and secondary information.
- Validate automatically after generation: Connect OCR-based validation to the output; retry or involve human review if it fails.
4. Non-Standard Resolution: From “Preset Ratios” to “Any Size”
4.1 Limitations of Traditional Approaches
Traditional image generation models usually restrict output resolution to a few preset ratios, such as 1:1, 4:3, and 16:9. During training and inference, the system “nearest-matches” the user’s request to these ratios, mainly via three strategies:
- Stretching: Direct scaling, which distorts the image;
- Cropping: Trimming the edges, which may lose the subject;
- Padding: Filling with blank space, which introduces invalid regions.
Scenarios such as posters, social media covers, and e-commerce main images each have different size requirements, and all of the above approaches harm the final quality.
4.2 An Improved Tokenizer Strategy
The model’s improvement is to “natively support any aspect ratio”:
- Training stage: A more flexible tokenization strategy is introduced in the image encoding stage, letting the model learn on samples with a wide range of aspect ratios; at inference time, you simply specify the width and height directly, with no need to match a preset ratio.
- Resolution range: According to the official statement, the model supports arbitrary aspect ratios between 1024×1024 and 2048×2048, such as non-standard sizes like 1240×1600.
- Implementation cost: A key precondition is that the native resolution is relatively high. Models with lower resolution have comparatively limited room to scale.
For developers, this means assets can be generated “on demand” at the required size, eliminating the extra cropping or outpainting step and connecting directly to business systems’ asset specifications.
4.3 Usage Recommendations
While non-standard resolution is useful, there are a few caveats:
- Respect the documented range: Both width and height should fall within the officially supported interval; refer to the technical report and documentation for the exact limits.
- Watch memory and latency: The number of image tokens grows with resolution, and so does attention cost. Generation around 1024×1024 needs noticeably less memory than at 2048×2048, so batch tasks should be planned accordingly.
- A training-side tip: Letting area, rather than aspect ratio, determine the token count (keeping the token budget roughly constant per image sample) can balance diversity and efficiency during training.
5. Engineering Practice: Cost, Deployment, and Evaluation
5.1 Inference Cost
The officially announced API price is roughly 0.1 yuan (RMB) per image. This figure is worth attention because it approaches the marginal-cost threshold for many content platforms, turning “batch image generation” from a demo into a business that can be run on the numbers.
From the inference side, cost is mainly made up of three parts:
- GPU occupancy: memory capacity determines how many concurrent requests a single card can serve;
- Generation time: the faster the output, the higher the throughput per unit time;
- Calling strategy: whether to batch and whether to cache.
If you self-host an inference service, you can roughly estimate concurrency capacity with these steps:
- Measure per-image latency T: including text encoding, the autoregressive backbone forward pass, diffusion decoder denoising, and image decoding, using the P95 of your actual business prompts.
- Measure per-card concurrency N: constrained jointly by the memory ceiling and batch efficiency.
- Estimate daily capacity: roughly N × 86400 / T images. When the P95 latency is high, leave some headroom.
In addition, caching results for high-frequency “same prompt + same size” requests can significantly reduce average cost.
5.2 Memory Estimation
Memory requirements can be broken into four parts:
- Model weights: at BF16 precision, roughly parameters × 2 bytes. With a ~9B backbone plus a ~7B decoder, totaling 16B parameters, the weights alone take about 32GB; adding peripheral modules such as text encoding, vision encoding, and glyph encoding pushes actual usage higher.
- Inference activations and cache: attention cache and intermediate latent tensors grow with resolution.
- Inference framework overhead: communication buffers, temporary workspaces, and the like.
- Additional training-side overhead: optimizer states (AdamW needs about two FP32 states per parameter, i.e., 8 bytes), gradients, and activation checkpoints.
def estimate_memory_gb(params_b: float, bytes_per_param: int = 2,activation_gb: float = 0.0, overhead_ratio: float = 0.1) -> float:"""Roughly estimate memory usage (GB). For capacity planning only, not areplacement for measurement."""weights = params_b * 1e9 * bytes_per_param / 1e9 # weights: params × bytes per parambase = weights + activation_gbreturn base * (1 + overhead_ratio)
Deployment trade-offs:
- When a single card can hold both backbone and decoder weights, the two modules can be loaded serially and share memory in time slices;
- When it cannot, cross-card partitioning is required, raising communication overhead;
- On the training side, memory can be reduced with techniques such as ZeRO-style optimizer-state partitioning, mixed precision, and activation recomputation;
- For latency-sensitive workloads, optimizations such as fixing the random seed, reducing sampling steps, and graph-mode compilation help, but require a trade-off between quality and performance.
5.3 Open-Source Ecosystem
Weights and code have been released simultaneously on mainstream code-hosting platforms and model communities. Online access is available on the model vendor’s open platform, with plans to integrate it into the vendor’s own AI-native application products. The value of open-sourcing is that:
- Enterprises can deploy privately, avoiding data-egress and compliance risks;
- Developers can do domain fine-tuning on top of the weights, such as for a brand’s visual identity style or images of specific product categories.
It is worth noting that open source does not mean zero cost. Enterprises still need to assess inference resources, operations capability, and the boundaries of technical support when deploying.
5.4 Comparison Before and After Open-Sourcing
| Aspect | Closed-source API phase | Open-source release phase |
|---|---|---|
| Access method | Call the vendor’s open-platform API | Download weights and code from a hosting platform, or keep calling the API |
| Data compliance | Data must leave the enterprise boundary | Private deployment; data stays on the internal network |
| Customization | Limited to platform parameters | Fine-tuning, quantization, and secondary development |
| Operations cost | Low | Requires in-house compute and an operations team |
| Suitable scenarios | Rapid validation and elastic, spiky workloads | Data-sensitive, long-term stable, high-volume workloads |
6. Conclusion: Multimodal Generation on Domestic Compute Is Just Beginning
The significance of this model goes beyond “you can train a multimodal model on domestic chips”; it lies in providing a complete, reusable path:
- Architecture: an autoregressive backbone handles understanding and planning, while a diffusion decoder handles generation and detail, balancing controllability with high quality.
- Training: built on a domestic deep learning framework, with an in-house training suite covering the full pipeline of data preprocessing, pretraining, SFT, and RL, combined with multi-level pipeline dispatch for dynamic graphs, multi-stream parallelism, and fused operators to complete large-scale training on domestic compute.
- Capability: focused on Chinese text rendering and arbitrary-ratio generation, directly addressing the real pain points of scenarios such as posters and covers.
- Engineering: an inference cost of roughly 0.1 yuan (RMB) per image, together with an open-source form that supports private deployment, provides a practical basis for enterprises to scale up.
At the same time, this route is still in its early stages: generation quality, speed, and controllability all have room for improvement, and the ecosystem maturity and operator coverage of domestic chips are evolving quickly. For developers and enterprises, what deserves attention is not any single release itself, but the complete closed loop of “domestic compute + open-source models + scenario-specific fine-tuning” that it forms. Whoever runs this loop through first will gain an early advantage in the era of multimodal applications.
