Why End-to-End Multimodal Image Generation Training on Domestic AI Accelerators Matters
This article examines the architecture and engineering practices behind training multimodal image generation models entirely on domestic AI accelerators. It explains the division of labor between autoregressive models and diffusion decoders, techniques for improving Chinese text rendering and supporting non-standard resolutions, and the system-level optimizations required for stable, large-scale training. It also summarizes practical recommendations for building a reproducible training pipeline across pretraining, supervised fine-tuning, and reinforcement learning.
1. Background: Why End-to-End Training on Domestic AI Accelerators Matters
Over the past few years, large-scale multimodal model training has depended heavily on a small number of mainstream accelerators and their surrounding software ecosystems. When the entire pipeline—from data cleaning and tokenization to mixed-precision training and distributed communication—is tightly coupled to a particular hardware stack, supply-chain or policy changes can directly affect the pace of model iteration.
For this reason, completing the full workflow on domestic AI accelerators is about more than simply making a model run. It demonstrates three important capabilities:
- Ecosystem portability: After proper adaptation, mainstream deep learning frameworks can support multimodal training with tens of billions of parameters on domestic computing infrastructure.
- Usable performance: With system-level compiler optimization, operator fusion, and communication optimization, training throughput can reach an engineering-usable level instead of merely running too slowly to be practical.
- An end-to-end training loop: Pretraining, supervised fine-tuning (SFT), and reinforcement learning (RL) can be connected on the same computing platform, avoiding repeated migration between different hardware environments.
For developers and enterprises, this indicates that domestic AI accelerators can support demanding training workloads rather than being limited to inference or lightweight fine-tuning. The following sections examine the topic from four perspectives: architecture, capabilities, training, and optimization.
2. Hybrid Architecture: Why “Autoregressive Model + Diffusion Decoder” Is Becoming Common
Early image generation systems were dominated by pure diffusion models. More recently, the industry has increasingly combined two approaches: an autoregressive model for understanding and a diffusion model for generation. This hybrid architecture is becoming a common design for high-resolution, semantically rich, and controllable text-to-image generation.
2.1 Two Modules with Distinct Responsibilities
A typical hybrid architecture contains two major components:
- Autoregressive backbone for semantics and composition: This component uses autoregressive Transformer modeling to convert an image into a sequence of discrete tokens and predict them one by one. It is well suited to understanding complex semantics, controlling global composition, and using language-model-style reasoning to determine what should appear in an image and how its elements should be arranged.
- Diffusion decoder for rendering and detail: This component often uses a DiT, or Diffusion Transformer, architecture. It converts the semantic conditions produced upstream into high-quality pixels and is particularly effective at rendering fine details such as text, edges, and textures.
A useful analogy is that the autoregressive backbone acts as the director, deciding the subject, structure, and semantics of the image. The diffusion decoder acts as the art department, rendering the rough plan into a clear and realistic final result. The two modules are connected through condition injection, such as cross-attention.
2.2 Why Not Let One Module Handle Everything?
If a pure autoregressive model generates pixels directly, the cost of modeling long sequences becomes high and local details may appear blurry. If a pure diffusion model handles the entire process, its understanding of global semantics—such as what a particular image is intended to communicate—may be weaker, which can lead to disorganized compositions.
The value of a hybrid architecture lies in several areas:
- Better semantic control: The autoregressive backbone naturally inherits world knowledge and reasoning capabilities from language-model-style training. It can interpret complex instructions, such as generating a poster containing specific information.
- Higher-quality details: Diffusion decoders remain among the most mature approaches for recovering high-resolution visual details.
- Lower pressure on each module: Each component focuses on the subtask it handles best, which can make training more stable.
This design is sometimes described as cognitive generation: the model is expected not only to make an image look realistic, but also to establish a clear semantic plan before rendering it. The approach aligns with the broader industry trend of combining language-model reasoning with image generation.
3. Text Rendering: The Hardest Challenge in Multimodal Generation
For posters, covers, infographics, and similar applications, the text inside an image can be more important than the image itself. Text rendering is also one of the areas where generative models fail most often. Typical problems include distorted glyphs, missing strokes, incorrect or omitted characters, and broken layouts for long text. In practical terms, the model may appear to “forget how to write.”
3.1 Why Chinese Text Is More Difficult
Compared with English, Chinese text rendering presents several additional challenges:
- A large character set: Thousands of commonly used Chinese characters have complex structures and dense stroke patterns.
- A strong relationship between semantics and glyph shape: A small change to a glyph can turn it into another character or unreadable text, leaving little room for error.
- Long-text layout challenges: The arrangement, spacing, and line breaks of a Chinese slogan must all be handled appropriately.
3.2 An Engineering Approach: Introducing a Glyph Encoder
A common solution is to provide explicit text and glyph information in the decoding path instead of asking the model to draw characters from scratch. One approach is to introduce a dedicated Glyph Encoder:
- Convert the target text into glyph features, such as stroke structure and character embeddings.
- Inject these glyph features as strong conditions during diffusion decoding so that the model is guided to reproduce the intended forms.
- Combine the conditioning with OCR-style supervision to strengthen constraints on character correctness.
In effect, the glyph encoder acts like a proofreader working alongside the art department, checking whether the rendered characters are correct. To improve the reliability of Chinese text rendering, training data generally needs to include a broad range of Chinese glyph samples, while evaluation should measure text accuracy separately.
3.3 How to Evaluate Text Rendering
Text generation is commonly assessed in two areas:
- Complex visual text generation: Evaluates whether the model can render accurate text over complex backgrounds and with diverse fonts.
- Long-text rendering: Evaluates the completeness and correctness of longer text, as well as the quality of its layout.
Evaluation should not focus only on whether the output looks visually similar to the target. It must also determine whether the characters are correct, including accuracy at the character level.
4. Native Support for Non-Standard Resolutions
Traditional image generation models often support only a small set of preset aspect ratios, such as 4:3, 16:9, or 1:1. In real applications, however, product images, social media covers, and vertical posters often require custom dimensions, including 3:4, 9:16, or less common ratios. Generating at a fixed ratio and then cropping or stretching the result can reduce image quality or damage the composition.
4.1 Tokenizer Design Is Critical
To support variable aspect ratios and resolutions natively—commonly ranging from 1024×1024 to 2048×2048—the key is to improve the visual tokenizer and positional encoding strategy:
- Variable-resolution tokenization: Allow the visual-token grid to change dynamically with the input dimensions instead of requiring a fixed square patch grid.
- Adaptable positional encoding: Use interpolatable or scalable positional encodings, such as RoPE-style approaches, so that the model can generalize reasonably to resolutions not seen during training.
- Coverage of multiple sizes during training: Include samples with different dimensions and aspect ratios in the pretraining data so that the model learns to handle non-standard inputs.
4.2 Business Value
For posters, covers, and feed assets whose dimensions are determined by the distribution channel, native support for non-standard resolutions offers several benefits:
- The target-size asset can be generated directly without secondary cropping.
- The composition is not damaged by forced stretching, and the proportions of text and visual elements remain more natural.
- Multiple versions of the same theme can be produced more efficiently for different channels.
5. End-to-End Training on Domestic AI Accelerators
Training the architecture described above from scratch on domestic chips is primarily an engineering adaptation challenge rather than an algorithmic one.
5.1 An End-to-End Training Suite
A reusable training suite should cover the following stages and connect them into a continuous workflow:
- Data preprocessing: Image cleaning, image-text alignment, visual tokenization, and resolution normalization.
- Pretraining: Learning representation and generation capabilities from large-scale image-text pairs.
- Supervised fine-tuning (SFT): Using high-quality, instruction-oriented data to align the model with human preferences and task formats.
- Reinforcement learning (RL): Applying preference optimization based on reward models to further improve image quality and text accuracy.
The critical requirement is that all stages run on the same computing platform. This reduces the repeated adaptation cost caused by moving between different hardware environments. In practice, the workflow is usually adapted through a deep learning framework that supports domestic AI accelerators.
5.2 Stability Considerations
- Numerical stability: Loss scaling and gradient clipping under mixed-precision training need to be tuned for the behavior of domestic operators.
- Checkpoint and resume support: Checkpoints must be saved and restored reliably during long-running training to handle hardware interruptions or instability.
- Observability: Training curves, device memory usage, throughput, and other metrics should be logged and monitored so that problems can be diagnosed efficiently.
6. Practical Training Performance Optimization
On the same hardware, throughput can differ by several times depending on how well the system is optimized. The following techniques are particularly important for domestic AI accelerators.
6.1 Overlapping Computation and Communication
In large-scale training, major bottlenecks often come from communication—such as gradient synchronization and tensor or pipeline parallelism—as well as host-side dispatch overhead. Common techniques include:
- Multi-stream execution: Separate computation streams from communication streams so that communication can run in the background and overlap with computation, hiding communication latency.
- Multi-level pipelined dispatch: Split a single operator into multiple subtasks and dispatch them to the device in stages to reduce host-side dispatch bottlenecks.
- Computation-communication overlap: Partition backpropagation appropriately so that communication can be triggered as early as possible.
The following pseudocode illustrates the basic idea of pipelined dispatch:
# Example: dispatch operators by pipeline stage to reduce# serialized host-side dispatch overheadfor stage in model.pipeline_stages:dispatch(stage.forward, device_stream=compute_stream)# Run communication and computation on separate streams# to maximize overlapoverlap(comm_op, stream=comm_stream, with_op=compute_stream)
6.2 Fusion Operators Optimized for Ascend Hardware
For a specific chip architecture, combining multiple small operators into larger fused operators can significantly reduce memory-access and scheduling overhead. Common approaches include rewriting and fusing the following operator categories so that they better match the hardware’s vector and matrix computing units:
- Normalization operators: For example, fuse RMS Normalization with the residual path to reduce repeated memory reads and writes.
- Optimizer operators: For example, combine weight decay, momentum updates, and exponential moving average (EMA) operations into a single AdamW step.
- Activation and attention operators: Combine multiple attention-related matrix operations to improve data reuse.
Fused operators generally need to be used together with compiler and graph optimization so that operator replacement and memory reuse can be automated and fully exploited.
6.3 Distributed Parallelism Strategies
At the scale of thousands of accelerators, a single parallelism strategy is rarely sufficient to balance memory usage and communication overhead. In practice, teams commonly combine several approaches:
- Data parallelism: Increases throughput, typically together with gradient accumulation and efficient All-Reduce operations.
- Tensor parallelism: Splits parameters within a layer to reduce the memory pressure on each device.
- Pipeline parallelism: Divides layers across different stages and uses micro-batches to improve device utilization.
When these strategies are combined, the system must be tuned repeatedly to find a balance between communication volume and memory consumption.
7. Engineering Recommendations and Best Practices
Teams planning to train or reproduce this type of model on domestic AI accelerators should consider the following recommendations:
- Make the full pipeline work before optimizing it: First ensure that the complete path—from data processing to training and evaluation—runs successfully. Then introduce fused operators, parallelism strategies, and other optimizations step by step to avoid premature optimization.
- Treat data quality as a limiting factor: Text-rendering performance depends heavily on high-quality image-text pairs and broad glyph coverage. Investment in data often produces greater returns than additional hyperparameter tuning.
- Create dedicated evaluation sets: Establish separate benchmarks for text accuracy, non-standard resolutions, and long-text rendering so that the effect of each change can be quantified.
- Use cloud infrastructure where appropriate: Large-scale training requires substantial resources. Cloud AI computing, container platforms, object storage for datasets and checkpoints, logging services, and monitoring systems can help manage the full training lifecycle and reduce operational overhead.
- Preserve reproducibility: Fix random seeds and record data versions and environment configurations so that experiments can be compared and rolled back when necessary.
8. Conclusion
Completing the full training workflow for a multimodal image generation model on domestic AI accelerators demonstrates that an engineering loop spanning architecture design, training, performance optimization, and application deployment is feasible.
The hybrid design—using an autoregressive model for understanding and composition and a diffusion decoder for rendering—addresses two of the most important challenges in current image generation applications when combined with explicit glyph conditioning and flexible tokenizer strategies: rendering Chinese text correctly and supporting custom image dimensions.
For developers, the broader lesson is methodological. The division of responsibilities in a hybrid architecture, explicit constraints for text rendering, encoding adaptations for non-standard resolutions, and system optimizations such as pipelined dispatch, multi-stream execution, and fused operators are all techniques that can be transferred across different hardware platforms and model architectures. Understanding these principles offers more lasting value than memorizing any single product or hardware model.
