Full-Stack Training of Multimodal Image Generation Models on Domestic AI Accelerators: A Practical Engineering Path

    This article presents a practical engineering path for training multimodal image generation models on domestic AI accelerators. It defines what makes a full-stack training workflow usable, describes a minimal reproducible environment, and outlines key considerations across data preprocessing, pretraining, supervised fine-tuning, and reinforcement learning alignment. It also explains a hybrid autoregressive-plus-diffusion decoder architecture, training performance optimization techniques, Chinese text rendering, arbitrary aspect ratio generation, troubleshooting guidance, and engineering recommendations for repeatable large-scale training.

    1. Conclusion First: What Makes Full-Stack Training “Usable”

    When training multimodal image generation models on domestic AI accelerators, the real challenge is not whether a single image can be generated. The harder problem is building a repeatable engineering loop that covers data preprocessing, pretraining, fine-tuning, and alignment.

    The conclusion is straightforward: if the goal is full-stack training of a multimodal image generation model on domestic AI accelerators, start with a minimal reproducible loop, then scale gradually. Data supply, communication-computation overlap, numerical stability, glyph quality, and arbitrary aspect ratio support should be treated as priority acceptance points.

    A full-stack training workflow should not be judged only by a single inference run or a single forward step. It should meet at least four acceptance criteria:

    1. Data throughput is not the bottleneck: Data preprocessing should keep pace with training computation, reducing idle time on accelerators.
    2. Training is observable: Loss, gradient norm, operator latency, communication ratio, and device utilization should be recordable and comparable.
    3. Checkpoint recovery is reliable: Large-scale training inevitably encounters restarts, scaling changes, and failover. The training framework should support stable checkpoint saving and recovery.
    4. Key capabilities are measurable: Beyond image quality, the workflow should also evaluate capabilities such as Chinese text rendering, layout consistency, and arbitrary aspect ratio output.

    Based on these criteria, a portable engineering sequence is recommended: first complete smoke training on a single node with multiple devices, then scale to multi-node and multi-device training; first ensure numerical stability, then pursue higher throughput; first decouple and validate the autoregressive base model and diffusion decoder, then jointly optimize the overall training strategy.

    2. Reproducible Anchor: Build a Minimal Training Environment

    To support engineering reproducibility, the example environment below is illustrative. Specific versions should be selected according to actual hardware and software compatibility.

    Hardware example: Ascend 910B series or equivalent domestic accelerators.

    Software stack example: CANN 8.x and MindSpore 2.x, or PyTorch 2.x with torch_npu.

    Precision strategy: Prefer bf16. If fp16 is used, configure loss scaling or an equivalent stabilization mechanism.

    Parallelism strategy: For small models or smoke validation, tensor parallelism can be used first. As scale increases, introduce pipeline parallelism and data parallelism.

    A minimal launch command can be abstracted as follows. It is only intended to express the configuration approach:

    1. python train.py \
    2. --stage pretrain \
    3. --npu 8 \
    4. --tensor-parallel 4 \
    5. --pipeline-parallel 2 \
    6. --precision bf16 \
    7. --prefetch 8 \
    8. --save-every 200

    In this minimal environment, it is recommended to first validate:

    • Whether data loading can provide a continuous supply.
    • Whether loss remains smooth under mixed precision.
    • Whether saving and restoring are consistent.
    • Whether per-step latency is stable.

    Only after these basic issues are ruled out does it make sense to scale up.

    3. Full-Stack Training Pipeline: Four Stages and Key Focus Areas on Domestic AI Accelerators

    Multimodal image generation training can usually be divided into four stages: data preprocessing, large-scale pretraining, supervised fine-tuning, and reinforcement learning alignment. Each stage has different concerns on domestic AI accelerators.

    Stage Core Task Focus on Domestic AI Accelerators Suggested Acceptance Metrics
    Data preprocessing Image-text pair cleaning, deduplication, resolution normalization, text encoding, visual tokenization Overlap CPU-side data processing with device-side computation; use prefetching and multiprocessing Data wait time, throughput, CPU/GPU utilization
    Large-scale pretraining Build foundational representation capability Mixed use of data parallelism, tensor parallelism, and pipeline parallelism; adapt collective communication Communication ratio, step time, loss stability
    Supervised fine-tuning (SFT) Enable instruction-following image generation Long-sequence length distribution is uneven; dynamic padding and sequence packing are needed Whether long sequences cause OOM, instruction-following quality
    Reinforcement learning (RL) Preference optimization and alignment with human aesthetics Rollout throughput, efficiency of switching between inference and training Reward changes, resource utilization, inference-training switching overhead

    1. Data Preprocessing

    The key to data preprocessing is not how fast a single image is processed, but whether the overall pipeline is stable. A common approach is to split decoding, resizing, cropping, and tokenization into independent steps, then use prefetching and multiprocessing so that data supply overlaps with computation tasks.

    If device utilization frequently drops to a low level during training, it often means the data pipeline is not fully unblocked.

    2. Large-Scale Pretraining

    The core challenge of pretraining lies in distributed parallelism and communication adaptation. In domestic AI accelerator environments, it is important to validate the stability and efficiency of collective communication primitives such as AllReduce and AllGather.

    During mixed-precision training, attention should be paid to loss spikes and abnormal gradient norms. If training is interrupted, it should be possible to recover from the latest checkpoint and verify whether the loss curve remains continuous after recovery.

    3. Supervised Fine-Tuning (SFT)

    SFT usually uses less data than pretraining, but the sequence length distribution of instruction-image samples is often more uneven. Padding all samples to a fixed length can create substantial wasted computation.

    A more robust approach is dynamic padding combined with sequence packing, so that samples within the same batch have similar lengths or can be packed efficiently.

    4. Reinforcement Learning Alignment

    The RL stage focuses more on rollout throughput and the efficiency of switching between inference and training. If the inference service and training tasks switch frequently, resource utilization can drop.

    From an engineering perspective, it is recommended to pipeline reward model inference, policy generation, and training updates as much as possible to reduce device idle time.

    4. Hybrid Architecture: Autoregressive Base + Diffusion Decoder

    A route that is often validated is a hybrid architecture consisting of an autoregressive base and a diffusion decoder. The two modules have different responsibilities and can be optimized separately.

    Module Main Responsibility Example Parameter Scale Advantages Limitations
    Autoregressive base Semantic understanding, layout planning, global composition Billion-parameter level Strong at complex instruction understanding and global layout Does not directly restore pixel-level details
    Diffusion decoder Convert high-level semantics into high-quality pixels Tens of billions of parameters Better details, clarity, and image quality Requires stable conditioning and glyph enhancement

    1. Autoregressive Base: Understanding and Planning

    The autoregressive part can be understood as the model’s “planner.” It models discrete image token sequences in an autoregressive way, enabling it to understand complex instructions such as “place the main subject in the center and leave blank space at the top for the title.” It can also perform globally consistent planning for the overall layout.

    Its output determines the skeleton and narrative structure of the image.

    2. Diffusion Decoder: Correct Drawing and Restoration

    The diffusion decoder is usually implemented with a DiT-like structure. It performs denoising restoration under the representation conditioned by the autoregressive base, converting high-level semantics into pixel-level results.

    To improve text generation, the decoder is often paired with a glyph encoder, which injects glyph information of the text to be rendered as an additional condition. When the glyph encoder quality is high and the training data has sufficient coverage, this approach can significantly alleviate issues such as missing or incorrect Chinese characters and disordered character structures. However, it is not an unconditional guarantee. It still depends on training data quality and evaluation criteria.

    The value of this hybrid design is that it preserves the reasoning and planning capabilities of language models while inheriting the pixel-quality advantages of diffusion models. It moves the generation process from “drawing by intuition” to “understanding first, then generating.”

    5. Three Levers for Training Performance Optimization

    To make training both fast and stable on domestic AI accelerators, common optimization approaches can be grouped into three categories: easing dispatch bottlenecks, overlapping communication and computation, and using fused operators.

    1. Multi-Level Pipeline Dispatch for Dynamic Graphs

    In large-scale training, the overhead of dispatching computation instructions from the host to devices can accumulate under high-concurrency scenarios. Multi-level pipeline dispatch for dynamic graphs can split the computation graph and dispatch it asynchronously in batches, allowing dispatch and execution to overlap and reducing device idle waiting.

    This approach is suitable when the training framework supports dynamic graphs or graph operators. During implementation, dependencies must be considered carefully to avoid synchronization errors caused by out-of-order execution.

    2. Multi-Stream Parallelism: Overlap Communication and Computation

    When communication and computation are executed serially, communication time is directly added to the total duration. Multi-stream parallelism allows the communication stream and computation stream to run concurrently, masking each other’s latency and approaching the ideal state of near-zero communication overhead.

    In communication-intensive scenarios such as tensor parallelism, the benefit is usually more obvious.

    A conceptual example is shown below:

    1. # Conceptual example: communication stream and computation stream run in parallel and overlap latency
    2. compute_stream = get_stream("compute")
    3. comm_stream = get_stream("comm")
    4. comm_stream.wait_stream(default_stream)
    5. with stream_context(comm_stream):
    6. all_reduce_async(gradients) # Start gradient synchronization asynchronously
    7. with stream_context(compute_stream):
    8. # Continue computation while communication is in progress
    9. loss = forward_backward(batch)
    10. comm_stream.synchronize() # Wait for communication to finish after computation

    In engineering practice, the benefit should be evaluated based on communication ratio, link bandwidth, and parallelism strategy. If the communication ratio is too high, multi-stream overlap alone may be insufficient. It may also be necessary to adjust the balance between tensor parallelism, data parallelism, and pipeline parallelism.

    3. Ascend-Friendly Fused Operators

    Fused operators can reduce memory read/write operations and kernel launch overhead. Modules suitable for fusion include AdamW optimizer updates, exponential moving average (EMA), normalization layers such as RMSNorm, and fused convolution + activation + normalization operators such as Conv+Act+Norm.

    In engineering documentation, ambiguous labels such as “COC, etc.” should be replaced with explicit wording, for example, a fused convolution + activation + normalization operator, to avoid confusion or misinterpretation.

    The core goal of fused operators is to reduce intermediate tensor materialization, improving both training stability and throughput. However, correctness and performance must be validated on the target hardware.

    6. Key Capability 1: Chinese Text Rendering

    Text rendering is a dividing line for multimodal image generation models. English is mostly based on Latin letters and has relatively simple structures. Chinese, by contrast, has complex strokes, dense glyphs, and strong structural constraints, which place higher demands on the model’s glyph priors.

    A hybrid architecture combined with glyph encoding can help models generate clearer and more structurally correct text in scenarios such as poster titles, annotations in explanatory illustrations, and social media covers.

    For evaluation, multiple dimensions are recommended:

    1. Human readability: Whether the text is clear and free from obvious stroke errors.
    2. Glyph consistency: Whether characters are correct and follow Chinese writing conventions.
    3. Layout consistency: Whether text coordinates with the subject, whitespace, and overall layout.
    4. OCR-assisted validation: OCR tools can be used as a supplementary reference, but they cannot replace human judgment.

    It is important to note that text quality is not determined only by the diffusion decoder. It also depends on the glyph encoder, training data quality, text conditioning method, and coverage of the evaluation set.

    7. Key Capability 2: Arbitrary Aspect Ratio Resolution

    Traditional image generation models often support only fixed aspect ratios, such as 4:3 or 16:9. Outputs are then cropped or resized to fit the target size, which can easily damage composition or cause text misalignment.

    A more practical approach is to improve the tokenizer strategy so that arbitrary aspect ratio generation is supported natively. Within common size ranges, such as 1024×1024 to 2048×2048, the model can support custom aspect ratios. Non-standard resolutions such as 1240×1600 can also be generated directly.

    Approach Method Advantages Risks
    Fixed aspect ratio generation Support only a small set of predefined ratios Simple to implement Limited composition
    Crop after generation Generate in a standard ratio, then crop Reuses existing workflows Composition and text may become misaligned
    Native arbitrary aspect ratio Improve tokenizer and positional encoding strategy Better suited for posters, covers, and other scenarios Higher requirements for training and evaluation

    For scenarios such as posters, social media covers, and information cards that require precise canvas sizes, arbitrary aspect ratio generation can reduce composition damage and text misalignment caused by secondary cropping.

    8. Common Pitfalls and Troubleshooting Checklist

    Symptom Possible Cause Diagnosis Approach Handling Recommendation
    High communication ratio Communication is not overlapped, collective communication is slow, or parallelism configuration is unreasonable Check communication ratio, communication latency, and link utilization Enable multi-stream overlap and adjust TP/PP/DP ratios
    Loss spikes or NaN Mixed precision is unstable, learning rate is too high, or gradients are abnormal Record loss, gradient norm, and loss scaling values Enable loss scaling, gradient clipping, and roll back to a stable checkpoint
    Garbled text Weak glyph conditioning, insufficient training data, or unstable decoder conditioning Sample-check OCR results, glyph conditions, and sample quality Enhance the glyph encoder and add high-quality Chinese text samples
    GPU memory OOM Too many long sequences, memory fragmentation, or inappropriate parallelism configuration Check memory usage curves and sequence length distribution Use dynamic padding, sequence packing, activation recomputation, and adjust parallelism
    Insufficient data supply Decoding, resizing, or tokenization becomes the bottleneck Check data wait time and CPU utilization Add prefetching, multiprocessing, operator splitting, and data caching
    Training cannot be resumed Checkpoints are incomplete or recovery logic has not been validated Compare loss and step count before and after recovery Fix software versions and validate checkpoint save/recovery workflows

    9. Engineering Recommendations

    To reproduce similar capabilities on domestic AI accelerators, the following points should be considered:

    1. Prioritize the full-stack closed loop: Do not validate only isolated components. Run the workflow end to end from the data pipeline to pretraining as early as possible to expose real bottlenecks.
    2. Fix software versions: Record driver versions, framework versions, operator library versions, model configurations, and launch parameters to avoid environment drift that makes results irreproducible.
    3. Align precision and numerical stability: Under mixed precision, pay close attention to loss curves and gradient norms. Enable loss scaling and gradient clipping when necessary.
    4. Strengthen observability: Operator latency, communication ratio, and device utilization must be measurable. Otherwise, it is difficult to locate performance bottlenecks.
    5. Use modular decoupling: The autoregressive base and diffusion decoder can be iterated and validated separately, reducing the complexity of overall debugging.
    6. Use elastic compute resources: General cloud capabilities such as cloud container platforms, object storage, logging, and monitoring/alerting can be used to build elastic training and data management environments, reducing the operational burden of self-built clusters.

    10. Conclusion

    Completing full-process training of multimodal image generation models on domestic chips is essentially a systems engineering effort that requires hardware-software collaboration. At the lower level, it must solve operator, communication, and precision-alignment problems. At the higher level, it must design a proper hybrid architecture and training strategy.

    It transforms the question from “Can it be trained?” into a methodology for “How to train stably, quickly, and correctly.” For teams that want to build generative capabilities on an autonomous compute foundation, the key path from “getting it running” to “making it usable” is to first establish a minimal closed loop, then optimize performance and capabilities.