Training Multimodal Image Generation Models End-to-End on Domestic AI Accelerators: Practice and Lessons Learned
An end-to-end training pipeline—covering data preprocessing, training, and inference on a single domestic AI software and hardware stack—can avoid the accuracy drift and performance gaps that arise when models are trained on GPUs and later ported to domestic chips for inference. This article examines the engineering principles behind fully domestically trained multimodal image generation models, including operator-level adaptation, hybrid parallelism, numerical stability, data pipelines, training fault tolerance, and inference optimization. It offers a practical reference for teams planning to train large models on domestic compute infrastructure.
Why “Full-Pipeline Training on Domestic Chips” Deserves Its Own Discussion
In today’s large model ecosystem, a common path looks like this: pretrain the model on mainstream GPU clusters, then port the weights to domestic chips for inference adaptation. The problem with this route is that the software and hardware stacks for training and inference are fragmented—issues such as accuracy misalignment, operator differences, and inconsistent numerical behavior accumulate over time, ultimately producing a familiar outcome: “great training results, degraded deployment quality.”
A more thorough alternative is to complete the entire pipeline—data preprocessing, model training, and inference deployment—on the same domestic AI software and hardware stack. This approach offers three key benefits:
- Guaranteed numerical consistency. Training and inference share the same operator library and precision strategy. Numerical errors accumulated during mixed-precision training can be fully validated at deployment time, avoiding quality regression caused by train-inference mismatch.
- Deep, end-to-end performance optimization. Since training and inference run on the same architecture, optimizations such as operator fusion, communication tuning, and memory layout can be shared across the entire pipeline instead of being tuned in isolation.
- Supply chain and technology autonomy. Model capability is built on the domestic software and hardware stack, so later iterations, capacity expansion, and secondary development are not constrained by external factors.
Recently, open-source multimodal image generation models trained entirely on domestic compute have reached quality on par with mainstream results, validating this route. This article does not focus on any specific product; instead, it breaks down the engineering logic behind such practices from a general technical perspective.
Technical Foundation: The General Architecture of Text-to-Image Diffusion Models
Before discussing chip adaptation, let’s review the general technical form of text-to-image generation models. Most current mainstream solutions are based on the diffusion model paradigm and consist of several core modules:
Text encoder (understands prompt semantics)↓Conditional injection mechanism (Cross-Attention, etc.)↓Denoising network (U-Net or Diffusion Transformer)↓VAE encoder-decoder (converts between latent and pixel spaces)
During training, noise is added to real images, and the denoising network learns to “recover the image from noise,” with text conditions enabling semantically controllable generation. At inference, the process starts from random noise and, through multiple iterative denoising steps, eventually decodes into an image.
This architecture drives three major compute requirements:
- High-throughput matrix operations: The denoising network’s backbone consists of large-scale attention and convolution/linear transformations—a typical compute-intensive workload;
- Large memory and high bandwidth: Diffusion model training must hold the text encoder, denoising network, VAE, and other modules in memory simultaneously, with significant activation memory footprint;
- Long-duration stable training: Large-scale datasets often contain hundreds of millions of samples, training cycles are measured in weeks, and cluster stability requirements are extremely high.
Key Challenges on Domestic Chips and Adaptation Strategies
3.1 Operator-Level Adaptation
Architectural differences between NPUs and GPUs mean that operator combinations validated on GPUs cannot simply be copied over. Common issues include:
- Highly optimized fused operators available on GPUs may degrade into multiple independent operators on NPUs, causing repeated reads and writes of intermediate results to memory;
- Attention mechanism implementation paths differ, and performance gaps can widen dramatically for long-sequence workloads;
- Certain numerical functions (such as normalization and specific activation function variants) behave differently in low-precision implementations.
The engineering response is to consider the target chip’s characteristics at the model design stage, rather than “adapting after the fact” once training is done:
# Example: run operator-level benchmarks before training# to determine the optimal implementation pathops_to_bench = ["fused_attention", # fused attention operator"layernorm_fwd", # normalization forward pass"conv_act_fuse", # convolution + activation fusion]# Based on measured throughput and numerical error,# select operator combinations that are NPU-friendly
Typical practices include: manually fusing multiple small operators into larger ones natively supported by the chip; adjusting memory access patterns to match the NPU’s on-chip cache granularity; and choosing acceleration implementations for attention computation that suit the local hardware.
3.2 Parallelism Strategies and Communication Optimization
Large-scale training requires parallelism, and since domestic clusters have different interconnect topologies than GPU clusters, parallelism strategies must be redesigned:
- Data parallelism: Relatively universal; the focus is on gradient synchronization with communication compression and pipelined overlap;
- Tensor parallelism: Highly sensitive to communication bandwidth; partition granularity should be determined by intra-node interconnect characteristics;
- Pipeline parallelism: Well-suited for deep denoising networks; the key is a scheduling strategy that minimizes pipeline bubbles;
- Optimizer state parallelism: Diffusion models have substantial parameters and optimizer states; sharding optimizer states can significantly reduce per-device memory pressure.
In practice, hybrid parallelism is often adopted, combined with communication-computation overlap (hiding gradient synchronization inside backward computation) to improve overall efficiency. High-end training server form factors optimized for Transformer and multimodal workloads provide the hardware foundation for these hybrid parallel strategies.
3.3 Precision and Stability
Mixed-precision training on domestic chips requires additional validation: loss scaling strategies, forced high precision for specific layers (such as normalization and summation reductions), and the behavior of diffusion-specific noise scheduling parameters under low precision. A practical recommendation is to establish a layer-by-layer numerical alignment baseline—regularly comparing intermediate results against a reference implementation early in training to catch precision drift before it compounds.
Data and Training Engineering Practices
4.1 Data Pipeline
The data pipeline for text-to-image models involves four stages: collection, cleaning, annotation, and pairing. For multilingual scenarios, the proportion of data supporting the target language’s semantic understanding must be carefully managed to ensure generation quality for prompts in that language. The data pipeline itself is a typical IO- and CPU-intensive workload, well-suited to elastic compute and object storage infrastructure, using parallel decoding and distributed caching to keep up with training throughput.
4.2 Training Monitoring and Fault Tolerance
For long-duration training, stability matters more than single-point performance. Engineering requirements include:
- Multi-dimensional monitoring of loss curves, gradient norms, and generated sample quality—not just loss alone;
- Regular checkpointing, with precise state restoration for resuming after interruptions (including data loading position and learning rate schedule progress);
- Periodic manual sampling and review of generated outputs to catch mode collapse or semantic drift early.
If training jobs run on a cloud-based AI training platform, you can directly reuse generic capabilities such as containerized scheduling, logging services, and monitoring/alerting, keeping your focus on the model itself.
Inference Deployment: Turning Training Results into Business Value
Training completion is only the first step. Text-to-image model inference deployment has several typical optimization points:
- Denoising step compression: Through distillation or more efficient samplers, tens of denoising steps can be compressed to a few, substantially reducing per-generation latency;
- Quantization: Applying INT8 or lower-precision quantization to the denoising network—quantized operators’ accuracy retention must be validated on the target NPU;
- Dynamic batching: Image generation is a classic bursty workload; request queues and dynamic batching improve per-device utilization;
- Service wrapping: Encapsulate the model with a standard inference serving framework, using message queues for traffic shaping and monitoring/alerting to ensure production stability.
A simplified example of a service invocation:
# Example: typical invocation contract for a text-to-image serviceresult = image_client.generate(prompt="solar system diagram, children's science book style",steps=20, # number of denoising stepsguidance=7.5, # guidance scalesize="1024x1024",)
Summary and Outlook
Multimodal image generation models trained entirely on domestic chips validate an important technical route: deep adaptation beats post-hoc porting. The underlying methodology—considering hardware characteristics at the operator design stage, redesigning hybrid parallelism strategies, and ensuring full-pipeline numerical consistency—offers valuable reference for any team planning to train large models on domestic compute.
For developers looking to replicate this kind of work, the recommended sequence is: operator benchmarking → small-scale training validation → parallelism tuning → full-scale training → inference quantization and deployment, establishing numerical and performance baselines at every stage. As the domestic software and hardware stack continues to mature, end-to-end domestically trained multimodal models are moving from “possible” to “excellent”—a technical direction worth sustained attention in the period ahead.
