End-to-End Training of Text-to-Image Models on Domestic AI Chips: Architecture and Engineering Practice

    This article examines the technical requirements for training a text-to-image model end to end on a domestically developed AI hardware and software stack. It covers the motivation for full-pipeline localization, unified generative modeling for image generation, multimodal alignment, diffusion training, operator coverage and fusion, parallel strategy design, training stability and fault tolerance, and inference deployment. It also discusses the remaining gaps in operator ecosystems, framework usability, debugging tools, data scale, and total training compute.

    End-to-End Training of Text-to-Image Models on Domestic AI Chips: Architecture and Engineering Practice

    1. Why “Full-Pipeline Domestic Training” Is a Serious Technical Problem

    Text-to-image generation has advanced rapidly in recent years, and diffusion models have become a mainstream approach. However, most open-source image generation models still rely heavily on a specific international GPU ecosystem for data processing, pretraining, and fine-tuning. Domestic AI chips have more often appeared as alternative options on the inference side. This split model—training on one ecosystem and inference on another—creates a series of engineering problems:

    1. High cost of precision alignment. When training and inference use different hardware, differences in numerical precision and operator implementations can accumulate into deviations in generation quality. Additional calibration and regression testing are often required.
    2. Heavy migration and validation workload. Every model iteration must be validated separately on two software and hardware stacks, which significantly reduces engineering efficiency.
    3. Supply chain and technology autonomy risks. If the training side depends entirely on a single ecosystem, any constraint on compute supply can directly affect the pace of model iteration.

    Therefore, the goal of completing data processing, model training, and inference deployment entirely on a domestic software and hardware stack—while reaching mainstream performance—has much greater technical significance than a standalone model quality evaluation. It validates a complete technical chain: whether domestic compute is sufficient to support large-scale generative model training, whether the software stack has adequate operator coverage and compiler optimization capability, and whether engineering teams have mature parallel training and fault-tolerance solutions.

    2. Model Architecture: Image Generation Under a Unified Generative Paradigm

    2.1 Bringing Image Generation into a Unified Generative Modeling Framework

    Rather than treating a text-to-image model as an independent image module, it is more useful to view it as a natural extension of a unified generative paradigm into the multimodal domain. From this perspective, image generation and text generation are essentially the same type of problem: given a condition—such as text semantics—the model constructs a representation of the target modality either autoregressively or iteratively. This unified paradigm offers several benefits:

    • Reuse of modeling ideas: Training stability techniques, loss design, and sampling strategies accumulated on the language model side can be transferred relatively smoothly to visual generation.
    • More natural multimodal synergy: When the base language model and the image generation model share similar modeling assumptions, coordination between them in multimodal understanding and generation tasks—such as understanding an image before editing it—is cheaper to implement.
    • Unified training pipeline: Data cleaning, tokenization/encoding, and distributed training frameworks can share one infrastructure, reducing engineering complexity.

    2.2 Multimodal Alignment Is Critical

    The quality of alignment between text conditions and image representations directly determines whether generated results “understand” the prompt. Common alignment methods in practice include: using large-scale image-text pairs for contrastive learning pretraining as semantic anchors; freezing the text encoder or using a low learning rate during generation training to prevent text representations from being “pulled away” by the generation task; and mixing pure text, pure image, and image-text pair data in certain proportions during training to maintain the independence of each modality’s representation. Typical symptoms of poorly designed alignment mechanisms include weak prompt adherence and ignored long-text descriptions. These problems are often harder to fix through later fine-tuning than image quality issues.

    2.3 Relationship with the Diffusion Route

    Current mainstream image generation approaches generally adopt the diffusion idea: learning the data distribution through gradual noise addition and denoising. It is important to note that diffusion is a training strategy and sampling paradigm, and it is orthogonal to the choice of model backbone and hardware. Therefore, when training diffusion models on domestic chips, the core work is not in the algorithm itself, but in mapping the algorithm efficiently onto the underlying hardware. This is the focus of the next section.

    3. Domestic Compute Adaptation: The Real Battle Is at the System Layer

    3.1 Operator Coverage and Fusion

    The training hot path of diffusion models contains many specific operators: convolution, attention mechanisms, various elementwise operations related to noise scheduling, and the batch-internal dual forward pass introduced by Classifier-Free Guidance. A domestic NPU operator library may not cover all combinations out of the box. In practice, the following steps are usually required:

    • Use equivalent mathematical transformations to rewrite unsupported operators as combinations of natively supported operators.
    • Apply graph-level fusion to high-frequency operator sequences, such as normalization plus activation, or noise addition plus scaling, to reduce memory movement.
    • For compute-intensive modules such as attention, choose implementations that fit the characteristics of the hardware’s matrix compute units—for example, whether to tile and how to split the head dimension.

    An illustrative optimization approach can be expressed in pseudo-code:

    1. # Before optimization: three separate kernel calls, with intermediate results written back to memory
    2. x = layer_norm(x)
    3. x = gelu(x)
    4. x = x * scale
    5. # After optimization: fused into a single kernel, with intermediate results kept in on-chip memory
    6. fused_layernorm_gelu_scale(x, weight, bias, scale)

    The time saved by such optimizations in a single training step may seem small, but across hundreds of millions of image-text pairs and multiple diffusion time steps, the cumulative benefit can be substantial.

    3.2 Parallel Strategy and Memory Bandwidth Characteristics

    Large-scale training depends on parallel design. When building a hybrid data-parallel and model-parallel solution on a domestic cluster, the partitioning method must be replanned according to the hardware’s communication topology: the split dimension for tensor parallelism should match inter-card interconnect bandwidth; pipeline parallelism stage partitioning should account for transfer latency between devices; and the sharding strategy for optimizer states and activations depends on per-card memory capacity and bandwidth characteristics. A common lesson is that parallel parameters tuned on other hardware cannot simply be copied. Profiling and partitioning experiments should be redone based on the actual communication bandwidth of the target cluster.

    3.3 Training Stability and Fault Tolerance

    Another hidden challenge of large-scale cluster training is stability. Engineering practice usually requires activation checkpointing to trade memory for compute; loss spike monitoring and automatic rollback mechanisms; high-frequency asynchronous checkpoint saving that relies on reliable distributed file storage or object storage; and automatic eviction of faulty hardware nodes together with task resumption. Together, these capabilities determine whether a training job can run to completion with a high effective training time ratio.

    4. Inference Deployment: A Natural Extension of Training-Side Advantages

    The benefits of full-pipeline domesticization are most direct on the inference side. Because training and inference use the same software and hardware stack, numerical behavior is consistent and cross-ecosystem precision calibration is unnecessary. Model formats, compilation toolchains, and runtimes are all connected, resulting in a shorter deployment path. A typical deployment workflow includes model export and graph optimization; precision regression validation for quantization such as INT8/FP8 on domestic chips; batching and dynamic resolution scheduling; and serviceization through container platforms and monitoring/alerting systems. For enterprise users, this means image generation capabilities can be incorporated more smoothly into their own compute infrastructure without maintaining a separate heterogeneous environment for inference.

    5. Challenges and Outlook

    Objectively, full-pipeline domestic training still faces challenges. The richness of the operator ecosystem, framework usability, and maturity of debugging tools still lag behind mainstream international ecosystems. The ceiling of generation quality also remains constrained by data scale and total training compute. However, the emergence of work such as the first open-source image generation model to complete the full training pipeline on a domestic chip platform and reach mainstream performance signals that the domestic AI software and hardware stack is already capable of supporting frontier generative model training. As operator libraries continue to improve and parallel training frameworks mature, models in more modalities will be trained in a “natively domestic” way, and same-stack integration of training and inference will become a more attractive technical route for enterprises deploying large models.