End-to-End Training of Multimodal Text-to-Image Models on Domestic AI Chips
This article explores the engineering practice of training multimodal text-to-image generation models end to end on domestic AI chips. It explains why full-stack training is more challenging than inference migration, covering key areas such as operator coverage, distributed parallelism, numerical stability, data quality, framework adaptation, inference deployment, and model evaluation. It also provides practical recommendations for teams that want to build similar capabilities on domestic AI compute stacks.
1. Reassessing the Role of Text-to-Image Models: More Than Generating an Image
At first glance, the core capability of a text-to-image generation model appears straightforward: it accepts text prompts in Chinese or English and produces images in styles such as photorealistic rendering, illustration, or design sketches. These models are often used as part of a larger multimodal model system.
However, if a text-to-image model is understood only through the quality of its generated images, it is easy to overlook its real engineering value.
The more important technical distinction lies in where the training stack is built. There are two common paths in the industry:
- Train the model on mainstream accelerators first, and then try to migrate only the inference workload to domestic AI chips.
- Build the entire workflow on a domestic AI software and hardware stack, including data processing, model training, and inference deployment.
Only the second path forms a complete, end-to-end, controllable loop. It is significantly more difficult, and it also carries greater engineering and strategic value. This article focuses on this more challenging path.
2. Why “Fully Domestic” Is the Real Technical Dividing Line
Getting a model to run and training a model from scratch are separated by a huge engineering gap.
Inference migration usually involves adapting operators and verifying numerical consistency, and its cost is relatively controllable. End-to-end domestic training, however, requires the entire pipeline to maintain stable performance and convergence quality. Every stage—data, operators, communication, and numerical precision—can become a bottleneck.
The main challenges can be grouped into four categories:
Operator coverage and graph compilation: Training requires a large number of forward and backward operators. A missing operator or a performance bottleneck in any key operator can limit overall throughput. Robust graph optimization and compilation capabilities are therefore essential.
Distributed parallelism and communication: Large model training often relies on hybrid parallelism strategies, including data parallelism, tensor parallelism, and pipeline parallelism. The efficiency of collective communication across chips directly determines how well the system scales.
Numerical precision and training stability: Different hardware may implement floating-point operations differently. If precision alignment is not handled carefully, training may suffer from abnormal gradients or slow convergence.
Memory management and throughput optimization: Image generation models place heavy pressure on memory during stages such as sampling and attention computation, requiring careful scheduling and optimization.
3. Data: The First Gate for Training Quality
No matter how capable the hardware stack becomes, the scale and quality of training data still determine the upper limit of the model.
Text-to-image training depends on large-scale, high-quality image-text paired data. The key data engineering tasks include:
Cleaning and deduplication: Filtering out low-resolution images, watermarks, and irrelevant text, while removing near-duplicate samples to avoid memorization and overfitting.
Quality scoring and selection: Using image aesthetic scores and text-image semantic consistency scores to retain higher-value samples.
Chinese corpus governance: If both Chinese and English are supported, constructing Chinese image-text pairs, tokenization, and normalizing descriptions become especially important.
At the engineering level, large volumes of data are typically stored in object storage and fed into the training cluster through high-throughput reading pipelines. Data versions, sample lists, and metadata can be tracked through logging services and monitoring alerts, helping ensure reproducibility across the entire training process.
4. Training: Framework Adaptation and Parallelism
The depth of framework adaptation on domestic AI chips largely determines how much hardware performance can be released. A practical engineering path looks like this:
# Pseudocode: end-to-end training loop (language/framework agnostic)for epoch in range(num_epochs):for batch in dataloader: # paired text-image batchwith mixed_precision(): # balance memory usage and stabilitytext_feat = text_encoder(batch.text)latent = vae_encode(batch.image)noise = sample_noise(latent.shape)pred = unet(noisy_latent=add_noise(latent, noise),timestep=t, condition=text_feat)loss = mse_loss(pred, target_noise) # denoising regression targetloss.backward() # backward pass (relies on domestic operator stack)optimizer.step()optimizer.zero_grad()if step % save_interval == 0:save_checkpoint(...) # periodically save checkpoints
Along with this training loop, several engineering considerations are particularly important:
Mixed-precision strategy: Mixed precision can reduce memory usage while maintaining convergence. It needs to be calibrated according to the floating-point characteristics of the target domestic hardware.
Parallel partitioning: The text encoder, diffusion backbone, and VAE have different memory and compute profiles. They may use different parallelism settings.
Checkpoints and resumable training: Long-running training must support stable recovery. Checkpoints are best stored in object storage for better reliability.
Training observability: Metrics such as loss curves, throughput, and memory usage should be connected to monitoring and alerting systems. This helps identify issues such as gradient explosion or idle compute resources quickly.
5. Inference Deployment: From Training Results to Usable Services
Once training is complete, inference-side optimization is still essential for turning the model into a practical service.
Common techniques include:
- Graph compilation optimization
- Operator fusion
- Weight quantization
- Dynamic batching
These methods help reduce latency and improve throughput.
For deployment, container platforms are often used to support elastic scaling and canary releases. Logging services and monitoring alerts can then continuously track service health and resource utilization.
It is important to note that training and inference should ideally be completed within the same domestic software and hardware ecosystem. This allows operators, numerical precision, and quantization strategies to be designed together, avoiding numerical drift caused by a mismatch between training and inference environments.
6. Evaluation: Multiple Dimensions, Not Just Generated Images
Evaluating a text-to-image model should not rely only on a few sample images. A more reliable evaluation system should cover multiple dimensions, including:
Text alignment: Whether the generated image accurately reflects the meaning of the prompt.
Image quality: Clarity, structural correctness, and absence of distortion.
Aesthetic quality: Composition, color, and lighting.
Chinese-English consistency: Whether prompts in both Chinese and English can produce stable and reasonable results.
7. Practical Recommendations for Developers
For teams that want to build similar capabilities on domestic AI compute stacks, a practical strategy is to start small and scale gradually:
Assess ecosystem maturity first: Confirm whether the target chip provides sufficient operator coverage and acceptable performance.
Build a small end-to-end pipeline: Use a small model and limited data to validate the full workflow from data processing to training and inference, and identify bottlenecks.
Establish observability and reproducibility from day one: Include monitoring, logging, and checkpoints in the engineering process from the beginning.
Scale up gradually: Once the pipeline is stable, increase the data volume and model size while balancing convergence quality and throughput.
Overall, completing end-to-end training of a text-to-image model on domestic AI chips marks an important step for domestic AI compute stacks: moving from “capable of inference” to “capable of training.”
This task tests not only the performance of individual hardware components, but also the coordination of the entire system, including data, frameworks, operators, communication, and deployment. For developers, understanding these system-level challenges and establishing a complete engineering methodology is the key to following this technical trend.
