Engineering Practices for Training Text-to-Image Models on a Domestic AI Software and Hardware Stack in China
This article examines the engineering challenges and practices of running the full text-to-image model lifecycle on a domestically developed AI software and hardware stack in China, covering data processing, training, and inference deployment. It explains the technical baseline of diffusion-based text-to-image models, analyzes key obstacles related to operator coverage, framework adaptation, and numerical stability, and presents practical optimizations for distributed training, memory management, data pipelines, fault tolerance, inference acceleration, evaluation, and operations. It also offers recommendations for teams that need to validate the full stack and maintain output quality alignment across different hardware environments.
1. Why a Full-Stack Domestic AI Hardware and Software Stack Is a Separate Technical Problem
When discussing text-to-image or multimodal generation models, many people first compare generation quality. However, from a higher-level perspective, a separate and technically meaningful question is whether a model can run through data processing, model training, and inference deployment on a domestic AI software and hardware stack.
In practice, there are two different paths:
- Inference migration: The model is first trained on mature general-purpose accelerators, and then the inference stage is migrated to domestic chips. This approach is relatively risk-controlled, but the difficulty is mainly focused on making inference work. The training side still depends on the existing ecosystem.
- Full-stack native implementation: Data processing, pretraining, fine-tuning, and inference deployment are all completed on domestic chips and the supporting software and hardware stack. Its value is not replacing one component, but connecting the complete chain: framework, operators, compilation, chips, and deployment.
The second path is the real challenge. It means there is no room for workarounds such as using something temporarily or relying on compatibility layers. Any weak point in any stage may be amplified during long-cycle training. Understanding this is a prerequisite for evaluating such solutions.
2. Technical Baseline of Text-to-Image Models: What Do They Actually Depend On?
To understand the challenges of full-stack domestic AI hardware and software, it is necessary to break down the computational modules of a typical text-to-image model. Most mainstream approaches are based on the diffusion model paradigm. Their training and inference processes generally include three key parts:
- Text encoder: It encodes natural-language prompts into conditional vectors and usually depends on large-scale pretrained language model capabilities.
- Denoising backbone network: It uses text conditions as cross-attention inputs, repeatedly predicts and removes noise in the latent space, and is mostly based on U-Net or Transformer structures.
- Latent codec: It converts between pixel space and low-dimensional latent space, balancing generation quality and compute cost.
This structure places clear demands on the hardware stack. It must support intensive operators such as convolution, normalization, and attention, while also maintaining numerical stability in long-sequence and high-resolution scenarios. In addition, the training stage often requires multi-GPU or even multi-machine parallelism, which places extremely high demands on the maturity of communication operators and collective communication libraries. If any basic operator is missing or does not meet precision requirements, it may cause loss not to converge or generation results to collapse.
3. Three Core Challenges of Full-Stack Domestic AI Hardware and Software
1. Operator coverage and operator library maturity
Deep learning frameworks eventually break down computation graphs into many basic operators for hardware execution. The common problem with domestic AI stacks is not that operators do not exist, but that operator coverage is incomplete or operator performance has not been well optimized. Some less common but critical operators, such as specific attention variants or dynamic shape support, may be missing, or may exist but have far-from-mature performance. Possible responses include:
- Using the operator extension mechanisms provided by the framework to fill gaps.
- Reducing kernel launch overhead through graph compilation and operator fusion.
- Performing targeted optimization for high-frequency operators.
2. Framework adaptation and computation graph compilation
Whether the framework layer can smoothly translate a model into instructions that can be executed by a chip determines the development experience. The ideal state is that model code barely needs to be changed, and switching to a new backend is sufficient to run it. In practice, developers often need to handle adaptation issues such as dynamic control flow, custom operator registration, and mixed-precision strategies. A pragmatic approach is to first validate the pipeline on a minimal viable model, then gradually expand it to the full model, avoiding the high cost of debugging everything at once.
3. Numerical precision and stability
Training is extremely sensitive to numerical ranges. Full-stack domestic AI hardware and software must focus on validating mixed-precision strategies across different operators to avoid gradient overflow or gradient underflow. Common methods for ensuring stable long-cycle training include:
- Gradient scaling.
- Keeping high-precision computation for critical operators.
- Using reasonable initialization.
4. Key Engineering Practices During Training
After the pipeline is connected, engineering optimization during training mainly focuses on how to achieve stable convergence under limited memory and compute resources:
- Combination of distributed parallel strategies: Data parallelism improves throughput, tensor parallelism splits single-layer computation, and pipeline parallelism divides model layers. These approaches often need to be used together, and their ratios need to be tuned according to the cluster interconnect bandwidth.
- Memory optimization: Gradient accumulation simulates a large batch under small memory conditions. Activation checkpointing trades compute for memory. Mixed precision reduces the storage overhead of weights and activations.
- Data pipeline: Large-scale loading and preprocessing of text-image pairs must use asynchronous prefetching and multi-process decoding. Otherwise, data supply may become an invisible bottleneck that limits GPU or chip utilization.
- Fault tolerance and checkpoint resume: Long-cycle training must regularly save checkpoints. It is also important to monitor metrics such as loss, gradient norm, and learning rate so that anomalies can be detected early.
The following is framework-neutral pseudocode for the training loop, used to explain the core logic. It does not involve any specific framework or hardware interface:
# Example code: pseudocode for a diffusion-based text-to-image model training loopfor step in range(total_steps):# 1. Encode text conditionstext_cond = text_encoder(tokenize(batch.prompts))# 2. Add noise to clean images according to a random timestep tt = sample_timesteps(batch_size)noisy_image, noise = forward_diffusion(batch.images, t)# 3. The backbone network uses text conditions as cross-attention inputs and predicts the noisewith autocast(dtype="fp16"): # Mixed precision reduces memory and compute costpred_noise = unet(noisy_image, t, text_cond)loss = mse_loss(pred_noise, noise)# 4. Backward propagation + gradient scaling + parameter updateloss.backward()grad_scaler.step(optimizer)grad_scaler.update()optimizer.zero_grad()
5. Key Points for Inference Deployment and Acceleration
Model convergence is only the first half. Inference deployment determines whether the model can be served at scale:
- Quantization and operator fusion: Converting some weights into lower-precision representations and fusing adjacent operators into a single kernel are two major approaches for reducing latency and improving throughput.
- Graph-level optimization: Computation graph rewriting can eliminate redundant nodes and reuse common subgraphs, reducing scheduling overhead.
- Serving and batching: Dynamic batching balances latency and throughput. Caching reusable results, such as text encoding outputs, can avoid repeated computation.
- Asynchronous execution and pipelining: Multi-step denoising can naturally be pipelined. Reasonable scheduling can significantly improve output per unit of compute.
The following shows example pseudocode for denoising sampling during inference:
# Example code: multi-step denoising sampling for inferencelatent = randn(shape=(1, C, H, W))for t in reversed(timesteps):pred_noise = unet(latent, t, text_cond) # The accelerated graph can use quantization and operator fusionlatent = scheduler.step(latent, pred_noise, t)image = vae.decode(latent) # Decode the latent space back into pixel space
6. Evaluation and the Engineering Closed Loop
The quality of text-to-image models cannot be evaluated using only a single metric. Objective metrics, such as text-image alignment and image quality measures, need to be combined with human subjective evaluation to cover dimensions such as semantic consistency, visual appeal, and detail plausibility. In a domestic-stack scenario, it is also necessary to establish a precision alignment baseline: comparing the outputs of the same model on different hardware stacks, quantifying differences, and ensuring that migration or full-stack operation does not cause a noticeable quality regression.
At the engineering level, it is recommended to use a logging service to accumulate metric data from training and inference, and to build a monitoring and alerting system that covers chip utilization, memory water level, latency, and error rate. A container platform can help make training and inference environments reproducible and support elastic scaling. Data, models, and checkpoints can be managed in object storage, forming a traceable asset management loop.
7. Recommendations for Teams Implementing This Work
- Validate the minimal pipeline before scaling up: Use a lightweight model to connect the full data-training-inference pipeline and expose operator and framework issues. This costs far less than directly running a large model.
- Treat precision alignment as a first-class concern: Establish cross-stack output comparison mechanisms from the beginning, so that later quality regressions are easier to locate.
- Pay attention to the data pipeline and fault tolerance: Much compute waste does not come from operators, but from insufficient data supply and interrupted training.
- Keep the architecture neutral: As much as possible, use standard framework operators and common parallel strategies, reduce strong coupling with a single backend, and leave room for multi-stack deployment.
In summary, the real barrier to completing text-to-image model training on a domestic AI software and hardware stack is not any single-point technology, but the systematic coordination of operators, frameworks, precision, parallelism, deployment, and the engineering closed loop. Only when the weak points at every layer are addressed can a model not only run, but also run stably and produce good results.
