Training Text-to-Image Models End to End on AI Chips Developed in China: A Practical Implementation Guide
Text-to-image models trained end to end on AI chips and software stacks developed in China have become open source, proving that such stacks can carry the full lifecycle of a multimodal model, from operator adaptation and distributed training to inference serving. This article explains what makes text-to-image models difficult, where training stacks typically break when moved to these platforms, and how to get from “it runs” to “it trains reliably and serves fast.” It walks through a four-phase migration path, starting with single-card numerical alignment and ending with inference serving and cost accounting. It also covers precision policy, parallelism choices, memory optimization and reproducibility practices, inference optimization and cloud deployment, and the metrics to use for acceptance. Code snippets illustrate mixed-precision training and parallel configuration. Use this guide if you are porting a diffusion model, evaluating one of these platforms, or planning image generation in production.
1. Why Full-Stack Training on Domestic AI Chips Is Worth Watching
Over the past year or two, multimodal models have become one of the mainstream forms of AI applications. Text-to-image generation is their most representative capability: given a natural-language prompt, the model produces images in styles ranging from photorealistic photos to illustrations and design sketches.
Open-source text-to-image models have recently appeared that were built entirely on AI hardware and software stacks developed in China. This covers the whole pipeline, from data processing and model training to inference deployment.
These projects attract attention not just because they run on China-based chips, but because they validate a complete pipeline. They show that such platforms can independently support the development of large-scale multimodal models, including operator adaptation, distributed training and inference serving.
For enterprises and development teams, these practices offer at least three points of reference:
- Supply-chain control: Training and inference do not depend on a single external hardware vendor, so compute supply is more likely to remain continuous.
- Ecosystem synergy: Experience adapting models, training frameworks, inference engines and cloud platforms can be accumulated and reused, so later model migrations do not have to start from scratch.
- Compliance and cost: Data, model weights and training artifacts stay within the domestic hardware and software stack. This makes it easier to meet data-localization requirements and to optimize costs against local compute pricing structures.
However, training end to end on domestic chips is only a starting point. An open-source model that has finished training is still usually some engineering work away from stable production service. The next section breaks down the model architecture and the main training difficulties. After that, we present a practical phased migration approach and an evaluation framework.
2. First, What Makes Text-to-Image Models Difficult
2.1 Architecture: Three Sub-networks with Different Compute Profiles
Most mainstream text-to-image models follow the latent-space denoising paradigm. Three components work together:
- Text encoder: encodes Chinese or English prompts into sequences of semantic vectors. These are injected into the generation process as conditioning information.
- Denoising backbone: repeatedly denoises noisy image features in the latent space. It is typically built on a Transformer or a convolutional backbone, and accounts for most of the compute in both training and inference.
- Image codec: the encoder compresses images into compact latent representations. The decoder turns them back into pixel images.
Traditional backbones built mainly on 2D convolution rely heavily on how well the hardware implements convolutions. Self-attention-based backbones shift the pressure to attention operators, normalization layers and communication. The choice of sub-network architecture therefore largely determines the main workload when adapting to domestic chips.
2.2 Training Objective: Learning to Denoise at Every Step
Training a diffusion model can be summarized as follows:
- Sample a timestep t and a noise tensor ε at random, then apply the noise schedule to obtain the noisy latent x_t.
- The backbone takes x_t, t and the text condition c as inputs and predicts the noise ε. Depending on the parameterization, it may instead predict the clean sample or the velocity field.
- The error between prediction and ground truth, commonly the mean squared error, serves as the loss. Backpropagation then updates the parameters.
This process looks simple, but it has three engineering challenges:
- Timestep sampling: Timesteps are drawn from a probability distribution, and this sampling has a major effect on both convergence speed and final quality.
- At large t, the task is close to “guessing the semantics from pure noise,” so the supervision signal is weak.
- At small t, prediction accuracy requirements are higher.
- Condition injection: Text conditions are injected repeatedly into the network, for example through cross-attention. If the condition encoding is numerically unstable, the problem persists throughout training.
- Resolution coupling: High-resolution training has a large memory footprint, while low-resolution training is fast, so the two are usually combined in stages. Inconsistent handling of positional encodings and upsampling layers when switching resolutions is a common pitfall.
2.3 Inference Path: Sampling Steps Determine Cost
Generation requires multi-step iterative sampling. Starting from random noise, the model updates the latent step by step according to the noise scheduler and finally decodes it into an image. Sampling steps, guidance strength and random seed together determine quality and latency.
As a result, text-to-image models have a very different workload profile from text generation models:
- Inference latency comes from multiple forward passes rather than a single autoregressive decoding pass.
- Compute per request is far higher than for most online text endpoints, and the memory footprint is more prominent too.
- Benefits from batch scheduling and graph-level optimization matter more.
3. Key Challenges of Moving the Training Stack to Domestic AI Hardware
The difficulty of moving the training pipeline to a domestic AI hardware and software stack usually lies not in “can it run” but in the following areas:
- Operator coverage and implementation quality: Support for some custom or newly introduced operators often lags behind mainstream hardware. Missing operators must either be replaced with equivalent combinations or implemented as custom kernels. Both approaches add work and introduce the risk of numerical differences.
- Numerical precision and mixed precision: The combinations of precision types a chip supports, along with the accumulation precision of fused operators, can all affect convergence and may even cause training to diverge partway through.
- Parallelism and communication: Domestic hardware differs in interconnect topology and collective communication implementations. In large-scale clusters, communication overhead, topology-aware parallel partitioning and fault tolerance directly determine whether training can run stably over the long term.
- Memory and bandwidth management: Caching of activations, memory fragmentation and the way temporary buffers for intermediate operator results are allocated can all become throughput bottlenecks.
- Inference latency and cost: Sampling takes multiple iterations, and the text encoder and codec add further overhead. Without graph-level optimization and batch scheduling, the cost per generated image on domestic hardware can be high.
4. A Phased Approach: From “It Runs” to “It Trains Reliably and Serves Fast”
4.1 Phase 1: Single-Card Numerical Alignment
Start with fixed inputs and fixed parameters, and compare single-step forward results across the two hardware and software environments. Tune the computation precision of attention implementations and normalization one at a time to track down numerical deviations. Once the numbers match, run backpropagation and check that the gradients line up as well.
4.2 Phase 2: Small-Scale Convergence Check
Train for a small number of steps on a small dataset, then compare the loss curve and the trends of key metrics. Most adaptation projects hit problems at this stage: the loss goes down normally, but generated images show artifacts or semantic mismatch. Check the timestep embeddings, the noise scheduler implementation and the order in which text conditions are encoded before adding more training steps.
4.3 Phase 3: Distributed Scaling Validation
Increase the number of cards step by step to evaluate combinations of data, tensor and sequence parallelism. Focus on how scaling efficiency changes as the card count grows, and on fault tolerance when a card misbehaves or communication times out.
4.4 Phase 4: Inference Serving and Cost Accounting
Once training is complete, convert the weights into a format the inference engine supports. Then enable quantization, operator fusion and batch scheduling. Measure average latency and peak memory per generated image, and use these figures to design an elastic scaling strategy.
5. Engineering Practices
5.1 Precision Policy: Deciding What Must Stay in fp32
Mixed-precision training is not simply switching every computation to lower precision. The following principles work well in practice:
- Main computation: The diffusion loss is sensitive to numerical range, so prefer formats with a wider dynamic range, such as bf16. If the hardware only supports fp16, pair it with loss scaling.
- Reduction operators: Run normalization layers, attention softmax, gradient accumulation and optimizer updates in fp32.
- Anomaly detection: Periodically track gradient norms and activation distributions during training so that abnormal spikes can quickly be traced to a specific sub-network.
import mathdef train_step(model, batch, optimizer, cfg, dtype):"""Skeleton of a single training step: mixed precision with numerical safeguards"""# 1) Forward: main compute runs in low precision; normalization falls back to fp32 inside the implementation layeroptimizer.zero_grad()pred = model(batch["latent"], batch["timestep"], batch["text_embed"])loss = ((pred - batch["noise"]) ** 2).mean()# 2) Backward: skip the step immediately on a non-finite loss so corrupted gradients never reach the weightsif not math.isfinite(loss.item()):print("[warn] non-finite loss, skip step")return Noneloss.backward()# 3) Gradient clipping: loss spikes are common in diffusion training, so limit the magnitude of each updategrad_norm = clip_grad_norm(model.parameters(), max_norm=cfg["clip"])optimizer.step()# 4) Monitoring: record gradient norm and learning rate to help trace divergencereturn {"loss": loss.item(), "grad_norm": grad_norm, "lr": cfg["lr"]}
5.2 Parallelism Strategy: Choosing How to Shard by Model Size
When the model fits on a single card, start with data parallelism. Enable gradient accumulation and sharded optimizer states as memory allows, to achieve a larger global batch size. For larger backbones, introduce tensor parallelism to split individual layer matrix multiplications across cards, but keep communication overhead under control.
# Parallelism configuration: choose sharding dimensions based on model sizeparallel_plan = {"data_parallel": data_parallel_size, # Data parallelism: increases throughput"tensor_parallel": 1 if base_bs >= 8 else 4, # Tensor parallelism: reduces per-card memory usage"sequence_parallel": True, # Sequence parallelism: shards activations along long sequences"grad_accum_steps": accum_steps, # Gradient accumulation: effectively enlarges the batch"sharded_optimizer": True, # Sharded optimizer states: lowers peak memory"recompute_granularity": "block", # Granularity of activation recomputation}
5.3 Memory Optimization: From “It Fits” to “It Runs at Full Capacity”
Memory optimization is one of the most cost-effective parts of the adaptation work. A practical order of attack is:
- Recomputation: Enable activation recomputation for compute-heavy, memory-hungry modules, such as the intermediate layers of the denoising backbone. A little extra compute buys memory space.
- Gradients and optimizer states: Combine sharding, mixed-precision storage and gradient accumulation to lower peak memory.
- Data pipeline: Run image-text decoding, resizing and other preprocessing on CPU worker processes or in the data service layer. This keeps them from competing with training for resources.
- Inference: Cache and reuse text condition encodings. Run the codec separately from the backbone to lower the memory peak per request.
5.4 Data and Reproducibility: Keeping Training Results Traceable
The quality of multimodal training depends heavily on the data. The ratio of image-text pairs, the resolution distribution and the quality of text descriptions all affect the final generation style. Recommended practices:
- Fix the data sampling order and random seeds so runs on the two hardware environments can be compared.
- Log loss, gradient norm, learning rate and the source of each data batch at every step so anomalies can be traced back.
- Evaluate image quality on a fixed validation set at regular intervals. Otherwise, watching only the loss go down can lead to misjudgments.
6. Inference Optimization and Production Deployment
Once training is complete, inference typically involves balancing throughput, latency and cost. Common approaches include:
- Quantization: Use low-bit quantization for weights and activations. This reduces memory usage and improves throughput while keeping accuracy loss under control.
- Graph-level optimization: Use an inference engine to perform operator fusion, constant folding and memory reuse to reduce scheduling overhead.
- Batch scheduling: Combine multiple requests into a single batch to improve hardware utilization.
- Staged deployment: Scale the text encoder, denoising backbone and codec separately according to their different load profiles.
- Cloud deployment: Training data and model weights can be stored in object storage, and model versions released through a container platform. In front of the inference service, place a load balancer and autoscaling. Use logging services and monitoring alerts to keep tracking latency and error rates. Cloud platforms such as Baidu AI Cloud provide this kind of general-purpose infrastructure and can serve as the runtime foundation for deploying China-trained models.
7. Evaluation and Acceptance Metrics
Whether training on domestic hardware is truly production-ready requires a set of quantifiable acceptance criteria. Build baselines along the following dimensions:
| Dimension | Metrics | Notes |
|---|---|---|
| Generation quality | Objective metrics such as FID and CLIP Score | Compare against open-source models on the same test set |
| Semantic consistency | Human preference evaluation | Semantic match and text rendering quality for Chinese prompts |
| Training efficiency | Per-card throughput, peak memory, scaling efficiency | Measure the benefits of the parallelization scheme as the card count grows |
| Stability | Recovery time after interrupted training, ability to isolate faulty cards | Critical for long-running training |
| Inference cost | Latency and memory usage per generated image | Determines the economics of online service |
Metric comparisons must use the same data, random seeds and evaluation scripts. Otherwise, the results are not comparable.
8. Common Issues and How to Avoid Them
- Loss looks normal, but images are blurry or semantically misaligned: First check that the text condition encoding order and timestep embeddings are aligned, then check the details of the noise scheduler implementation.
- Loss suddenly diverges partway through training: This usually comes from an anomalous data batch or a gradient spike. Anomaly detection, gradient clipping and skipping bad steps can mitigate it.
- Scaling efficiency drops sharply as the card count grows: Check whether the communication topology and parallel partitioning match the hardware interconnect structure, and adjust the tensor parallelism degree if necessary.
- Low inference throughput and high cost: Enable batch scheduling and quantization first, then assess whether the backbone needs distillation or a reduction in sampling steps.
9. Conclusion
Text-to-image models trained end to end on domestic chips show that the AI hardware and software stacks developed in China can support the complete R&D lifecycle of multimodal models. For enterprises, the real value lies not just in having a downloadable model, but in distilling reusable adaptation methods from it: single-card numerical alignment, small-scale convergence checks, distributed scaling and inference serving, each with clear checkpoints and acceptance criteria. Applying this methodology to your own business models is how you build multimodal capabilities on domestic compute that can keep iterating sustainably.
