Training Image Generation Models on Domestic Chips: Technical Pipeline and Adaptation Challenges
This technical review explains the model pipeline and engineering checks for training text-to-image generation models on domestic chips. It does not report measured results for a specific chip or model. The article covers prompt encoding, text-image alignment, denoising, decoding, the trade-offs of full-pipeline domestic compute, operator compatibility, memory and bandwidth management, distributed communication, training stability, evaluation templates, and acceptance checklists. Readers can reuse the checklists, evaluation templates, and acceptance metrics in their own project assessments, setting thresholds according to their own tasks.
This article is a technical review. It explains the model pipeline and engineering inspection methods; it does not include measured results for a specific chip or model. Readers can apply the checklists, evaluation templates, and acceptance metrics directly to their own project assessments. Thresholds should be set according to the task.
1. What Text-to-Image Models Do
Text-to-image generation creates images that meet requirements based on natural-language descriptions. For example, if a user enters “draw a solar system diagram in the style of a children’s science picture book,” the model must identify the main objects, spatial relationships, visual style, and intended use of the image, then translate that semantics into a concrete picture.
Early text-to-image models mainly addressed three questions: whether text semantics were understood correctly, whether the image matched the description, and whether image quality was stable. Today’s multimodal generation models also handle more complex requirements, such as specifying the position, size, and color of multiple objects, or generating posters, infographics, and visual content with large amounts of text. These tasks require a certain level of compositional reasoning.
Therefore, when evaluating an image generation model, it is advisable to examine at least the following metrics:
- Understanding of long-text prompts;
- Preservation of relationships among multiple objects;
- Accuracy of Chinese and English text generation;
- Spatial consistency in complex compositions;
- Style control and detail rendering;
- Stability under multi-round edits;
- Inference speed and per-image deployment cost.
2. Value and Cost of Full-Pipeline Training on Domestic Compute
In typical projects, data cleaning, pretraining, fine-tuning, evaluation, and inference deployment are often spread across different hardware and software environments. Some teams first train on one platform and then migrate to another platform for inference. This approach can be deployed quickly, but migration often introduces operator incompatibility, performance degradation, accuracy deviations, and higher maintenance costs.
Completing the entire training pipeline on the target domestic platform can expose these problems earlier. The table below summarizes its value, cost, and applicable boundaries.
| Dimension | Value | Cost | Applicable boundary |
|---|---|---|---|
| Software-hardware co-validation | Data preprocessing, mixed precision, distributed communication, checkpoints, and inference deployment are tested in the same pipeline | Requires investment in low-level adaptation staff, and operator coverage must be confirmed item by item | Applicable when the team has low-level adaptation capabilities |
| Reduced migration risk | Training and inference environments are consistent, so there are fewer sources of cross-platform accuracy deviation | Early environment setup takes longer | Projects with a long-term fixed deployment environment |
| Engineering maturation | Network partitioning, batch design, and memory strategy must be redesigned according to device characteristics | Model structure requires additional adjustment, and some structural flexibility is reduced | Large-scale training or high-resolution generation |
Known limitations should also be stated. Coverage of third-party libraries and community ecosystems may be incomplete, and some operators may have only generic implementations without targeted optimization. Whether a platform meets requirements should be judged based on official documentation and the team’s own measured results, not theoretical peak compute.
3. Technical Pipeline for Image Generation
Text-to-image generation usually includes four stages. Each stage has a concrete impact on adaptation to domestic chips, as described below.
3.1 How Are Prompts Encoded?
User prompts first pass through a text encoder and are converted into vectors that the model can process. High-quality encoding requires recognizing nouns, modifier relationships, action relationships, spatial relationships, and style requirements.
For example, the prompt “generate a solar system diagram suitable for a children’s science reader, with a clean and bright layout, planets arranged along their orbits, and Chinese labels next to each celestial body” contains the following semantics:
- Subject: solar system;
- Use: children’s science reader;
- Style: clean and bright;
- Structure: planets arranged along orbits;
- Additional requirement: add Chinese labels.
If any of these parts is lost during encoding, the final image may have the wrong style, missing objects, or a confused layout.
Impact on adaptation: text encoders are usually based on attention structures, and the efficiency of attention operators on the target platform directly determines the time spent in this stage. The encoding stage should be timed separately during testing.
3.2 How Are Text and Images Aligned?
Text vectors need to establish a correspondence with visual space. The model must know what shapes, textures, and positions “tree,” “building,” “person,” and “poster” correspond to in an image.
When alignment is insufficient, common problems include:
- Objects are present but attributes are wrong;
- Colors and quantities do not follow the description;
- Multiple objects merge into each other;
- The relationship between人物 actions and the scene is unnatural;
- The overall style does not match the text requirements.
Therefore, training usually requires large amounts of high-quality image-text data, and noise is reduced through deduplication, quality filtering, and semantic annotation.
Impact on adaptation: if the data pipeline runs slowly on the target platform, the entire training process slows down. Data preprocessing throughput needs to be measured separately.
3.3 How Does an Image Move from Noise to a Clear Picture?
Mainstream methods start from random noise and gradually obtain a clear image through multiple rounds of computation. At each step, the current image state is adjusted according to the text condition so that it gradually approaches the target semantics.
This process is computationally demanding. During high-resolution and large-batch training, compute, memory usage, and communication overhead all increase significantly. Common engineering methods include mixed precision, gradient checkpointing, tensor parallelism, data parallelism, and memory reuse.
Impact on adaptation: during inference, the same network is called repeatedly, and the number of denoising steps is roughly proportional to latency. During training, intermediate results at each step occupy memory. Adaptation should record the number of steps, per-step time, and peak memory at the same time.
3.4 Where Do Decoding and Post-Processing Run?
The generation process produces a compressed visual representation, which finally needs to be restored to a displayable image by a decoder. The decoder is usually lighter than the main network, but it still consumes considerable compute during high-resolution or multi-image batch generation.
Some systems also add safety review, resizing, format conversion, and quality checks. These steps should be included in the complete inference pipeline; measuring only the speed of the model body is not enough.
4. Four Adaptation Challenges on Domestic Chips
4.1 Operator Compatibility
Training involves many types of operators, including matrix multiplication, normalization, attention, activation functions, convolution, sampling, and random number generation. Different platforms may have different requirements for operator support, input layouts, and data types.
During migration, focus on checking:
- Whether core operators have high-performance implementations;
- Whether data format conversions occur frequently between operators;
- Whether dynamic shapes cause compilation or execution efficiency to drop;
- Whether numerical instability occurs under mixed precision;
- Whether custom operators can participate in automatic differentiation.
Troubleshooting example. If “operator not registered” or a compilation failure appears during training startup, the log usually contains the operator name, input shape, and data type. You can locate the issue as follows:
- Record the operator name, dtype, and shape in the error;
- Call the operator separately with a fixed small-scale input and compare its output with the reference implementation;
- If it fails only under dynamic shapes, reproduce with a fixed shape to determine whether the problem comes from compilation or execution;
- Calculate relative error during validation. If the relative error exceeds the threshold agreed by the team, the operator cannot enter formal training.
If a frequently used operator can only be executed in an inefficient way, overall training speed will drop significantly even if theoretical compute is high.
4.2 Memory and Bandwidth Management
Image generation models have large parameter counts and many intermediate activations. During training, gradients, optimizer states, and temporary tensors must also be stored, so actual memory requirements are much higher than the parameters alone.
Common optimizations include:
- Use mixed precision to reduce tensor storage;
- Recompute some intermediate results;
- Use gradient accumulation to reduce the single-batch size;
- Arrange tensor lifetimes reasonably;
- Reduce unnecessary data copies;
- Design partitioning schemes according to device capacity.
Insufficient memory does not necessarily cause a direct error. It may also appear as frequent swapping, lower throughput, or training instability. Therefore, peak memory and throughput curves should be recorded and analyzed together.
4.3 Distributed Communication
Large-scale training requires coordination across multiple devices. Data parallelism, tensor parallelism, and pipeline parallelism all depend on efficient communication. Communication efficiency depends on inter-device bandwidth as well as communication topology, message size, synchronization frequency, and partitioning strategy.
If compute is fast but communication is slow, device utilization will drop. Common improvements include:
- Overlap compute and communication as much as possible;
- Reduce frequent synchronization of small messages;
- Choose a parallelism strategy based on model structure;
- Compress or bucket gradient communication;
- Set global and local batch sizes reasonably.
During measurement, record the proportion of per-step communication time and how throughput changes as the number of devices increases. If throughput does not grow with the number of devices, communication is likely the bottleneck.
4.4 Training Stability
Image generation models are sensitive to training stability. Learning rate, batch size, precision strategy, data quality, and random seeds all affect results. After migrating to a new environment, convergence speed and generation quality may change even if the model structure remains unchanged.
Therefore, verifying “whether it can run” is not enough. You should also compare:
- Whether the loss curve is stable and whether NaN appears;
- Whether training speed remains sustained;
- Whether generated samples at different stages are normal;
- Whether accuracy changes are within an acceptable range;
- Whether results are consistent after checkpoint recovery.
4.5 Adaptation Checklist
The table below can serve as a starting point for migration acceptance. The pass criteria for each item should be set by the team according to the task; the table gives the inspection direction.
| Item | Check content | Pass criteria |
|---|---|---|
| Operators | Single-operator comparison between core operators and reference implementation | Relative error within the agreed threshold |
| Accuracy | Loss and gradients under mixed precision | Loss trend consistent with reference implementation, no NaN |
| Memory | Peak training memory | Below device capacity with the agreed margin |
| Communication | Proportion of per-step communication time | Below the agreed upper limit, and throughput grows with the number of devices |
| Stability | Checkpoint recovery | Loss in the first several steps after recovery matches uninterrupted training |
| Deployment | Inference latency and concurrency | P99 latency and concurrency meet business requirements |
5. Evaluation Methods
Subjective judgment is easily affected by prompt selection and randomness. Evaluation should use a fixed prompt set and unified sampling parameters.
5.1 Fixed Prompt Set
It is recommended to cover four categories of prompts, with at least five prompts in each category:
- Simple objects: for example, “an orange cat sitting on a windowsill”;
- Complex combinations: for example, “three red balloons on the left, an elderly man in a blue coat walking a dog on the right, with a seaside background at dusk”;
- Chinese infographics: for example, “an infographic introducing the four seasons, titled ‘Four Seasons,’ with an icon and Chinese description for each season”;
- Style transfer: the same subject in realistic, illustration, and concept-design styles.
5.2 Sampling Parameters
- Fix resolution, sampling steps, and scheduler;
- Set a fixed random seed list for each prompt, for example five seeds;
- Generate one image per seed and record all generation parameters;
- Compare the same model version with different model versions using exactly the same settings.
Because generation results are random, a single output cannot represent model capability. When comparing two versions, compare the average score and dispersion across multiple samples.
5.3 Scoring Template
| Dimension | Full score | Scoring standard |
|---|---|---|
| Semantic alignment | 5 | Main subjects in the image match the prompt |
| Object count and attributes | 5 | Count, color, and size match the description |
| Chinese text accuracy | 5 | Text is readable, without typos, and labels correspond to objects |
| Spatial relationships | 5 | Positions and front-back occlusion relationships are correct |
| Image quality | 5 | No obvious distortion, merging, or noise |
| Style consistency | 5 | Style meets requirements while subject content remains unchanged |
| Sampling stability | 5 | Differences across multiple samples for the same prompt are acceptable |
Scoring should ideally be completed independently by two reviewers, and disagreements between them should be recorded. Samples with large disagreements should be adjudicated by a third person. Automatic metrics, such as text recognition accuracy, can supplement manual scoring.
6. Seven-Step Engineering Rollout Process
The table below defines the input, output, and acceptance metric for each step. If any step fails acceptance, return to that step to fix it rather than moving directly to the next step.
| Step | Input | Output | Acceptance metric |
|---|---|---|---|
| 1. Data processing | Raw image-text pairs | Deduplicated, filtered dataset | Duplicate rate, retention ratio, sampled accuracy of image-text matching |
