Text-to-Image Model Training and Deployment on a Domestic AI Compute Stack

    This article examines the engineering practices for training and deploying text-to-image models on a domestic AI compute stack, with data processing, training, and deployment kept within the same software and hardware environment. It explains why cross-platform migration can introduce precision, performance, and stability issues, reviews the core text-to-image architecture, and details data processing, operator adaptation, distributed training, and inference optimization. It also presents evaluation methods for reproducible results and a troubleshooting table for common problems, helping teams maintain consistent numerical behavior and diagnose issues more directly.

    1. Why Focus on the Training Pipeline on Domestic AI Compute

    Training a high-quality image generation model requires data, algorithms, compute, and engineering systems to work together. Many teams are used to completing training on one hardware ecosystem and then moving the model to other chips for inference. The problem with this approach is that the operator libraries, communication libraries, compilers, and scheduling systems used during training differ from those used during inference. After migration, the model can exhibit precision deviations, performance degradation, and stability issues.

    If data processing, training, and deployment all run on the same software and hardware stack, fewer cross-platform adaptation steps are needed. R&D teams can maintain consistent numerical behavior across phases and diagnose issues more directly. This article follows that principle and outlines the key engineering considerations for a text-to-image model from data to deployment.

    2. The Basic Paradigm of Text-to-Image Models

    Text-to-image models generate images that correspond to textual descriptions. Most current approaches center on generative modeling. The model learns to gradually recover an image from noise or a latent representation, while text encodings serve as conditional inputs that guide the generation process.

    The overall structure can usually be divided into three modules:

    • Text encoding module: maps prompts to semantic vectors. Both Chinese and English prompts need to be handled in a stable way.
    • Generation backbone: performs iterative denoising or continuous transformation in latent space and accounts for most of the computation.
    • Image decoding module: reconstructs a pixel image from the latent representation.

    Multimodal alignment is critical. If the mapping between text semantics and visual representations is unstable, training can suffer from slow convergence, poor prompt adherence, and disordered local details. Therefore, the quality of image-text pair data directly determines the upper bound of model quality.

    3. Four Engineering Phases in the Full Training Workflow

    3.1 Data Processing

    Data is the foundation of image generation model quality. Common processing steps include:

    1. Deduplication: remove duplicate images based on perceptual hashing or feature similarity to prevent the model from over-memorizing individual samples.
    2. Filtering: remove samples with low resolution, severe watermarks, or prohibited content.
    3. Recaptioning: rewrite original captions with more accurate and complete text to improve the correspondence between prompts and image content.
    4. Bucketing: group images by aspect ratio to reduce compositional information loss caused by uniform cropping.

    The sample code below shows the processing order for data filtering. Thresholds and hash functions are placeholders:

    1. MIN_SIDE = 512
    2. seen_hashes = set()
    3. def build_training_set(samples):
    4. cleaned = []
    5. for s in samples:
    6. # Filter out samples with insufficient resolution
    7. if s.width < MIN_SIDE or s.height < MIN_SIDE:
    8. continue
    9. # Filter out missing or blank captions
    10. if s.caption is None or not s.caption.strip():
    11. continue
    12. # Remove duplicate images with the same perceptual hash
    13. if s.phash in seen_hashes:
    14. continue
    15. seen_hashes.add(s.phash)
    16. cleaned.append(s)
    17. return bucket_by_aspect_ratio(cleaned)

    Note that both Chinese and English captions should be included in the training set. When only one language is used for captions, model performance in the other language often becomes noticeably weaker.

    3.2 Model Architecture and Operator Adaptation

    Model architecture sets the upper bound of hardware efficiency. The memory access pattern of attention computation, the implementation of normalization layers, and the choice of mixed precision all significantly affect training throughput.

    Operator adaptation usually includes three tasks:

    • Operator fusion: combine multiple small operators into one kernel to reduce scheduling overhead and memory reads and writes for intermediate results.
    • Precision strategy: use low-precision formats such as BF16 or FP16 for main computation, while retaining higher precision for sensitive steps such as accumulation and normalization.
    • Implementation replacement: replace general implementations with more efficient implementations on the target hardware. Results must be verified after replacement.

    Each time an operator is replaced, compare it with the reference implementation under identical inputs and check the maximum absolute error and cosine similarity. The following is a simplified validation function:

    1. def check_operator(ref_op, new_op, inputs, atol=1e-3):
    2. ref_out = ref_op(*inputs)
    3. new_out = new_op(*inputs)
    4. max_err = max_abs_diff(ref_out, new_out)
    5. assert max_err < atol, f"Error exceeds threshold: {max_err}"
    6. return max_err

    Only operators that pass numerical consistency tests should enter the main training workflow. Otherwise, anomalies that appear late in training are difficult to trace back to a specific step.

    3.3 Distributed Training

    Large-scale training relies on a combination of parallel strategies:

    • Data parallelism: each device holds a full copy of the model and processes different data batches.
    • Tensor parallelism: split the weight matrices of a single layer across multiple devices.
    • Pipeline parallelism: split the model by layers into stages, with different stages executed in a pipeline on different devices.
    • Parameter sharding: distribute optimizer states and gradients across devices to reduce per-device memory usage.
    • Activation recomputation: do not store all intermediate activations; recompute them during backpropagation, trading time for memory.

    The choice of parallel strategy depends on model size, per-device memory, and interconnect bandwidth between devices. Communication overhead should overlap with computation where possible; otherwise, multi-device scaling efficiency drops rapidly.

    The example configuration below uses values only to illustrate the structure. The values must be reset according to cluster size and model size:

    1. parallel:
    2. data_parallel_size: 64
    3. tensor_parallel_size: 4
    4. pipeline_parallel_size: 2
    5. optimizer:
    6. shard_states: true
    7. precision: bf16
    8. activation_checkpointing: true
    9. checkpoint:
    10. interval_steps: 1000
    11. keep_last: 5

    Checkpoint strategy is also important. Training cycles are long, and hardware failures are almost inevitable. Checkpoints should be saved periodically, and a load validation should be performed after saving to ensure training can resume from any saved checkpoint.

    3.4 Inference Deployment

    Inference focuses on balancing latency, throughput, and cost. Common optimization methods include:

    • Reducing sampling steps: use methods such as step distillation to maintain acceptable generation quality with fewer iterations.
    • Caching text encodings: reuse encoded results for identical or similar prompts.
    • Dynamic batching: combine requests arriving within a short window to improve accelerator utilization.
    • Quantization: lower weight precision to reduce memory usage, but this may harm details and color; it must be evaluated for quality.

    Deployment also requires setting request queue length and timeout policies. An overly long queue can sharply increase tail latency, while too short a timeout can cause a large amount of wasted computation.

    4. Evaluation: From Demo Samples to Reproducible Conclusions

    A few demo images cannot prove overall model quality. Reliable evaluation requires a fixed test set, fixed random seeds, and fixed sampling parameters. The test set should cover the following categories:

    Category Evaluation focus
    Chinese prompts Accuracy of semantic understanding
    English prompts Consistency with the Chinese results
    Text rendering Clarity and readability of text in the image
    Multi-subject composition Whether subject count and position match the description
    Realistic and illustration styles Stability of style control
    Scientific diagrams Correctness of structure and annotations

    Evaluation metrics can be divided into two types. Automatic metrics include distribution-distance metrics and image-text consistency scores. They are suitable for quickly comparing versions on large samples. Human scoring focuses on instruction following, image quality, and structural correctness, and is suitable for identifying issues that automatic metrics cannot reflect. The two types of results should be considered together, with sample size and variability range reported.

    5. Common Issues and Troubleshooting

    Symptom Possible cause Troubleshooting
    NaN in training loss Learning rate too high, or underflow under low precision Monitor gradient norm, enable gradient clipping, and compare with high-precision results
    Out-of-memory Batch size too large, or activation recomputation disabled Reduce per-device batch size and enable activation recomputation
    Low multi-device speedup Communication overhead too high Profile communication time and adjust parallel partitioning
    Garbled text in generated images Insufficient high-quality Chinese image-text pairs Add Chinese caption data and evaluate text rendering separately
    Large result differences for the same prompt across environments Inconsistent operator implementation or random seeds Align operator implementations and random seed settings