Image Generation Models on Domestic AI Accelerators: A Full-Pipeline Training and Deployment Checklist
This article is an engineering checklist for training and deploying image generation models on domestic AI accelerators. Moving beyond generation quality, it covers the full lifecycle—data preparation, training, inference, monitoring, and rollback—and explains how to diagnose operator incompatibilities, mixed-precision and numerical instability, distributed communication bottlenecks, and memory constraints. It includes a troubleshooting table, a data checklist, a minimal reproducible training configuration, guidance on quantization/pruning/distillation for inference, precision-alignment criteria, acceptance metrics, and a logging and rollback checklist, so that every stage becomes checkable, reproducible, and rollback-ready.
1. From Generation Quality to Full-Pipeline Trainability
Training image generation models on domestic AI accelerators takes more than checking whether generated images are crisp or whether prompts are followed. You also need to confirm that data, training, inference, monitoring, and rollback form a closed loop. Once a model is migrated to domestic chips, the common risks are no longer single quality issues but rather missing operators, numerical instability, inefficient distributed communication, insufficient memory, and inference-time precision drift. This article uses an engineering checklist to break down how to train image generation models on domestic compute, how to troubleshoot operator incompatibilities, how to roll back interrupted training, and which acceptance metrics you need before going live.
2. Full-Pipeline Challenges and a Troubleshooting Table
When migrating to domestic compute, start by locating problems with the table below. Rather than concluding that “the model quality is poor,” classify failures into four categories: operators, precision, communication, and memory.
| Challenge | Typical Symptoms | Diagnostic Metrics | Alternatives |
|---|---|---|---|
| Operator and framework compatibility | Forward or backward pass errors indicating an unsupported operator; custom modules fail to run; computational graph construction fails | Operator error stack, operator coverage list, unit test pass rate, framework compatibility logs | Equivalent operator substitution, fused operators, fall back to an unoptimized implementation, write an adaptation plugin |
| Mixed precision and numerical stability | NaN or Inf appears; training diverges after some steps; loss curves differ markedly across chips | loss curve, grad_norm, gradient clipping threshold, fp16/bf16 comparison experiments | Prefer bf16; keep fp32 in critical layers; tune loss scaling; lower the learning rate; strengthen gradient clipping |
| Distributed communication efficiency | Throughput does not improve noticeably when adding cards; step time increases; communication wait ratio is too high | communication time ratio, All-Reduce throughput, scaling efficiency, per-step communication time | Overlap compute and communication, gradient accumulation, optimize the parallel strategy, adjust tensor or pipeline parallel partitioning |
| Memory and memory management | OOM mid-training; batch size cannot be increased; activation memory usage is too high | peak memory, activation usage, optimizer state usage, data loading wait time | Gradient checkpointing, activation recomputation, ZeRO or tensor partitioning, lower batch size, add gradient accumulation |
Common issues can first be localized into five types: unsupported operator, OOM, NaN, communication timeout, and non-scaling throughput. Suggested troubleshooting order: first reproduce a minimal single-card pipeline, then scale to multiple cards; first inspect logs and monitoring, then tune hyperparameters; first ensure stable convergence, then optimize throughput.
3. Data Preparation Checklist
Image generation models are highly sensitive to data quality. On domestic compute, data preparation is not just cleaning images and text—it also means matching the data loading approach to chip memory access, the framework’s read path, and the pace of distributed training.
A suggested directory structure and sample format:
data/├── images/├── train.jsonl├── val.jsonl└── edge_cases.jsonl
Example train.jsonl line:
{"image": "images/000001.jpg", "text": "A Shiba Inu running in the snow", "height": 256, "width": 256, "aspect_ratio": "1:1", "source": "internal", "hash": "phash_8f3a"}
Data checklist:
| Check Item | Suggested Target | Verification Method |
|---|---|---|
| Loading throughput | Data loading should not be the bottleneck; data_wait stays clearly below step time | Track samples/s, data_wait, step_time |
| Prefetch depth | Buffer 2 to 4 batches | Observe loading stalls; adjust the number of prefetch threads |
| Deduplication | Remove identical or highly similar image-text pairs | Hashing, perceptual hashing, vector similarity; clean if the duplication rate is too high |
| Text encoder version | Pin encoder version, weight hash, tokenizer config | Save a config snapshot to avoid train/inference mismatch |
| Image-text alignment | Non-empty text, no mismatches, no abnormal truncation | Manual sampling checks; keep failed samples |
| Resolution and aspect ratio | Uniform target resolution or safe scaling | Track resolution distribution and abnormal-size ratio |
| Data security and compliance | Avoid unauthorized data, sensitive content, and private information | Establish intake, de-identification, and audit processes |
On domestic compute platforms, the text encoder should also run within the same software and hardware stack as training. If the image branch trains on domestic chips while text encoding depends on an external service, it can introduce version mismatches, feature-distribution mismatches, and an unstable pipeline.
4. Getting Training to Run: A Minimal Reproduction Configuration
To verify that domestic compute can train an image generation model, start by building a minimal reproducible case. The goal is not to generate high-quality images but to validate whether the data pipeline, operator pipeline, distributed communication, and logging system are complete.
| Item | Minimal Reproduction Setting |
|---|---|
| Hardware | 4 domestic GPGPU or NPU cards; at least 32GB memory per card is recommended; use the actual procured hardware model |
| Framework | A domestic deep learning framework such as MindSpore or PaddlePaddle, or a vendor-customized PyTorch distribution; pin the version |
| Model | A lightweight diffusion or DiT-style model, around 0.2B to 0.5B parameters |
| Resolution | 256×256, extendable to 512×512 later |
| Data scale | About 1000 image-text pairs, including train, validation, and edge-case sets |
| batch size | 8 per card (global batch 32); on 4 cards at 4 per card the global batch is 16, so gradient accumulation of 2 is needed to match a global batch of 32 |
| Learning rate | 1e-4 to 5e-5, with warmup |
| Optimizer | AdamW |
| Mixed precision | Prefer bf16; if unsupported, use fp16 with loss scaling configured |
| Memory optimization | Gradient checkpointing, activation recomputation, optimizer state sharding, ZeRO or tensor parallelism |
| Communication backend | Vendor collective communication library supporting All-Reduce and All-Gather |
| Stability configuration | Gradient clipping, loss monitoring, abnormal exit, periodic checkpoint saving |
The minimal reproduction data should follow the data/train.jsonl format, with each line containing fields such as image, text, height, width, aspect_ratio, and hash; the directory structure can follow data/images, data/train.jsonl, and data/val.jsonl. If resolution or aspect ratio is inconsistent, apply safe scaling in advance and remove abnormal samples.
Before minimal reproduction, first validate the environment and data:
Dependency version detection:
python -c "import torch; print('torch', torch.__version__)"python -c "import mindspore; print('mindspore', mindspore.__version__)"python -c "import paddle; print('paddle', paddle.__version__)"pip freeze > requirements.lock.txt
Hardware and device detection:
npu-smi info
or the vendor’s device query command. If using a custom PyTorch, also confirm that
torch.npu.is_available()returns true or that the vendor plugin loaded successfully.Data validation:
python tools/check_data.py --data data/train.jsonl --resolution 256 --max-samples 1000
Config entry point:
configs/minimal_repro.yaml
Example launch command:
python tools/train.py --config configs/minimal_repro.yaml --device npu:0-3
For distributed launch with MindSpore, switch to the
msrunentry point; for PaddlePaddle, usepaddle.distributed.launch. The key is to pin the launch command, config file, and log directory.
Recommended progression:
- Run forward and backward passes on a single card, confirming no operators are missing.
- Scale to 2 cards and observe gradient synchronization.
- Scale to 4 cards and check scaling efficiency and communication time.
- Train for 1000 to 2000 steps and observe whether the loss decreases smoothly.
- Save a checkpoint and verify resume-from-checkpoint.
- Record failure cases, including error stacks, operator fallbacks, and numerical anomalies.
If a critical operator is missing, first look for an equivalent or fused operator; if no ready implementation exists, temporarily fall back to a slower but stable implementation; if precision drops markedly, revert to higher-precision computation or shrink the model and re-validate. Write every alternative path into the config and logs to avoid non-reproducibility later.
5. Stabilizing Inference Deployment: Risks and Metrics of Quantization, Pruning, and Distillation
After training, the model must be turned into a usable image generation service. Inference optimization can carry side effects, so you must establish quality spot-checks and rollback mechanisms.
Quantization reduces memory and inference cost but can cause detail loss, color deviation, or reduced semantic fidelity. Fix a baseline prompt set and compare similarity before and after quantization, focusing on text, edges, textures, and human faces. You can set a manual blind-evaluation pass rate that drops no more than 5% relative to baseline, and automatic similarity that drops no more than 3%.
Pruning reduces computation but can weaken style diversity. Record the change in parameter count, compare visual quality in typical scenarios, and roll out high-risk scenarios gradually (canary release) first.
Distillation improves inference speed, but the student model may show style drift or reduced prompt-following ability. Build a comparison set of teacher and student models to check semantic consistency, image quality, and style stability. If key metrics degrade beyond the agreed threshold, keep a path to fall back to the teacher model.
Latency measurement must define its scope clearly: input resolution, batch size, whether text encoding, sampling, and post-processing are included, whether warmup is done, whether to report P50/P95/mean latency, and whether image storage or object-storage upload time is included. Report at least two scopes: core inference latency excluding text encoding and storage, and end-to-end latency including text encoding and storage.
Serving also requires attention to data security and rollback: apply content-safety filtering to user input; run sensitive-content detection on generated results; establish canary release; keep a version rollback path; version-manage container images, model files, and configs; and stay alert to precision changes after quantizing models and migrating containers.
6. Precision Alignment and Acceptance Metrics
If a reference implementation exists, build a precision-alignment baseline. You can compare:
- Per-layer output relative error;
- loss differences;
- grad_norm differences;
- logits differences on the same input;
- Average pixel difference or vector similarity of generated results.
Suggested initial criteria: with the reference implementation as the baseline, keep the median per-layer relative error within 1e-2; keep the relative loss difference within 2%; keep the relative grad_norm difference within 5%; keep the cosine similarity of generated results no lower than 0.98, with an average pixel L1 difference no higher than 0.05. If a threshold is exceeded, trigger an alert and first check the random seed, data order, operator version, mixed-precision mode, and communication reduction method.
Acceptance metrics should look at samples/s, images/s, or steps/s, with test conditions noted. Example scope: resolution 256×256, global batch 32, bf16, including text encoding. Suggested acceptance items:
| Acceptance Item | Suggested Scope |
|---|---|
| Training success rate | Full jobs start and complete stably |
| Convergence steps | Steps needed to reach the target loss or validation metric |
| Throughput | samples/s or images/s, noting resolution, batch size, precision mode, and whether text encoding is included |
| Communication efficiency | 4-card throughput near the 4-card theoretical value; scaling efficiency no lower than 70% |
| Inference latency | P50 and P95 within a business-acceptable range |
| Precision deviation | Explainable versus the reference implementation or the previous version |
| Ops alerting | Alerts on abnormal loss, memory overflow, and communication timeouts |
| Rollback capability | Fast recovery to a stable version after failure |
7. Logging, Rollback, and the Launch Checklist
Every experiment should retain a config snapshot, data version, and model version. Logs should record at least: step, loss, learning_rate, grad_norm, throughput, step_time, data_wait, comm_time, peak_memory, precision_mode, checkpoint_id, fallback_operator, warning_message, and job_status.
The failure rollback path must be actionable. Before rolling back, confirm the data snapshot, config snapshot, model version, and rollback permissions to avoid deleting data or overwriting the latest checkpoint. Create “stable, experimental, failed” directories for checkpoints, and set permissions or a second confirmation for overwrite operations. Rollback is not a simple restart—it must be able to answer: which config version, which batch of data, and at which step the anomaly began.
An end-to-end minimal example can include:
- Request input: prompt, random seed, target resolution.
- Text encoding: generate the semantic vector and record the encoder version.
- Image generation: run inference with the diffusion sampler or autoregressive generator.
- Result output: return the image URL or object-storage link.
- Log record: request_id, prompt_hash, encoder_version, model_version, encode_time, sample_time, total_time, resolution, status, fallback_reason.
Completing the full-pipeline training of an image generation model on domestic compute is not just about getting the model to run—it is about bringing data, training, inference, monitoring, and rollback into the same engineering system. As long as every step becomes a checkable, reproducible, and rollback-ready action, a domestic AI software and hardware stack can move from “migratable” toward “self-buildable.”
