Training Multimodal Image Generation Models on Domestic AI Accelerators: A Full-Pipeline Practice Guide
This guide walks through the complete workflow for training multimodal image generation models on domestically produced AI accelerators (home-grown AI accelerator cards). It covers pre-training prerequisites, image-text data cleaning thresholds, glyph preservation for Chinese text, sharding and prefetch parameters, per-card throughput acceptance metrics, staged training (pre-training, SFT, and preference optimization), hardware-software co-optimization such as compute-communication overlap, multi-level dispatch, and fused operators, the autoregressive-plus-diffusion-decoder hybrid architecture, Chinese text rendering injection and acceptance, arbitrary-resolution bucketing and extrapolation, and practical advice for teams moving to production.
1. Can Domestic AI Accelerators Reliably Train Multimodal Image Generation Models?
Image generation models are moving from “able to draw” to “drawing accurately and writing text correctly.” During this transition, whether domestically produced AI accelerator platforms can complete end-to-end training of multimodal image generation models is not only a question of hardware availability. It also depends on whether data, training, communication, operator libraries, evaluation, and engineering support form a closed loop.
Putting the training pipeline on domestic AI accelerators delivers value in three main areas. The first is supply chain resilience: it reduces dependence on a single hardware ecosystem at critical training stages. The second is hardware-software co-design: frameworks, operator libraries, communication libraries, and schedulers can be tuned around a specific chip at a finer granularity. The third is reproducibility: once a standardized training kit exists, it can be ported to more models and business scenarios.
However, adaptation cannot stop at “it runs.” Several questions still need answers: How should domestic accelerator cards be selected? How do frameworks and communication libraries need to be matched? Can the data pipeline keep up? Which errors show up most often in Chinese text rendering? How do you evaluate arbitrary-resolution output? The sections below address these practical problems.
2. Prerequisites and Key Engineering Points Across the Pipeline
Before starting training, define the prerequisites first: accelerator card model, driver and firmware versions, deep learning framework version, communication library version, mixed-precision format, operator library compatibility, container image baseline, and the distributed launch method. When prerequisites are inconsistent, later performance problems tend to surface as “random slowness” or numerical anomalies that are hard to reproduce.
1. Data Preprocessing: Actionable Parameters and Acceptance Criteria
Image-text pair datasets are usually large. The core of preprocessing is not cleaning alone, but establishing quantifiable acceptance criteria.
Image-text pair cleaning thresholds
Start with a coarse filter using an image-text consistency model or simple rules. As an example convention: samples with an image-text consistency score below 0.25 can be discarded directly; samples between 0.25 and 0.35 go into secondary review or downsampling; samples above 0.35 enter the main training pool. These thresholds are not a fixed standard and need to be recalibrated for the specific model and task. Basic quality filters should also be applied, such as excessively low resolution, large watermarks, severe compression artifacts, and obviously mismatched images.
Glyph preservation strategy
If the target is Chinese posters, e-commerce images, or text-heavy interfaces, text cannot be treated as ordinary tokens. It is advisable to preserve the original Unicode, font name, font size, position, color, and character-level alignment relationships, and to keep glyph reference images or font rendering samples in the training data. For long text, perform character-level segmentation while preserving context mapping, so the model is not left with semantic tokens only and stripped of glyph constraints.
Sharding and prefetch parameters
Data loading should use sharding, prefetching, and multi-process asynchronous decoding. The following can serve as starting values: keep each data shard between 1,000 and 5,000 samples, set the number of prefetched batches to 2–4, set DataLoader workers to 50%–75% of available CPU cores, and cap the read buffer at 4–8 batches to avoid excessive memory consumption. If image decoding and resizing consume CPU time, lightweight operations such as Resize and normalization can be moved to the device or cached in advance.
Per-card throughput metrics
Acceptance cannot be limited to “does it train.” Throughput stability matters too. Recommended metrics: samples per second per card, tokens per second, per-step time, data wait ratio, GPU utilization, peak VRAM usage, and the share of time spent in communication. A practical target is keeping the data wait ratio below 10%. If it stays above 20% over a long period, investigate I/O, decoding, uneven sharding, or an insufficient preprocessing queue first.
Common bottleneck troubleshooting
If GPU utilization is low but the loss behaves normally, first determine whether the constraint is data supply. Fix the batch size and disable random augmentation, then measure data loading time. Next, check whether the CPU is saturated, whether disk I/O is at capacity, and whether image decoding has become the bottleneck. If CPU decoding pressure is too high, pre-generate low-resolution caches or move decoding logic elsewhere. If communication waiting is clearly significant, check gradient bucket size and overlap strategy.
2. Staged Training: Pre-training, SFT, and Reinforcement Learning Alignment
Multimodal image generation models are better trained in stages rather than mixing all objectives into one run.
Pre-training stage
The goal is to build general joint semantic and visual representations on large-scale image-text pairs. Data volume is typically large, and input resolution can start at 256 or 512. Acceptance metrics may include image-text retrieval accuracy, CLIP similarity, stability of the training loss curve, and small-scale human review of generated visuals.
Supervised fine-tuning stage
The goal is to align the model to specific tasks, such as poster layout, rendering specified text, product image generation, or style control. Data requirements are higher: carefully annotated samples, unified layout specifications, and high-quality font resources are needed. Acceptance metrics should include human preference win rate, text OCR accuracy, layout integrity, and human review pass rate.
Reinforcement learning or preference optimization stage
The goal is to improve usability and visual quality by leveraging human preferences, reward models, or rule-based rewards. Inputs typically include preference data, failure samples, and reward signals. During acceptance, observe win rate, the proportion of degraded samples, and safety and compliance results together, so that short-term preference gains do not come at the cost of stability.
Staged training keeps objectives clear and problems localizable, and it also makes resource scheduling and stage-level rollback easier on domestic accelerators.
3. Selecting and Saturating Domestic Cards: Hardware-Software Co-Optimization
After moving training to domestic chips, performance work is usually not about a single algorithmic trick, but about making compute, communication, scheduling, and numerical stability work together.
1. Prerequisites and Monitoring Metrics
Before optimizing, lock down the version combination: framework version, driver version, communication library version, operator library version, and container image must all be reproducible. For mixed-precision training, decide explicitly between FP16 and BF16, and configure dynamic loss scaling. On domestic cards, also verify that key operators actually take the optimized path, for example LayerNorm, RMSNorm, attention, gradient reduction, and the AdamW update.
Recommended continuous monitoring: MFU, GPU utilization, VRAM usage, per-step time, share of communication time, data wait time, loss scale changes, gradient norm, and the count of NaN/Inf values. If MFU stays low over a long period, break it down further: is compute underutilized, is communication a large share, or is scheduling overhead excessive?
2. Overlapping Computation and Communication
In multi-card training, if gradient synchronization and backward computation run strictly serially, effective compute throughput drops. Multi-stream parallelism, gradient buckets, and asynchronous communication let communication overlap with computation on the timeline. Whether the gain is significant depends on parallel scale, communication topology, batch size, gradient bucket strategy, and the level of framework support. If the batch is too small, communication latency is high, or gradient buckets are configured poorly, the benefit may be limited or may even add synchronization overhead.
A safer approach is to validate in small steps: fix the model and batch size, change only the gradient bucket size or the communication overlap switch, and observe changes in communication time share and per-step time. Do not judge from a single test run; observe the stable range over at least several dozen steps.
3. Eliminating Dispatch Bottlenecks
In eager mode, dispatching tasks operator by operator from the host accumulates scheduling overhead. Multi-level pipelined dispatch decouples task submission from execution, letting the CPU prepare subsequent operators in advance and reducing device waiting. This strategy shows more value when there are many operators, the graph structure is complex, or the device executes quickly. If the device is already saturated, increasing dispatch concurrency further may yield no clear gain.
4. Fused Operators and Numerical Stability
To match the operator affinity of domestic chips, several small operators can be fused into larger ones, reducing memory access and scheduling overhead. Common directions include fused optimizer state updates, EMA weight maintenance, and RMSNorm normalization fusion. After fusion, speed alone is not enough — numerical verification is required: compare per-step loss, gradient norm, and weight update differences before and after fusion. If key metrics deviate beyond a preset threshold, roll back to the stable version.
The pseudocode below only illustrates how gradient accumulation, distributed synchronization, optimizer updates, and communication overlap are organized. It is not complete, production-ready engineering code. A real implementation still needs distributed launch, rank/world_size setup, device streams, event synchronization, mixed-precision loss scaling, and framework-specific APIs.
# Illustrates only how optimization steps are organized in the training loop; not runnable as production code.# Real deployment needs distributed launch, rank/world_size, gradient accumulation,# mixed precision, and a communication library.for global_step, batch in enumerate(dataloader):with autocast():for micro_step in range(grad_accum_steps):loss = model(batch) / grad_accum_stepsloss.backward()if (global_step + 1) % grad_accum_steps == 0:# If the framework supports it, gradients can be synchronized asynchronously# by gradient bucket during the backward pass.# The sync method may be all_reduce or reduce_scatter, depending on the parallel strategy.sync_gradients()optimizer.step()ema.update()log_metrics(global_step,loss=loss,grad_norm=grad_norm,comm_time=comm_time,step_time=step_time)
4. The “Autoregressive + Diffusion Decoder” Hybrid Architecture: Capability Boundaries and Cost
Models of this type usually split work across two modules. The autoregressive backbone handles semantic understanding and global composition; following the modeling approach of large language models, it generates sequences of image semantic tokens. The diffusion decoder handles pixel-level refinement, progressively decoding high-level semantic tokens into a clear image.
The advantage of this architecture is a clear division of labor: the backbone decides what the image contains and how it is laid out, while the decoder renders it. It also has boundaries. Semantic token length affects VRAM usage and inference latency; the number of diffusion steps determines the balance between quality and time; decoder resolution determines final output detail and compute cost. If the target is large posters or high-resolution long images, increasing resolution alone without adjusting semantic token length and positional encoding can lead to loose composition or blurred details.
Architecture selection should therefore consider more than the quality ceiling. Training cost, inference latency, VRAM usage, and the cost of repairing failure samples all need evaluation.
5. Chinese Text Rendering: Common Errors, Injection Approaches, and Acceptance
Common problems when rendering Chinese text in generated images include missing strokes, wrong characters, deformation, characters merging together, misaligned layout, and broken Chinese-English mixed typesetting. The root cause is usually not that the model “does not recognize text,” but that text is processed as ordinary semantic tokens without explicit visual glyph constraints.
Glyph injection layer
The mainstream approach is glyph encoding or glyph-conditioned injection. The glyph image of the target text, font vectors, and character position information can be injected as additional conditions into the cross-attention, adapter, or conditional embedding layers of the diffusion decoder. The earlier and stronger the injection, the higher the text accuracy typically is, but it also increases training and inference cost. Real projects need to choose per scenario: commercial posters should prioritize text accuracy, while creative illustrations can relax the constraints moderately.
Glyph training data pipeline
Training data needs to include text annotations, font information, glyph renderings, character-level bounding boxes, and OCR annotations. It is advisable to maintain a font whitelist, a licensed font library, and a glyph caching service. For long text, preserve character order and line-break information; for Chinese-English mixed content, record the different character sets and font strategies separately.
Acceptance metrics
Chinese rendering cannot rely on visual inspection alone; quantifiable acceptance criteria are needed. Useful measures include OCR character accuracy, character recall, missing-stroke rate, wrong-character rate, layout offset rate, and human blind-test win rate. Before launch, stress tests should also cover long text, small font sizes, complex backgrounds, and dense text scenarios.
6. Arbitrary-Resolution Generation: Bucketing, Extrapolation, and Failure Modes
Traditional models often support only fixed aspect ratios, such as 1:1, 4:3, and 16:9. In real business scenarios, however, vertical long images, e-commerce banners, social media covers, and specific placement dimensions appear frequently, so arbitrary-resolution generation has clear value.
Resolution bucketing
In engineering practice, bucketing can be applied first, for example covering 1:1, 4:3, 16:9, 3:4, 9:16, and long-image ratios close to 1:2.5. More buckets is not automatically better; buckets should be designed according to the distribution of business sizes. Each bucket needs separate statistics for composition quality, text clarity, and edge integrity.
Positional encoding extrapolation
Arbitrary resolution also depends on the tokenizer and positional encoding strategy. If the training and inference resolutions differ greatly, insufficient positional encoding extrapolation leads to repeated textures, stretched composition, or blurry edges. Record the range of training sizes and run a size validity check before inference. For extreme long images, set an upper limit so that inputs do not fall outside the training distribution.
Train and inference resolution differences
Using a smaller resolution during training lowers cost, but simply scaling up at inference time can produce insufficient detail. The gap between low-resolution training and high-resolution inference should be evaluated and closed through fine-tuning or resolution-ladder training.
Failure modes and evaluation
Common failures include cropped subjects, deformed text, repetition in long images, broken background textures, and imbalanced composition. Evaluation can break down CLIP similarity, OCR accuracy, human preference win rate, and edge integrity by resolution bucket. If one bucket has a noticeably higher failure rate than others, investigate data bucketing or positional encoding strategy.
7. Practical Recommendations for Implementation Teams
For teams planning to build multimodal image generation capabilities on domestic AI accelerator platforms, the recommended path is “get it running, then evaluate, then scale.”
First, establish a small-sample end-to-end loop early. Use a small amount of high-quality data to validate the full chain of data loading, training, checkpointing, evaluation, and inference, rather than starting large-scale training immediately.
Second, build a minimal observability system. Record at least loss, gradient norm, learning rate, VRAM usage, throughput, communication time, data wait time, and anomalous samples. Key metrics should feed monitoring and alerting, with support for failure rollback.
Third, put data compliance first. Training data needs clear provenance, licensing, and copyright records. When content involves faces, portraits, trademarks, or private information, apply de-identification, authorization verification, and content safety filtering. Generated content also needs safety review and risk filtering mechanisms.
Fourth, establish a cost model. Do not count only accelerator-hours; also account for storage, network, failed retries, data annotation, human evaluation, and tuning costs. For long-running training, set budget alerts and experiment quotas to avoid wasting resources on unproductive trial and error.
Fifth, make reasonable use of general cloud infrastructure. Training orchestration can rely on container platforms for elastic scheduling, samples and model assets can be stored in object storage, and training metrics can be continuously observed through logging, monitoring, and alerting services. This improves resource utilization and makes unified governance easier in domestic accelerator environments.
8. Summary
Completing end-to-end training of multimodal image generation models on domestic AI accelerators depends not on any single isolated optimization, but on forming a complete closed loop across data, training, inference, evaluation, and compliance governance. Whether domestic cards can train stably depends on the combined effect of framework adaptation, data throughput, communication overlap, operator fusion, and numerical stability; the hybrid architecture resolves the division of labor between semantic understanding and pixel generation; glyph encoding improves Chinese rendering capability; and arbitrary-resolution support addresses real business size requirements. For implementation teams, building an engineering system that is reproducible, observable, and rollback-capable matters more than chasing a single-point metric.
