End-to-End Training of Open-Source Image Generation Models on Domestic AI Accelerators
This article examines the engineering practices behind training an open-source image generation model entirely on a cluster of domestically developed AI accelerators. It covers the hybrid autoregressive–diffusion architecture, distributed data and training pipelines, non-standard aspect ratios, Chinese text rendering, evaluation, fault recovery, and deployment considerations. The focus is not on any specific vendor, but on the general engineering challenges involved in making the full training workflow reproducible and production-ready.
1. Why End-to-End Training on Domestic AI Hardware Is Difficult
A complete image generation training workflow involves far more than replacing individual operators. It typically includes data decoding and augmentation, text cleaning, image tokenization, distributed pretraining, supervised fine-tuning, preference optimization, checkpointing, fault recovery, evaluation, and inference deployment. A throughput bottleneck or numerical instability at any stage can affect the entire training job.
Image generation models also present three major challenges.
First, multimodal datasets are large. Image decoding, resizing, aspect-ratio bucketing, and caching place significant demands on CPU capacity and storage bandwidth.
Second, diffusion training is sensitive to noise scheduling, gradient stability, and mixed-precision strategies.
Third, Chinese text rendering requires the model not only to produce visually convincing images, but also to generate accurate characters. This involves glyph structure, long-text completeness, and layout positioning. Traditional models often rely on preset ratios such as 4:3 or 16:9. When generating posters, long-form images, or social media covers, they may crop or stretch the content.
As a result, full-workflow training on domestic AI accelerators, support for non-standard resolutions, and reliable Chinese text rendering form a comprehensive engineering problem rather than a demonstration of a single hardware capability.
2. A Hybrid Architecture: Autoregressive Understanding and Diffusion Decoding
The approach uses a hybrid architecture that combines an autoregressive module with a diffusion decoder.
The autoregressive component handles semantic understanding, text conditioning, and global composition. It typically contains billions of parameters. The diffusion decoder is responsible for high-fidelity pixel generation and works with a glyph encoder to improve text rendering. In simple terms, the autoregressive model handles “understanding,” while the diffusion decoder handles “accurate visual generation.”
Compared with a pure diffusion model, this hybrid design can provide stronger performance for complex semantics and long-text consistency. Compared with pure autoregressive pixel generation, a diffusion decoder is better suited to preserving visual details and maintaining generation efficiency.
The underlying idea resembles cognitive generation: language-model reasoning is combined with image synthesis. The system first builds a structured semantic representation, and the decoder then turns that representation into a visual result.
From an engineering perspective, the two modules must be aligned carefully. Key issues include:
- Mapping semantic tokens to visual latents
- Designing supervision signals for text rendering
- Choosing between joint training and staged training
- Monitoring the interaction between the two modules
- Using ablation studies to verify the contribution of each component
The hybrid architecture raises the model’s capability ceiling, but it also increases training complexity. Comprehensive observability and controlled experiments are therefore essential.
3. Training Engineering: An End-to-End Pipeline from Preprocessing to Reinforcement Learning
The data pipeline is the first major component of full-workflow training. Distributed decoding, resolution bucketing, text cleaning, glyph-feature extraction, and local caching can help prevent the CPU from becoming a bottleneck.
During pretraining, the system can use large-scale image–text pairs with multiple resolution buckets. During supervised fine-tuning, higher-quality instruction data can be introduced, including posters, covers, and illustrations. During the reinforcement learning stage, preference optimization can improve practical usability, although reward-model bias and training stability require careful attention.
At the parallelism and operator levels, common optimizations include:
- Dynamic multi-stage pipeline dispatch to reduce scheduling overhead
- Multi-stream execution to overlap communication and computation
- Fused operators designed for domestic AI accelerators, such as AdamW EMA and RMSNorm
Checkpoints should be saved asynchronously, and training jobs should support fault recovery. At a minimum, observability should cover throughput, memory usage, communication overhead, operator latency, gradient norms, and data quality.
# Pseudocode: end-to-end training pipelinefor batch in data_loader:tokens = tokenizer.encode(batch.images, batch.text)semantic = autoregressive_backbone(tokens)glyph = glyph_encoder(batch.text)latents = diffusion_decoder(semantic, glyph)loss = diffusion_loss(latents, batch.images) + text_render_loss(latents, batch.text)loss.backward()optimizer.step()
4. Non-Standard Resolutions: Improved Tokenization and Resolution Bucketing
Supporting non-standard resolutions is not simply a matter of allowing arbitrary image sizes during inference. The model must also be exposed to different aspect ratios during training.
One approach is to improve the image tokenization strategy by dividing images into a variable number of tokens or latents and adapting positional encodings to different aspect ratios. Training can use multi-resolution buckets and dynamic batching. During inference, the input can be resized to a supported range while preserving its target aspect ratio—for example, from 1024×1024 to 2048×2048, including non-standard sizes such as 1240×1600.
def build_canvas(w, h, min_pixels, max_pixels):# Example: preserve the aspect ratio while constraining the total pixel countscale = (max_pixels / (w * h)) ** 0.5return round(w * scale), round(h * scale)
This capability is useful for posters, long e-commerce images, and social media covers because it reduces cropping and distortion. However, arbitrary aspect ratios can increase memory fragmentation and create workload imbalance. Resolution bucketing, dynamic batching, and coordinated inference queues are needed to manage these issues.
5. Chinese Text Rendering and Evaluation: From Generation to Deliverables
Chinese text rendering requires a glyph encoder to provide structural constraints. Text-rendering losses can further reinforce character accuracy and layout consistency.
Evaluation should not rely solely on overall visual quality. A dedicated evaluation set should measure:
- Chinese character accuracy
- Completeness of long text
- Text position and layout
- Style consistency
- Failure rate
Public benchmarks for complex visual text generation and long-text rendering can serve as references, but specific rankings should be based on reproducible experiments.
For commercial posters, educational illustrations, and social media covers, a practical delivery workflow can combine template constraints, standardized prompts, batch generation, and manual review.
6. Open-Source Release and Deployment Considerations
Open-sourcing model weights and inference code can reduce the cost of reproduction and secondary development.
For enterprise deployment, the main areas of evaluation should include:
- Data rights and compliance
- Content safety
- Inference cost
- Heterogeneous accelerator scheduling
- Monitoring and observability
When building a similar pipeline on a general-purpose cloud platform, common service categories such as object storage, container platforms, logging, monitoring, and alerting can support data processing, training, and inference workloads. General-purpose offerings from Baidu AI Cloud may also be considered as one possible hosting option, with the final choice depending on cost, compliance requirements, and the team’s technology stack.
Low-cost per-image APIs are suitable for infrequent requests. Batch production, however, depends more heavily on queues, caching, and failure-retry mechanisms.
Overall, the significance of training an open-source image generation model end to end on domestic AI accelerators lies in moving beyond “it runs” toward a workflow that is reproducible, measurable, and deployable. Developers should focus on reproducible metrics, stable text rendering, and the engineering cost of non-standard resolutions rather than on isolated rankings. As operator ecosystems, hybrid parallelism, and inference optimization mature, these approaches may support more complete production practices for Chinese poster generation and custom-size image synthesis.
