Cognitive Image Generation Models: Full-Process Training Practices on Alternative Computing Hardware
Training cognitive image generation models presents significantly greater challenges than inference. This article explores the full-process training pipeline for cognitive image generation models, emphasizing the difficulties of text rendering and cognitive tasks. It details a hybrid architecture combining an autoregressive module for semantic planning and a diffusion decoder for detail rendering. Additionally, it covers strategies for multi-resolution and arbitrary aspect ratio generation, alongside the four key stages of the training workflow: data preprocessing, pretraining, and supervised fine-tuning.
1. Background: Training is Harder Than Inference
There is a long-standing consensus in the industry: while inference can run on various hardware, training is the true threshold. Inference only requires forward computation, with VRAM usage primarily from weights and activations. Training, however, must save gradients and optimizer states while synchronizing parameters across thousands of accelerator cards. Any issue—such as missing operators, insufficient communication bandwidth, or numerical overflow under mixed precision—can interrupt the entire training process.
Therefore, to determine whether a training pipeline is mature, you can focus on the following four aspects:
- Operator Coverage: Whether attention, normalization, and sampling-related operators have high-performance implementations on the target hardware.
- Distributed Efficiency: Whether throughput can scale linearly with an increasing number of cards when combining data parallelism, tensor parallelism, and pipeline parallelism.
- Training Stability: Whether fault detection, checkpoint resumption, and numerical monitoring during long-term training are comprehensive.
- Toolchain: The development experience and debugging costs from data processing to evaluation.
2. Why Cognitive Generation is Challenging
Early image generation models excelled at making visuals look realistic but often failed to render text correctly. Taking a science poster as an example, a model might generate an exquisite atomic structure and starry background, but the text on the poster often contains stroke errors or gibberish. This issue can be summarized as having strong visual capabilities but weak cognitive abilities.
There are two main reasons for this. First, text is highly symbolic information requiring pixel-level accuracy in glyph details, whereas the training objective of diffusion models leans towards fitting the overall image distribution, providing weaker constraints on local symbols. Second, text and physical relationships in images require language-level knowledge. Facts like “the boiling point of water at standard atmospheric pressure is about 100 degrees Celsius” are difficult to learn stably from visual data alone.
Cognitive generation refers to the model simultaneously possessing world knowledge, logical reasoning, and symbolic rendering capabilities. Its goal is to generate visuals with correct content, not just visually realistic images.
3. Hybrid Architecture: Separating Planning and Rendering
Public information indicates that this solution adopts a hybrid structure combining an autoregressive module and a diffusion decoder. The two modules have distinct responsibilities.
- Autoregressive Module: With a scale of about 9 billion parameters, it uses a language model as its base. It understands instructions, breaks down the main subjects of the image, and determines the spatial layout. Its advantage lies in inheriting the semantic understanding capabilities of language models, making it suitable for handling complex references, quantities, and relational constraints.
- Diffusion Decoder: With a scale of about 7 billion parameters, it adopts the DiT architecture. Conditioned on the output of the autoregressive module, it restores high-frequency textures and text strokes through multi-step denoising.
The data flow is roughly as follows:
Prompt -> Autoregressive Module (Semantic Understanding and Composition Planning) -> Conditional Representation -> Diffusion Decoder (Detail and Text Rendering) -> Image
The following pseudocode only illustrates the two-stage data flow and does not correspond to any specific implementation:
# Pseudocode: illustrating the two-stage generation dataflowdef generate_image(prompt, ar_model, diffusion_decoder, vae_decoder, steps=30):# Stage 1: Semantic understanding and global compositioncondition = ar_model.plan(prompt)# Stage 2: Guided by the condition, progressively denoise to restore detailslatent = diffusion_decoder.init_noise()for t in reversed(range(steps)):latent = diffusion_decoder.denoise_step(latent, condition, t)# Decode latent representation into pixel imagereturn vae_decoder.decode(latent)
This division of labor brings a direct engineering benefit: semantic errors and detail errors can be diagnosed separately. If the composition does not match the instruction, you should first check the instruction data and alignment quality of the autoregressive module. If the text strokes are rough, you should prioritize checking the decoder’s training data, resolution settings, and denoising steps.
4. Multi-Resolution and Arbitrary Aspect Ratio Generation
The model natively supports image generation at arbitrary aspect ratios from 1024×1024 to 2048×2048. This capability has clear requirements for both training and inference pipelines. Common practices include the following three approaches:
- Bucketing Training: Training images are grouped into several buckets based on aspect ratio, keeping the dimensions consistent within the same batch. This reduces composition loss caused by cropping to a uniform size and also improves VRAM utilization.
- Positional Encoding Interpolation: During inference, positional encodings from the training resolution are interpolated to the target size, avoiding structural distortion caused by direct extrapolation.
- Progressive Resolution Training: The model first learns overall composition at a lower resolution, then fine-tunes details at a higher resolution, reducing the computational cost of high-resolution training.
The pseudocode for assigning buckets by aspect ratio is as follows:
# Pseudocode: assigning the closest bucket based on aspect ratiodef assign_bucket(width, height, buckets):ratio = width / heightreturn min(buckets, key=lambda b: abs(b.ratio - ratio))
5. The Four Stages of Full-Process Training
“Full-process” typically refers to the complete pipeline from data to model alignment, which can be broken down into four stages.
1. Data Preprocessing
Data quality often determines the model’s upper limit earlier than the scale of computing power. Preprocessing generally includes:
- Deduplication: Removing near-duplicate images based on perceptual hashing or embedding similarity.
- Filtering: Eliminating samples with low resolution, heavy watermarks, or image-text mismatches.
- Caption Rewriting: Replacing short tags with more complete, structured descriptions to enhance the model’s instruction-following ability.
- Text Verification: Performing optical character recognition on images containing text and comparing it with the annotated text to ensure text labels are correct.
2. Pretraining
Pretraining simultaneously optimizes language modeling and denoising objectives on large-scale image-text pairs. In this stage, it is crucial to monitor the stability of the loss curve, gradient norms, and samples processed per second. If the gradient norm suddenly spikes, it usually requires checking the learning rate, mixed precision scaling strategy, and anomalous samples in the data.
3. Supervised Fine-Tuning (SFT)
Supervised fine-tuning uses high-quality instruction and image pairs for alignment. The goal of this stage is to improve style control capabilities, text accuracy, and adherence to complex instructions.
