Training Multimodal Image Generation Models on Domestic AI Chips: Architecture, Pipeline, and Engineering Practice

    Building image generation systems for posters, educational visuals, and social media covers requires accurate instruction understanding, layout planning, and text rendering—not just attractive pixels. This article explains a hybrid approach in which an autoregressive multimodal model interprets prompts and plans layouts while a diffusion decoder renders the final image. It then examines the full training pipeline on domestic AI chip clusters, covering data preprocessing, large-scale pretraining, instruction tuning, and reinforcement optimization. Particular attention is given to Chinese text rendering, flexible aspect ratios, parallel training, fused operators, stability monitoring, evaluation, and deployment.

    In recent years, competition in image generation has shifted from whether a model can produce an image to whether it can accurately understand complex requirements and consistently generate usable content. In commercial posters, educational explainers, social media covers, and similar scenarios, users expect more than visual appeal. Titles, captions, brand slogans, and layout structures must be rendered accurately. These tasks involve language understanding, visual composition, text layout, and pixel-level generation at the same time. A single conventional diffusion model often struggles to handle all of them well.

    A notable technical approach is to use an autoregressive multimodal model for understanding and planning, followed by a diffusion-based decoder for high-quality image generation, while moving the entire training process onto a cluster of domestically developed AI chips. The value of this approach goes beyond generation quality. It also validates the ability to adapt data preprocessing, large-scale pretraining, instruction fine-tuning, and reinforcement optimization to the domestic chip stack.

    1. Why Use a Hybrid “Understanding Model + Generative Decoder” Architecture?

    Traditional image generation models often perform well in style and local detail, but they can fail when handling long text, complex instructions, and multi-object relationships. Common issues include:

    • Inaccurate understanding of conditional constraints in the user description;
    • Deviations in the main subject, action, or spatial relationships;
    • Incorrect, missing, or distorted Chinese characters;
    • Difficulty preserving complete long sentences, tables, and multi-paragraph content;
    • Support for only fixed image sizes or a limited range of aspect ratios.

    The core idea of the hybrid architecture is to separate the understanding task from the rendering task.

    The first component is an autoregressive multimodal model. It parses text instructions, recognizes image semantics, plans the overall layout, and determines the title, subject, background, color scheme, and visual hierarchy. It functions more like a language reasoning module with visual capabilities, focusing on what should be generated.

    The second component is a diffusion-based image decoder. It converts high-level semantics and layout plans into actual pixels, adding texture, lighting, materials, and typographic details during iterative denoising. It focuses on how the content should be rendered.

    From a system perspective, the process can be abstracted as follows:

    1. User instruction
    2. ↓
    3. Multimodal understanding and structured planning
    4. ↓
    5. Layout, text, subject, and style conditions
    6. ↓
    7. Diffusion-based image decoding
    8. ↓
    9. High-resolution image

    This division reduces the difficulty of making a single model handle every task at once. It also makes it easier to optimize text rendering, composition control, and image quality separately.

    2. Why Full-Pipeline Training on Domestic AI Chips Matters

    If only the inference stage runs on domestic AI chips, the result mainly demonstrates deployment adaptation. When the entire process—from data preprocessing to large-scale pretraining—is completed in a domestic AI chip environment, it indicates that the training framework, operator implementations, parallel strategies, and resource scheduling have all been systematically validated.

    A complete training pipeline usually includes four stages.

    2.1 Data Preprocessing

    Data preprocessing is not simply downloading images and cleaning text. It requires a data pipeline suitable for multimodal training, including:

    • Image quality filtering;
    • Matching between images and description text;
    • Resolution and aspect-ratio normalization;
    • Duplicate sample removal;
    • Filtering of sensitive and low-quality content;
    • Chinese text normalization;
    • Construction of image-text pairs, regional text annotations, and layout information.

    For text generation tasks, additional annotations are needed for text regions, character-sequence validation, and layout structures. Data quality directly affects character accuracy, so the pipeline should avoid relying only on large-scale weakly labeled data.

    2.2 Large-Scale Pretraining

    The pretraining stage mainly teaches the model general visual knowledge, language semantics, and image distributions. The autoregressive module needs to learn image content understanding and conditional planning. The diffusion module needs to learn how to recover image details from noise.

    On domestic AI chip clusters, training systems usually need to address memory capacity, communication efficiency, operator coverage, and computational stability. Common methods include:

    • Combining data parallelism and tensor parallelism;
    • Using gradient accumulation to reduce per-device memory pressure;
    • Applying mixed-precision training to improve throughput;
    • Using activation recomputation to reduce intermediate memory usage;
    • Saving checkpoints in sharded form;
    • Overlapping communication and computation;
    • Using dynamic batching to improve resource utilization.

    2.3 Instruction Fine-Tuning

    A pretrained model has general capabilities, but it may not be able to perform poster design, text layout, or multi-round revisions according to user requirements. Instruction fine-tuning requires examples that are closer to real applications, such as:

    • Generating a poster in a specified style from a one-sentence request;
    • Arranging titles and illustrations according to a given size;
    • Preserving the main subject of an original image while modifying text;
    • Generating educational images that contain multiple explanatory text blocks;
    • Reconstructing the visual hierarchy based on a reference layout.

    Fine-tuning data should cover both simple instructions and complex composite instructions. It should also explicitly record text content, layout constraints, style requirements, and output dimensions.

    2.4 Reinforcement Optimization

    Once the model can generate basically usable images, further optimization is needed for user preferences and task metrics. Reward signals can be designed around text accuracy, instruction adherence, layout reasonableness, and visual quality.

    For a Chinese poster, for example, the system can separately calculate:

    • The match between OCR results and the target text;
    • Whether the target subject appears in the specified region;
    • Whether the image meets size and aspect-ratio requirements;
    • Image quality assessment results;
    • User preference scores for the overall result.

    Multiple reward objectives must be balanced. Otherwise, the model may sacrifice visual quality to improve text accuracy, or overemphasize composition while ignoring user-specified content.

    3. Why Chinese Text Rendering Is Difficult

    Text generation has long been a weak point for image models. Conventional image generation mainly focuses on texture and visual similarity, while text has a strict discrete structure. A single incorrect stroke can change the overall meaning.

    The main challenges of Chinese text rendering include:

    1. A large number of characters and complex glyph structures;
    2. The need to preserve the order and completeness of long text;
    3. The influence of font, size, line spacing, and alignment on layout;
    4. The need for text to coordinate with the background, subject, and whitespace;
    5. Numerous fine-grained visual differences across typefaces.

    A common improvement is to add a dedicated glyph encoding module in addition to the diffusion decoder. This module encodes text content and glyph structure into more explicit conditioning information, which is then passed to the image decoding process. The model does not need to rely entirely on pixel statistics to guess characters; it can instead refer to more structured glyph features.

    For engineering evaluation, it is not enough to measure whole-image similarity. Character-level and layout-level metrics should also be included, such as character accuracy, phrase completeness, text-region detection rate, and the ability to preserve long text.

    4. The Practical Value of Non-Standard Resolutions

    Many image generation systems support only a few fixed sizes, such as square images, wide horizontal banners, or vertical posters. Fixed aspect ratios simplify training and inference, but they differ from real design requirements.

    Real-world image sizes vary widely:

    • Mobile feed covers;
    • Long-form e-commerce campaign images;
    • Conference promotional posters;
    • Vertical video covers;
    • Infographics;
    • Display-screen backgrounds.

    If a model natively supports more flexible aspect ratios, users do not need to first generate a fixed-size image and then crop or stretch it for secondary processing. Enabling this capability usually requires improvements to the image tokenization strategy so that the model can adapt to different spatial dimensions and cover a sufficiently rich range of aspect ratios during training.

    It is important to note that supporting arbitrary aspect ratios does not mean that all sizes maintain the same quality. As the canvas becomes longer, text layout, memory usage, and attention computation costs all increase. The system therefore still needs techniques such as tiled processing, positional encoding design, staged resolution training, and memory optimization.

    5. Engineering Optimization Priorities for Training Systems

    When training large multimodal models on domestic AI chip clusters, performance bottlenecks may come not only from compute units but also from data loading, operator scheduling, and inter-device communication.

    Multi-Level Pipelines and Dynamic Scheduling

    Organizing data loading, preprocessing, host-to-device transfer, and device computation into a pipeline can reduce device idle time. Dynamic scheduling can adjust batches according to sample length and image size, preventing a small number of large samples from slowing overall throughput.

    Multi-Stream Parallel Execution

    During training, computation, communication, and data movement should not be fully serialized. Multi-stream mechanisms can overlap gradient synchronization with the next round of computation, reducing idle time caused by communication.

    Fused Operators

    If optimizer updates, normalization, activation functions, and gradient processing are each launched as separate operators, scheduling overhead can be high. Fusing consecutive operations helps reduce memory reads and writes and the number of kernel launches. For large-model training, this type of optimization is often more valuable in practice than simply increasing peak compute capacity.

    Training Stability

    Large-scale training also requires attention to loss anomalies, gradient overflow, node failures, and checkpoint recovery. A complete monitoring system should record loss curves, throughput, memory usage, communication time, gradient norms, and anomalous samples. A verifiable checkpoint mechanism should also be designed.

    6. How to Evaluate Whether Such a Model Is Truly Usable

    Model evaluation should not rely only on a small set of sample images. It should cover at least four dimensions:

    • Semantic consistency: whether the generated content follows the text instruction;
    • Image quality: whether details, clarity, lighting, and structure are natural;
    • Text capability: whether characters are correct and long text remains complete;
    • Engineering capability: whether target sizes are supported and whether inference latency and resource consumption are acceptable.

    For enterprise applications, stability testing should also be included. For example, does text accuracy fluctuate significantly when the same request is generated multiple times? Does the model preserve content integrity when fonts, styles, and backgrounds change? Does it ignore key conditions when handling very long instructions or multiple constraints?

    7. Recommendations for Deployment

    A team planning to build an independently controlled multimodal image generation system can proceed in the following order:

    1. Define the core scenarios first, such as poster generation or educational illustrations.
    2. Build high-quality image-text and text-layout datasets.
    3. Choose a hybrid architecture and clarify the boundary between the understanding module and the generation module.
    4. Prioritize Chinese text and non-standard image sizes.
    5. Optimize training parallelism, communication, and operator layers.
    6. Iterate using a combination of automatic metrics and human review.
    7. Finally, improve content safety, permission management, log auditing, and inference services.

    Overall, training multimodal image generation models on domestic AI chips is not merely a matter of replacing the hardware environment. It also promotes a complete loop across training frameworks, data engineering, model architecture, and application evaluation. The autoregressive model handles understanding and planning; the diffusion decoder handles detail and final presentation; a glyph encoding module strengthens text accuracy; flexible tokenization supports multiple canvas ratios; and parallel training with fused operator optimization improves efficiency. Together, these elements can help image generation move from “being able to produce an image” toward “being stable and usable.”