Training Multimodal Image Generation Models on Domestic AI Chips: A Technical Path

    Image generation is shifting from basic image synthesis toward instruction following, accurate text rendering, and production-grade output. This article examines a technical path for training multimodal image generation models end to end on domestically developed AI chips, combining an autoregressive module for semantic understanding and layout planning with a diffusion decoder for high-quality rendering. It covers the challenges of Chinese poster generation, the hybrid architecture, data and training pipeline engineering, communication–computation overlap, operator optimization, arbitrary-aspect-ratio support, text-rendering evaluation, deployment validation, and the value and limitations of this approach.

    In recent years, competition in image generation models has gradually shifted from whether a model can generate an image to whether it can understand complex requirements, render text accurately, and adapt to real production scenarios. For commercial posters, social media cover images, information cards, and event promotional graphics, visual appeal is only a baseline. Whether the text is correct, the layout is reasonable, and the dimensions meet publishing requirements often determines whether a model can be deployed in practice.

    Against this background, a notable technical path is taking shape: using domestically developed AI chips to complete the full training pipeline for multimodal image generation models, and using a hybrid architecture that combines an autoregressive model with a diffusion model to address semantic understanding, layout planning, and high-quality rendering at the same time. Compared with conventional image generation approaches that focus primarily on visual detail, this path places greater emphasis on the model’s cognitive capabilities, text handling, and engineering controllability.

    1. Why Chinese poster generation is more difficult

    Ordinary natural image generation mainly focuses on subjects, backgrounds, colors, and spatial relationships. For example, if a user enters “a car on a city street at night,” the model only needs to generate an image that matches the semantics. Poster generation, however, must handle a more complex set of constraints:

    1. The text content must match the prompt.
    2. Chinese characters must not have missing strokes, incorrect characters, or random glyphs.
    3. Titles, subtitles, and body text need to form a clear hierarchy.
    4. Text must be placed in reasonable areas and must not obscure the main subject.
    5. Different font sizes, type styles, and colors must remain coordinated.
    6. The canvas size may not follow a fixed aspect ratio.

    Chinese text has high character complexity: a single Chinese character contains many strokes and structural details. If a model relies only on ordinary image features, it may produce text that looks like writing at a global level but is partially unreadable. The problem becomes more severe when the image contains long titles, multiple lines of explanatory text, or small-font text.

    A true image generation model for poster scenarios therefore cannot learn only what an image looks like. It must also understand what the text says, where it should be placed, and how to present it in a correct visual form.

    2. Hybrid architecture: one module for understanding, one for drawing

    This type of model typically uses a structure that combines an autoregressive module with a diffusion decoder.

    2.1 The autoregressive module handles semantics and layout

    The autoregressive module can be understood as a multimodal foundation model with strong language reasoning capabilities. It is responsible for the following tasks:

    • Parsing the user’s natural language description;
    • Identifying the subject, scene, style, and color requirements;
    • Understanding the title, body text, and hierarchy in a poster;
    • Inferring the spatial relationships among visual elements;
    • Forming a relatively complete image layout and generation plan.

    For example, suppose a user requests: “Create a vertical poster for a technology event, with the main title at the top, an abstract chip pattern in the middle, and time and location information at the bottom.” The autoregressive module does not directly output pixels. Instead, it first translates the request into structured visual intent, including canvas orientation, subject position, text regions, and style constraints.

    The advantage of this design is that the model can perform semantic-level planning before rendering by the subsequent module, reducing the probability that an image looks good but fails to meet requirements.

    2.2 The diffusion decoder handles visual generation

    The diffusion decoder converts high-level semantic and layout information into the final image. It generates details through iterative denoising and can handle:

    • Lighting and materials;
    • Local structures of people and objects;
    • Background textures;
    • Color transitions;
    • Font regions and decorative elements;
    • The overall visual style.

    In text generation scenarios, the diffusion decoder usually needs to work with a dedicated text or glyph encoding mechanism. A glyph encoder can pass the structural information of Chinese characters to the generation process in a more explicit form, so that the model does not rely entirely on fuzzy visual memory but receives clearer constraints on strokes and forms.

    The responsibilities of the two modules can be summarized as follows:

    • The autoregressive module understands what should be generated.
    • The diffusion decoder determines how it should be drawn.

    This division of labor helps balance complex instruction following with image quality and is better suited to application scenarios that require substantial text content.

    3. Engineering significance of full-pipeline training on domestic AI chips

    Completing the full process from data preprocessing to large-scale training on domestic AI chips is not merely a matter of getting the model to run. The real challenges come from software-hardware adaptation, operator performance, communication efficiency, memory management, and training stability.

    3.1 Data preprocessing must work with the training pipeline

    Image generation models usually need to process massive amounts of images, text, text annotations, and layout information. The data preprocessing stage may include:

    • Image cleaning and quality filtering;
    • Resolution and aspect-ratio statistics;
    • Image-text matching checks;
    • Text-region detection;
    • Character-content validation;
    • OCR result filtering;
    • Training-sample deduplication;
    • Unification of multimodal annotation formats.

    If preprocessing is inefficient, training accelerators will wait for data. From an engineering perspective, it is important to use parallel data processing, sharded storage, caching mechanisms, and prefetch strategies so that the input pipeline can continuously supply training batches.

    3.2 The training framework must cover the full lifecycle

    A practical training suite cannot support pretraining only. It should also cover:

    • Large-scale pretraining;
    • Instruction fine-tuning;
    • Supervised image-text training;
    • Preference optimization;
    • Reinforcement learning or reward model training;
    • Model evaluation;
    • Checkpoint saving and restoration;
    • Inference and service deployment.

    Pretraining establishes general visual and language capabilities. Supervised fine-tuning strengthens target scenarios such as posters and covers. Preference optimization can further improve composition aesthetics, text readability, and instruction following.

    3.3 Communication and computation should overlap as much as possible

    In large-scale training, devices must frequently exchange gradients, parameters, and activations. If communication completely blocks computation, device utilization will drop significantly.

    Common optimization approaches include:

    • Using multiple parallel streams so that communication and computation can proceed simultaneously;
    • Splitting gradient synchronization into multiple stages;
    • Compressing or bucketing communication data;
    • Optimizing tensor partitioning and device-topology mapping;
    • Reducing per-task waiting time through pipeline parallelism;
    • Pipelining input processing, forward passes, backward passes, and updates.

    In addition, dynamic graph execution can lead to frequent task dispatch and high scheduling overhead. Multi-level pipeline dispatch, operator fusion, and execution-graph optimization can reduce control overhead and improve training throughput.

    4. Operator optimization is critical for adapting to domestic AI chips

    Whether a model can be trained efficiently often depends on whether low-level operators match hardware characteristics. For large-scale generative models, normalization, optimizer updates, matrix computation, and attention are all high-frequency operations.

    Common optimization directions include:

    • Fusing multiple consecutive operations into one compute unit;
    • Optimizing memory access for normalization operators;
    • Reducing reads and writes of intermediate tensors;
    • Using mixed precision to reduce memory and bandwidth pressure;
    • Sharding optimizer states;
    • Optimizing parameter updates such as exponential moving averages;
    • Selecting more suitable compute paths for different tensor shapes.

    Take mixed precision as an example. Some computations can be performed at lower precision to reduce memory usage and improve throughput, but loss computation, normalization, and some sensitive layers still require higher precision to ensure training stability. Mixed precision is therefore not simply a matter of reducing the precision of all tensors uniformly. It requires fine-grained configuration based on model structure.

    5. Why arbitrary aspect ratios matter

    Traditional image generation models usually use a small number of fixed sizes, such as square, wide landscape, or vertical canvas. This approach simplifies batching and positional encoding design, but it does not fully match real content-production needs.

    Actual canvas sizes vary widely:

    • Mobile long images;
    • E-commerce detail pages;
    • Event promotional posters;
    • Video covers;
    • Article images for social platforms;
    • Display-screen banners;
    • Feed advertising creatives.

    If a model supports only a few fixed aspect ratios, users often need to generate a standard image first and then crop or pad it. This adds post-processing work and may also truncate subjects, shift text positions, or destabilize the composition.

    Supporting arbitrary aspect ratios usually requires improvements in three areas: the tokenizer, positional encoding, and training data.

    First, the tokenizer must flexibly convert images of different sizes into visual tokens rather than adapting only to a fixed grid. Second, positional encoding must be able to express spatial relationships under different widths and heights. Third, the training data should cover a sufficiently rich range of aspect ratios so that the model genuinely learns composition rules for landscape, portrait, and non-standard canvases.

    It is important to note that “supporting arbitrary aspect ratios” does not mean unlimited generation at any size. Real systems are still constrained by GPU memory, computation, minimum and maximum resolution, and image quality. A more accurate description is that, within a supported size range, the model can offer more flexible canvas settings than traditional fixed-ratio models.

    6. How to evaluate text rendering capability

    Image quality cannot be judged only by human perception. A dedicated evaluation system for text generation is also needed.

    6.1 Text accuracy

    A text recognition model can be used to recognize generated results and compare the recognized text with the target text. Common metrics include character-level accuracy, word-level accuracy, and edit distance.

    6.2 Layout consistency

    Evaluation should check whether titles appear in expected regions, whether body text overflows, whether line spacing is reasonable, and whether text and subjects overlap or obscure each other.

    6.3 Glyph quality

    Even when character recognition results are mostly correct, issues such as distorted fonts, connected strokes, and blurred edges may still occur. Local zoom inspections and structural similarity evaluation are therefore also needed.

    6.4 Instruction following

    For complex prompts, evaluation should check whether the model satisfies subject, style, color, text, and size requirements at the same time, rather than generating only some of them.

    6.5 Real-world business usability

    The final evaluation should also consider the cost of manual correction. If a poster requires designers to substantially rearrange text and subjects, it may not have practical production value even if its visual quality is high.

    7. Practical issues in deployment

    After training is complete, the model still needs compression, inference optimization, and service-oriented processing before ordinary users can rely on it consistently.

    First, different inference modes should be selected according to business needs. Real-time interaction places greater emphasis on response speed, while batch poster production places greater emphasis on throughput and unit cost. Quantization, caching, staged inference, and batch scheduling can reduce resource consumption.

    For text-heavy content, it is advisable to add an automatic validation step after generation:

    1. Run text recognition on the image.
    2. Compare the recognition result with the user input.
    3. Check whether key fields are missing.
    4. Determine whether text regions exceed boundaries.
    5. Trigger a retry or local repair when abnormal results are detected.

    This “generation plus validation” loop is more reliable than a single generation pass, especially for information that requires high accuracy, such as dates, locations, prices, and event names.

    8. Value and limitations of this technical path

    The value of training multimodal image generation models on domestic AI chips lies not only in replacing a particular hardware option. It also lies in driving the formation of a complete loop across model training, compiler optimization, operator adaptation, and toolchain development. For enterprises, this can improve control over compute supply, reduce dependence on a single hardware environment, and accumulate engineering experience for subsequent industry-model training.

    However, this path still faces clear challenges. Training costs are high, the software and hardware ecosystem needs continuous improvement, model migration requires substantial work, and text generation may still contain errors in scenarios involving small fonts, complex typefaces, and multilingual mixed layouts.

    Future optimization efforts may therefore focus on three areas: improving data quality and layout annotation accuracy; further strengthening text-specific encoding and controllable generation; and improving automated toolchains spanning training, evaluation, validation, and deployment.

    Overall, “autoregressive understanding + diffusion generation + dedicated text encoding + full-pipeline training on domestic AI chips” represents a relatively complete technical approach to multimodal generation. It no longer treats image generation as a pure pixel-synthesis task. Instead, it unifies language understanding, spatial layout, text rendering, and engineering deployment. For teams that need to produce posters, covers, and visual marketing assets at scale, this architecture is closer to real business requirements and offers more room for continuous optimization and large-scale deployment.