Technical Pathways and Practical Lessons for Training Multimodal Image Generation Models on Domestic AI Chips

    This article examines an end-to-end approach to training multimodal image generation models on domestic AI chips. It covers hybrid architecture design, Chinese text rendering, arbitrary aspect-ratio generation, distributed training optimization, data and evaluation strategies, deployment considerations, and the broader engineering value of building an independently controllable AI infrastructure stack.

    Image generation models are evolving from simply producing pictures to accurately understanding user requirements and completing complex visual tasks. In applications such as commercial posters, infographics, educational illustrations, and social media covers, users expect more than visual appeal. They also need accurate text, clear layouts, and output dimensions that match publishing requirements.

    Against this backdrop, multimodal image generation models trained entirely on domestic AI chips have attracted attention. Their value lies not only in image quality, but also in demonstrating an engineering path that covers data preprocessing, large-scale pretraining, instruction tuning, and quality optimization within a domestic computing environment. For organizations building independently controllable AI infrastructure, this type of practice offers useful technical reference.

    1. Why End-to-End Training on Domestic AI Chips Matters

    Large-scale multimodal model training typically involves data processing, distributed computing, memory management, communication scheduling, operator execution, and model evaluation. If any part of the hardware or software stack is poorly adapted, training efficiency may decline or the system may become unstable.

    End-to-end training generally includes the following stages:

    1. Data preprocessing: Clean images, filter text, normalize resolutions, pair images with text, and check annotation quality.
    2. Pretraining: Use large-scale image-text data to learn visual content, language semantics, and layout structures.
    3. Supervised fine-tuning: Improve the model’s ability to understand complex requirements through high-quality instruction data.
    4. Preference or reinforcement optimization: Further improve generation quality based on human preferences, rule-based scores, or automated evaluation results.
    5. Inference deployment: Convert the model into an execution format suitable for production services and optimize latency, throughput, and memory usage.

    When all of these stages run on domestic chips and their supporting software stack, the process demonstrates coordination among hardware, compilers, training frameworks, communication libraries, and operator implementations. This is more meaningful than simply porting a completed model to another hardware platform, because it provides a stronger indication of the maturity of the underlying computing infrastructure.

    2. A Hybrid Architecture: Language Understanding for Planning, Diffusion for Rendering

    This type of model commonly adopts a hybrid design that combines an autoregressive module with a diffusion decoder. The central idea is to divide a complex task among specialized components.

    2.1 The Autoregressive Module: Understanding Requirements and Organizing the Scene

    The autoregressive module is mainly responsible for text understanding, visual-semantic analysis, and overall composition planning. For example, when a user asks for “a vertical promotional poster with a title, product highlights, and footer notes,” the model needs to determine:

    • What the main subject should be;
    • Where the title and body text should appear;
    • Which elements should receive visual emphasis;
    • What style the overall image should use;
    • How text, background elements, and decorations should be layered.

    This part is closer to the reasoning capabilities of a language model. The system must do more than convert text into visual features. It must understand relationships among multiple objects and the functional role of each area in the composition.

    2.2 The Diffusion Decoder: Generating Details and Visual Quality

    The diffusion decoder converts high-level semantics into a concrete image. It handles colors, textures, lighting, human poses, object boundaries, and local details.

    Traditional diffusion models can perform well when generating general artistic images. However, they often struggle with continuous text, complex layouts, and small-font content. Common problems include incorrect characters, missing characters, distorted strokes, and disorganized text structures. Image generation systems designed for text-heavy content therefore often introduce a dedicated text encoding module, allowing the model to represent glyphs, character order, and text regions more explicitly.

    2.3 The Text Encoding Module: Improving Chinese Text Rendering

    Chinese text generation is more challenging than rendering English words. Chinese contains a large number of characters with highly varied structures, and posters often include multiple lines, different font sizes, and mixed layouts.

    A dedicated text encoding module can help in several ways:

    • Establish a correspondence between text content and character-level visual features;
    • Strengthen character order and semantic consistency;
    • Provide spatial layout information for text regions;
    • Reduce garbled output and randomly formed glyphs;
    • Improve the readability of titles, buttons, labels, and other short text.

    However, a text encoding module cannot guarantee perfect accuracy for every complex text prompt. Long paragraphs, small fonts, and dense layouts still require higher-quality training data, staged generation, and post-processing to improve reliability.

    3. Supporting Non-Standard Resolutions

    Many image generation systems support only a limited set of fixed sizes, such as square images, widescreen formats, or standard vertical posters. Fixed dimensions simplify batch training and memory management, but real-world applications often require custom canvas sizes.

    Posters, feed covers, and mobile content cards use a wide range of aspect ratios. If a model can generate only fixed-size images, users must crop or recompose the output afterward. This can cut off important subjects, shift text positions, or create unbalanced empty space.

    Supporting arbitrary aspect ratios generally requires solving three problems.

    3.1 Variable Token Counts

    After an image is encoded, it is represented as visual tokens. When resolution or aspect ratio changes, the number of tokens changes as well. The model must support visual sequences of different lengths rather than relying solely on fixed positional encoding.

    3.2 Sufficient Training Data Coverage

    If nearly all training data consists of square images, the model may produce unbalanced compositions on narrow or elongated canvases. Training data should therefore cover landscape, portrait, and other non-standard aspect ratios. It should also provide appropriate annotations for subject placement, edge margins, and text regions.

    3.3 Memory and Compute Management

    Higher resolutions generally produce more visual tokens, significantly increasing computation and memory consumption. Engineering approaches such as staged training, dynamic batching, gradient accumulation, mixed precision, and resolution bucketing can help balance quality and cost.

    For practical applications, supporting custom canvases across a range of resolutions reduces the need for post-generation cropping and layout adjustments. This is particularly useful for marketing materials, content covers, and image-text cards.

    4. Training Optimization in a Domestic Computing Environment

    When training large models on domestic AI chips, optimization must address not only the model architecture but also the execution pipeline.

    4.1 Reducing Dynamic Dispatch Overhead

    When a model contains substantial dynamic computation logic, frequent task dispatch can become a bottleneck. Multi-level pipeline scheduling can organize computation in advance and reduce device idle time.

    The basic approach is to separate data preparation, computation, and result transfer into multiple stages. Different batches can then execute in an overlapping manner, improving hardware utilization.

    4.2 Overlapping Communication and Computation

    Distributed training typically requires gradient synchronization, parameter updates, and activation transfer. If communication is performed entirely in series, devices may remain idle while waiting for data transfer.

    A more efficient approach is to use multiple streams so that selected communication tasks run concurrently with computation. For example, gradient communication can begin immediately after certain network layers finish computing while backpropagation continues through other layers. This reduces the impact of communication on total training time.

    4.3 Fusing Operators to Reduce Memory Access

    A significant amount of time in deep learning workloads is spent reading and writing tensors and transferring intermediate results. Fusing multiple consecutive operations into a single operator can reduce intermediate tensor materialization and repeated memory access.

    Common optimization directions include:

    • Improving parameter update procedures;
    • Fusing normalization and activation-related computations;
    • Increasing the execution efficiency of root mean square normalization;
    • Reducing repeated reads and writes during exponential moving average updates;
    • Optimizing memory access for attention mechanisms.

    These changes may not alter the theoretical model architecture, but they can have a substantial impact on training stability and actual throughput.

    5. Training Data and Evaluation Are Equally Important

    The quality of an image generation model is not determined by parameter scale alone. Data quality, annotation accuracy, and evaluation design all directly affect the final result.

    For Chinese posters and text-heavy images, evaluation should cover at least the following dimensions:

    1. Semantic consistency: Whether the image matches the user’s description and whether the subject and actions are correct.
    2. Text accuracy: Whether the generated text matches the input and whether characters are missing or incorrect.
    3. Layout quality: Whether the title, body text, main subject, and decorative elements form a clear hierarchy.
    4. Visual quality: Whether the colors, composition, sharpness, and details appear natural.
    5. Aspect-ratio adaptability: Whether the main subject remains complete when the canvas size changes.
    6. Long-text stability: Whether multiple lines retain the correct order and remain readable.

    For text-rendering tasks, manual subjective scoring alone is not sufficient. An OCR model can be used for automated detection, followed by manual sampling and review. This improves evaluation efficiency and helps identify specific failure patterns, such as character errors concentrated in small-font regions or degradation in the latter part of long text.

    6. Deployment Recommendations for Production Applications

    Organizations planning to use this type of model in production can adopt a layered deployment strategy.

    During offline processing, the system can perform model inference, content moderation, and asset filtering, making it suitable for batch generation of marketing materials. Online services can then expose parameterized generation capabilities, including canvas aspect ratio, style, text content, and subject position. High-concurrency scenarios also require request queues, caching, batch inference, and task-priority management.

    Appropriate safety controls should be established as well:

    • Filter sensitive content in input text;
    • Review generated images for compliance;
    • Record model versions and generation parameters;
    • Retry failed requests and classify their causes;
    • Add human confirmation for high-risk scenarios.

    7. Technical Value and Future Directions

    The significance of training multimodal image generation models on domestic AI chips is not limited to replacing one type of hardware. More importantly, it encourages the model, framework, compiler, and application layers to form a complete and coordinated system.

    Future development is likely to focus on three areas:

    1. Further improving complex Chinese typography so that the model can reliably generate longer text and more detailed layouts;
    2. Reducing the cost of high-resolution generation so that non-standard canvas sizes do not impose a significant inference burden;
    3. Enhancing controllability, allowing users to specify regions, font styles, layout structures, and modification areas more precisely.

    Overall, the hybrid architecture of language understanding, visual generation, and dedicated text encoding provides a clear path for addressing the weaknesses of traditional image generation models in Chinese text rendering and complex composition. End-to-end training on domestic computing infrastructure also contributes reusable engineering experience for building independently controllable multimodal AI systems.