Training Multimodal Image Generation Models on Domestically Developed AI Accelerators: Technical Paths and Practices
This article examines how multimodal image generation models can be trained entirely on domestically developed AI acceleration hardware, covering data pipelines, hybrid autoregressive-diffusion architectures, glyph encoding for Chinese text rendering, non-standard resolution support, full-stack training challenges, and deployment considerations. It explains why Chinese text generation remains difficult, how planning and pixel generation can be separated, and what engineering practices matter when moving from pretraining to production use.
In recent years, competition in image generation models has shifted from whether a model can generate an image to whether it can understand complex requirements, lay out text correctly, and reliably serve real business scenarios. In Chinese-language posters, marketing assets, infographics, and social media covers, for example, a model must not only produce a visually coherent image but also understand long text, hierarchy, layout constraints, and target dimensions.
A recent class of multimodal image generation models has attracted attention because it was trained entirely on domestically developed AI acceleration hardware. Its technical significance is not merely the replacement of training hardware. More importantly, it explores a complete domestic path from data preprocessing and large-scale pretraining to supervised fine-tuning and reinforcement-based optimization. For developers and enterprises, such practice offers three useful references: how to build data systems suitable for Chinese visual tasks, how to design a model architecture that balances semantic understanding and text generation, and how to reduce large-scale training costs through software-hardware co-design.
1. Why Chinese Text Generation Remains Difficult
Traditional text-to-image models are often good at generating natural landscapes, people, and abstract artistic effects. However, once a large amount of text appears in the image, accuracy drops noticeably. Common problems include:
- Missing strokes or deformed structures in Chinese characters;
- Text content that does not match the user input;
- Incorrect ordering of multi-line text;
- Unstable control over font, font size, and spacing;
- Repetition, truncation, or garbled characters in long text;
- Text that exists but becomes difficult to read after blending with the background.
The root cause is that image generation models usually learn visual scenes through continuous visual features, while text itself is discrete, precise, and strongly sequential. A subtle stroke change in a Chinese character can completely change its meaning. A model that can “draw textures that look like text” does not necessarily understand and output the correct text.
Therefore, models for Chinese posters and infographics need to solve two problems at the same time. First, they must understand the theme, style, layout, and semantic relationships in the user description. Second, they must accurately render specified text into the image in a readable and controllable form.
2. Hybrid Architecture with an Autoregressive Model and a Diffusion Decoder
A more practical approach is to split the model into two parts with clearly defined responsibilities.
2.1 Autoregressive Module for Understanding and Planning
The autoregressive module can be understood as a language model with visual generation planning capabilities. It is mainly responsible for:
- Parsing user input;
- Identifying the subject, scene, and style;
- Determining spatial relationships in the image;
- Planning text regions such as titles, subtitles, and labels;
- Organizing the overall composition and visual hierarchy;
- Handling multi-turn instructions and complex constraints.
It is closer to “the designer’s thinking process.” For example, given the instruction “Generate a vertical poster for a technology event, with the main title at the top, a product illustration in the center, and the time and location at the bottom,” this module must first understand the poster structure and then translate the relationships among text, subject, and layout into visual conditions for subsequent generation.
2.2 Diffusion Decoder for Pixel-Level Generation
The diffusion decoder is responsible for turning high-level semantics and layout plans into a concrete image. It is suitable for handling:
- Color and material;
- Lighting and texture;
- Graphic details;
- Fusion of background and subject;
- Visual presentation of font outlines and text regions.
In text generation scenarios, a glyph encoding module can be introduced to encode character structure information as visual conditions. In this way, the decoder no longer merely “guesses what the text might look like” from vague semantics. Instead, it can refer to clearer glyph, stroke, and layout information.
In simplified terms, the autoregressive module “understands the requirement and decides how to draw,” the diffusion decoder “draws the planned content,” and the glyph encoding module “ensures the text is written correctly.”
3. How Glyph Encoding Improves Chinese Rendering
In Chinese scenarios, glyph encoding is an important component for improving text accuracy. The basic idea is not simply to feed text as an ordinary prompt, but to preserve structural information about the text at the same time.
A feasible data processing pipeline includes:
- Detect text regions in training samples;
- Extract text content, position, orientation, and line-level relationships;
- Generate corresponding glyph representations based on character information;
- Align text semantic features with glyph features;
- Constrain generation results during diffusion training;
- Enhance stability under complex layouts through supervised fine-tuning.
This approach is particularly suitable for content with explicit character requirements, such as titles, buttons, labels, prices, times, and addresses. For enterprise applications, text correctness is often more important than purely visual appeal, because an incorrect date, amount, or brand name can make an entire asset unusable.
It is important to note that glyph encoding does not automatically solve all layout problems. A practical system also needs to handle font style, letter spacing, line spacing, text box size, and background contrast. Therefore, training data should include diverse combinations of fonts, font sizes, orientations, colors, and layouts.
4. Engineering Value of Non-Standard Resolution Support
Traditional image generation systems are often trained around a few fixed sizes, such as square, landscape, or portrait ratios. This design simplifies batching and memory management, but it differs from real design requirements.
In real business scenarios, image dimensions are often determined by the specific channel:
- Posters need to approximate paper or display-board ratios;
- Social media covers need to fit different devices;
- E-commerce hero images need to meet platform display specifications;
- Infographics and long images need more vertical space;
- Screen backgrounds may use ultra-wide ratios.
If a model supports only fixed sizes, users often need to generate first and then crop or pad the image. This can cause problems such as truncated subjects, shifted text positions, and imbalanced composition.
Supporting arbitrary ratios requires more than exposing an output size parameter. It also requires improvements to the image tokenizer, positional encoding, and training sampling strategy. The model needs to encounter enough width-height combinations during training and develop generalization across different spatial layouts. At the same time, memory usage and computational complexity must be controlled to avoid instability in training or inference when dimensions change.
For developers, non-standard resolution capability can reduce post-processing steps and allow the “required size” to enter the generation pipeline directly. This improves visual consistency and makes it easier to integrate the model into poster generation, advertising creative, content operations, and automated design systems.
5. Key Challenges in Full-Pipeline Training on Domestically Developed Chips
Completing the full process from data processing to large-scale training on domestically developed acceleration hardware is usually less about a single operator and more about coordination across the entire software stack.
5.1 Data Preprocessing Efficiency
Multimodal training data often includes images, text, region annotations, glyph information, and quality labels. If data reading, decoding, cropping, and encoding cannot keep up with computation, the accelerators will sit idle.
Engineering practices need to use sharded storage, parallel preprocessing, cache reuse, and asynchronous loading to minimize data preparation time in the main training loop. For high-resolution images, abnormal samples should also be controlled on the data side to prevent extreme sizes from slowing overall throughput.
5.2 Balancing Dynamic Graphs and Static Execution
Image generation tasks may have large variations in input size, text length, and number of conditions. Dynamic execution improves flexibility, but frequent graph submission and compilation introduce additional overhead.
A common optimization is to use a multi-stage pipeline that separates graph preparation, data transfer, and device execution, while reducing repeated submission through caching. For frequently executed subgraphs, strategies such as stable shapes, precompilation, or operator fusion can be used.
5.3 Overlapping Communication and Computation
Large-scale training usually requires multi-card parallelism. Parameter synchronization, gradient aggregation, and activation transfer consume significant time. If communication and computation are fully serialized, device utilization drops noticeably.
Therefore, the training system needs to use multi-stream parallelism so that some communication tasks can execute at the same time as computation. Without affecting numerical stability, gradient bucketing, communication topology optimization, and hierarchical synchronization can also reduce waiting time.
5.4 Operator Adaptation and Numerical Stability
Optimizers, normalization, exponential moving averages, and attention modules can all become performance bottlenecks. Designing fused operators for domestically developed hardware can reduce intermediate tensor reads and writes and the number of kernel launches.
However, more fusion is not always better. The training system must also monitor overflow under mixed precision, gradient underflow, and accumulated errors. Techniques such as loss scaling, keeping key layers in higher precision, and gradient clipping help ensure training stability.
6. Complete Pipeline from Pretraining to Alignment Optimization
An image generation model intended for practical applications usually needs to go through the following stages.
Data Pretraining
This stage mainly learns the basic correspondence between images and text, including objects, scenes, styles, layouts, and common text structures. Data scale and quality are equally important. Low-quality captions, duplicate images, and incorrect text directly affect model capability.
Supervised Fine-Tuning
Supervised fine-tuning is used to improve performance on specific tasks, such as Chinese posters, knowledge illustrations, product covers, and complex instruction understanding. Training samples should include explicit prompts, target images, text regions, and layout information.
Preference Optimization
The same prompt may correspond to multiple reasonable results, so human or automatic evaluation is needed to select samples that better meet requirements. Optimization objectives can include text accuracy, composition rationality, aesthetic quality, instruction following, and subject consistency.
Quality Evaluation
Evaluation should not rely only on image similarity. It should also cover:
- Text recognition accuracy;
- Long-text completeness;
- Layout compliance;
- Stability at non-standard sizes;
- Subject-background consistency;
- Ability to follow complex instructions;
- Inference speed and memory consumption.
For enterprise deployment, it is advisable to build both an offline evaluation set and an online feedback mechanism, feeding user edits, retries, and manual review results back into training data.
7. Application Scenarios Suitable for Deployment
This type of model is best suited to scenarios where text and visuals jointly create core value, including:
Commercial poster generation
Automatically generate titles, selling points, time and location, and visual subjects based on the event theme, reducing the time needed for initial design drafts.Educational and explanatory illustrations
Transform complex knowledge into visual content with explanatory text, step annotations, and structural diagrams.Social media covers
Generate covers at specified ratios and automatically lay out title areas and subject areas.E-commerce marketing assets
Generate promotional labels, product selling points, and contextual backgrounds, while still requiring human review before official publication.Content operations support
Batch-generate illustrations, summary cards, and column visual templates based on article topics.
8. Notes for Developers
First, do not treat a generative model as fully automated design software. For critical text such as amounts, dates, addresses, and legal statements, recognition checks and human review must be added.
Second, prompts should be structured. Developers can explicitly specify canvas size, subject, text content, text position, font style, color, and whitespace requirements instead of providing only a broad description.
Third, production systems should ideally adopt a closed loop of “generate, detect, and correct.” After generation, use text recognition, layout detection, and image quality models for automatic checks, then regenerate incorrect results or send them to a manual review queue.
Finally, deployment requires a comprehensive assessment of model size, memory, concurrency, latency, and cost. For high-concurrency services, text parsing, image generation, and quality detection can be split into independent modules and processed asynchronously through task queues.
Overall, the value of training multimodal image generation models on domestically developed chips lies in the combined advancement of training infrastructure, model architecture, and application capability. The autoregressive module improves understanding of complex requirements, the diffusion decoder ensures visual generation quality, glyph encoding improves Chinese text accuracy, and non-standard resolution support enhances adaptability to real business needs. In the future, whether such systems can become more widespread will still depend on data quality, training efficiency, text reliability, and engineering deployment cost.
