Training Multimodal Image Generation Models on Domestic AI Chips: Architecture, Text Rendering, and Engineering Challenges
This article examines a multimodal image generation training workflow completed on domestic AI chips, from data preprocessing and large-scale pretraining to instruction tuning and reinforcement-based optimization. It explains why hybrid autoregressive and diffusion architectures are useful, why Chinese text rendering remains difficult, how arbitrary aspect-ratio generation affects system design, and which engineering optimizations are required for full-scale multimodal training.
In recent years, competition in image generation has moved beyond whether a model can produce an image at all. The focus is increasingly on whether it can accurately understand complex instructions, reliably generate usable content, and correctly render text in scenarios such as posters, covers, and promotional materials.
This is especially important for Chinese-language applications. Text in these settings is often highly structured and closely tied to business requirements. Traditional diffusion-only models frequently struggle with misspelled characters, merged strokes, disordered layouts, and unreadable long text.
A recently disclosed technical result shows that a multimodal image generation model has completed a full training workflow on domestic AI chips, covering data preprocessing, large-scale pretraining, instruction fine-tuning, and reinforcement-based optimization. The value of this work is not simply that it uses different hardware. More importantly, it demonstrates the engineering feasibility of training complex multimodal models on domestic compute platforms and provides a reference for building more self-controlled training toolchains.
Why Use a Hybrid Image Generation Architecture?
Traditional image generation systems are typically built around diffusion models. Diffusion models are effective at progressively reconstructing images from noise and can produce strong textures, lighting, and visual detail. However, they often require additional prompt encoders or control modules when handling complex semantic understanding, long-text planning, and relationships among multiple objects.
Another approach is autoregressive visual generation. Similar to how language models predict text tokens step by step, autoregressive visual models predict visual tokens sequentially. They can offer advantages in contextual understanding, user-intent analysis, and global composition planning. However, generating high-resolution images entirely through autoregressive decoding can increase computational cost and latency, while fine-detail quality may still need improvement.
A practical engineering approach is therefore to combine the two methods:
The autoregressive module handles understanding
This module processes user instructions, scene relationships, object attributes, layout requirements, and text content. Its primary role is to determine what should be generated and how the composition should be organized.The diffusion decoder handles image synthesis
The diffusion component converts high-level semantics into visual output. It focuses on textures, colors, lighting, edges, and local details.A text encoding module supports accurate text rendering
For poster headlines, product descriptions, event slogans, and similar content, a dedicated text encoding capability can pass character shapes and layout information into the image generation process. This reduces cases where the model understands the intended meaning but renders the text incorrectly.
This architecture can be summarized as follows: the autoregressive component “understands,” the diffusion component “draws,” and the text encoding component “writes accurately.” Compared with a single-model approach, it is better positioned to balance semantic consistency with visual quality.
Why Chinese Text Rendering Is Difficult
Text rendering in image generation is not simply a problem of visual detail. It sits at the intersection of visual generation and discrete symbol generation.
English has a relatively limited set of characters, making it easier for a model to learn letter shapes and common combinations. Chinese, by contrast, contains a large number of characters, each with complex stroke structures. Even when a model understands the meaning of a requested title, it may still produce issues during rendering, including:
- Incomplete character structures, with missing or extra strokes
- Adjacent characters merging together
- Font styles that do not match the intended visual context
- Unstable long-text layout
- Distorted punctuation, numbers, and special symbols
- Generated text that differs from the input instruction
Improving these results requires more than image-caption pairs. Training data should include structured information whenever possible, such as text regions, character content, positional relationships, font-size ratios, and layout labels. Poster-oriented data should also cover common patterns such as centered text, left-aligned text, wrapped text, multi-column layouts, title hierarchies, and mixed text-image compositions.
At the model level, a text encoder can convert character sequences into representations that preserve spatial relationships and then fuse them with visual features. This enables the model to learn not only that a title belongs in a particular region, but also its exact content, length, and placement.
The Engineering Value of Non-Standard Resolution Support
Many image generation models support only a small number of fixed output sizes, such as square images, wide landscape formats, or vertical poster formats. This simplifies batch training and memory management, but it does not always match real business requirements.
In practical applications, canvas dimensions are often defined by platform requirements or design specifications, including:
- Long images for mobile devices
- Vertical social media covers
- E-commerce campaign posters
- Feed advertising creatives
- Presentation pages
- Widescreen display backgrounds
When a model can generate only fixed aspect ratios, users often need to create an image close to the desired size and then crop, pad, or upscale it. This adds complexity to the workflow and can damage subject composition or text layout.
Supporting arbitrary aspect ratios is not just a matter of exposing adjustable output dimensions. It requires solving three related problems.
1. Dynamic Organization of Visual Tokens
The model must organize visual tokens dynamically according to the requested width and height rather than always applying a fixed grid. As the number of tokens changes, attention computation, positional encoding, and feature alignment must remain stable.
2. Scale Adaptation for Composition
When the canvas aspect ratio changes, the placement of subjects, whitespace, and text regions must change as well. Training data needs to cover enough width-height combinations to prevent the model from memorizing only a limited set of fixed templates.
3. Training and Inference Resource Control
Higher resolutions usually require more visual tokens, which increases memory use and computational cost. Engineering implementations therefore need to balance quality and resource consumption through techniques such as tiled processing, dynamic batching, gradient checkpointing, and mixed precision.
From an application perspective, non-standard resolution support allows an image generation model to fit more naturally into content production, advertising design, and visual marketing workflows instead of functioning only as an experimental tool for producing standard samples.
Key Challenges in Full-Workflow Training on Domestic AI Chips
Completing a full training workflow on domestic AI chips involves more than raw compute capacity. It also depends on software stack adaptation, operator implementation, communication efficiency, and job scheduling.
1. Data Preprocessing Pipelines
Multimodal training commonly involves image cleaning, text filtering, image decoding, resizing, text-region annotation, and sample packaging. If preprocessing cannot keep up with training, compute devices will spend time waiting for data.
Common optimization methods include:
- Parallel data loading
- Pre-caching frequently used metadata
- Decoupling image decoding from training workloads
- Bucketing samples by resolution
- Reducing repeated data-format conversions
2. Overlapping Computation and Communication
Large-model training requires frequent gradient synchronization and parameter updates. If communication fully blocks computation, device utilization can drop significantly. A more mature approach separates compute and communication tasks into different execution streams so that some communication operations can run in parallel with computation for the next batch.
The goal is to minimize idle time spent waiting for communication after computation has completed.
3. Operator Fusion
If optimizer updates, normalization, activation functions, and tensor transformations are executed independently, they can produce many intermediate tensors and memory-access operations. Fusing consecutive operators reduces data movement and scheduling overhead.
Common optimization directions include:
- Fusing normalization and scaling operations
- Optimizing parameter update workflows
- Reducing unnecessary tensor copies
- Using matrix-computation paths suitable for the hardware architecture
- Reducing task-launch overhead in dynamic execution workflows
4. Training Stability
Because multimodal models process both text and images, their training dynamics can be less stable than those of single-modality models. Techniques such as parameter moving averages, gradient clipping, mixed-precision loss scaling, and staged training can help improve convergence quality.
Recommended Training Workflow
A complete multimodal image generation training pipeline can be organized into the following stages.
Stage 1: Dataset Construction
First, establish mappings between images, text, and structured annotations. For text-heavy poster data, it is useful to additionally store text-box coordinates, character content, font attributes, and layout labels.
Stage 2: Foundation Pretraining
Use large-scale image-text data to help the model learn associations among objects, scenes, styles, and language. The goal at this stage is not to optimize for a specific task, but to establish general visual-semantic capabilities.
Stage 3: Instruction Fine-Tuning
Introduce data that better reflects real usage patterns, such as “generate a vertical event poster” or “retain the specified title and adjust text hierarchy.” This helps the model understand constraints expressed in natural language.
Stage 4: Text-Focused Optimization
Conduct specialized training for Chinese characters, numbers, symbols, and long text. Training samples should cover different fonts, font sizes, colors, backgrounds, and layout patterns.
Stage 5: Preference and Quality Optimization
Use human preferences or automated rules to further optimize composition integrity, text accuracy, style consistency, and instruction following. However, visual-quality scores cannot fully replace character-level accuracy metrics. These two dimensions should be evaluated separately.
Evaluation Must Go Beyond Visual Appeal
An image generation model should be evaluated across at least four dimensions:
- Semantic consistency: Whether the image includes the subjects, scenes, and actions requested by the user.
- Composition quality: Whether subject placement, scale, hierarchy, and whitespace are appropriate.
- Text accuracy: The character-level match between generated text and the input text.
- Resolution adaptability: Whether images exhibit stretching, cropping, or layout imbalance across different aspect ratios.
For Chinese posters, character-recognition results should be compared with the input text, alongside human review of character structure and readability. For long text, evaluation should also include line breaks, punctuation, line spacing, and text-region overflow.
Implications for Developers and Enterprises
This technical direction shows that multimodal image generation is moving from single-model generation toward coordinated multi-module systems. When deploying these systems, organizations should consider more than model parameter counts. Data quality, training hardware, inference cost, content safety, and workflow integration are all important.
Teams building controllable image generation systems should first clarify several questions:
- Is accurate Chinese text generation required?
- Is support needed for fixed templates, arbitrary canvas sizes, or both?
- Is private training and deployment required?
- Is sufficient high-quality industry data available?
- Is there a complete evaluation and review mechanism?
- Can datasets and model versions be continuously improved?
Overall, completing full-process multimodal image generation training on domestic AI chips reflects progress in training frameworks, hardware adaptation, and model architecture coordination. More importantly, the combination of autoregressive understanding, diffusion-based generation, and specialized text modeling provides a practical technical path for Chinese posters, cover design, and content production. As operator optimization, data governance, and evaluation systems continue to mature, domestic compute platforms are likely to support a broader range of multimodal model training and inference workloads.
