Training Multimodal Image Generation Models on Domestic AI Compute: A Practical Deep Dive

    This article walks through the engineering practice behind multimodal image generation models trained on domestic AI compute, using Chinese-language poster generation as the main scenario. It explains why the task evolved from "generating images" into a combined system of semantic understanding, visual generation, text layout, and quality alignment; how an autoregressive semantic planning module and a diffusion decoder divide the work; what it takes to render Chinese text reliably; how to train for arbitrary, non-standard aspect ratios; the full pipeline from data preprocessing through pretraining, supervised fine-tuning, and preference alignment; how to adapt training to domestic compute through framework, operator, parallel, and observability work; the engineering metrics that decide whether a model is actually usable; a reference service architecture; common failure modes and mitigations; and where the technology is heading next.

    1. Background: From “Can Produce an Image” to “Controllable Generation”

    Image generation has shifted over the past few years from producing natural-looking images to producing high-quality, editable images that fit specific business scenarios. Early text-to-image models mostly cared about whether the picture looked good and whether the semantics matched roughly. As poster design, social media covers, e-commerce creatives, brand collateral, and explainer illustrations became mainstream use cases, the bar moved considerably. Five requirements stand out:

    1. Accurate Chinese understanding: the model has to recognize the subject, but also understand headlines, sub-headlines, selling points, and calls to action written in Chinese.
    2. Stable text rendering: Chinese text in a poster must not be split into wrong strokes, lose strokes, distort, or turn into meaningless glyphs.
    3. Sensible layout: the subject, text, background, and whitespace need to follow visual design logic.
    4. Flexible resolution: real scenarios are rarely a standard ratio. Portrait covers, long vertical images, squares, and wide banners all need native support.
    5. Controllable training and inference cost: on domestic compute, training, fine-tuning, and deployment all have to balance performance, stability, and engineering feasibility.

    As a result, multimodal image generation has evolved from a single image generation model into an integrated system covering semantic understanding, visual generation, text layout, and quality alignment. In that context, training such models on domestic compute platforms has become a direction that technical teams pay close attention to.


    2. Core Architecture: Autoregressive Semantic Planning Combined with Diffusion Decoding

    To handle complex tasks, a multimodal image generation model usually separates “understanding” from “generating” into two relatively independent stages.

    2.1 The Autoregressive Semantic Module Reads and Plans

    The autoregressive semantic module acts as the brain of the system. It parses the user input and turns natural language into a more structured generation instruction. For example, if the user types “Create a summer new-product promotional poster with a cool, refreshing feel and the headline ‘Limited-Time Offer’”, the semantic module needs to identify:

    • Theme: summer, new product, promotion;
    • Style: refreshing, bright, youthful;
    • Copy: headline, sub-headline, promotional message;
    • Layout: headline at the top, subject in the center, whitespace at the bottom;
    • Color: blue, green, gradients, and similar visual tendencies.

    The goal at this stage is not to paint pixels, but to convert a vague request into a machine-executable generation plan.

    2.2 The Diffusion Decoder Actually Draws It

    The diffusion decoder maps semantic conditions to image content. It typically takes the structured description, text conditions, layout conditions, or region masks produced by the semantic module, then generates the final image through a multi-step denoising process.

    Compared with a purely end-to-end approach, this division of labor has several advantages:

    1. More controllable semantics: text, theme, and style can be constrained separately, which reduces generation drift.
    2. More stable layout: element positions can be controlled through structured layout input.
    3. Better text rendering: text conditions can be reinforced independently instead of being diluted by the overall image generation process.
    4. Easier re-editing: if the copy or layout needs to change later, only part of the conditions have to be updated rather than regenerating everything.

    The flow can be described with pseudocode:

    1. user_input = "Generate an autumn new-product poster with a warm atmosphere"
    2. # 1. Semantic understanding and structured planning
    3. plan = semantic_model(user_input)
    4. # plan may include: theme, style, copy, layout, color scheme
    5. # 2. Diffusion decoding to generate the image
    6. image = diffusion_decoder(
    7. prompt=plan.prompt,
    8. layout=plan.layout,
    9. text_conditions=plan.text_conditions,
    10. resolution=plan.target_resolution
    11. )
    12. # 3. Post-processing and quality checks
    13. image = post_process(image)

    This “semantic planning plus diffusion decoding” approach matches the cognition-oriented generation direction widely adopted in the industry: the model has to form an understanding of the task first, then carry out the visual generation.


    3. Chinese Posters and Complex Text Rendering: From “Looks Like a Picture” to “Usable”

    Text rendering is one of the biggest challenges in poster-style scenarios. Chinese is particularly demanding because of the large character set, complex strokes, and rich structural variation. Without sufficient typographic priors, a model tends to produce these failures:

    • Missing or broken strokes in glyphs;
    • Chinese characters decomposed into incorrect strokes;
    • Chaotic line breaks in long text;
    • Unclear hierarchy between headline and body copy;
    • Text that blends unnaturally into the background;
    • Incorrect promotional messages, prices, or button labels.

    3.1 The Hard Part Is Understanding Text, Not Drawing It

    High-quality Chinese poster generation is not simply pasting text onto an image. The text has to become part of the image semantics. The model needs to understand:

    • Which region is the headline;
    • Which region is the sub-headline;
    • Which region is body copy;
    • Which region is the button label;
    • Which regions should stay empty;
    • What contrast the text color should have against the background;
    • Whether the typeface matches the brand tone.

    Text rendering therefore breaks down into three levels:

    1. Character-level recognition: the model can represent the structure and strokes of each Chinese character.
    2. Sentence-level understanding: the model understands the semantic relationship between headlines, body copy, and promotional phrases.
    3. Layout-level organization: the model can place text in appropriate positions and build a visual hierarchy.

    3.2 Typographic Priors and Glyph Constraints

    In complex poster scenarios, common optimizations include introducing glyph priors, text-region constraints, and character-level condition injection. Put plainly, the model should know not only “there should be text here”, but also “which text should appear here”, “roughly how much area it occupies”, and “whether the typeface matches”.

    For a promotional line such as “限时优惠” (limited-time offer), the model has to keep the characters intact while controlling font size, position, color, and the relationship with the background. If the background is busy, readability usually requires outlines, shadows, color blocks, or semi-transparent plates.

    3.3 Long Text and Multi-Region Layouts

    A poster is rarely just one headline. Typical posters contain a headline, a sub-headline, a list of selling points, price tags, a button, a QR-code placeholder, and brand information. The model needs multi-region text layout and should adjust the composition dynamically based on content length.

    Training data typically has to cover:

    • Single-headline posters;
    • Headline plus sub-headline;
    • Multi-line body copy;
    • Mixed image and text;
    • Promotional badges;
    • Button copy;
    • Left-right split layouts;
    • Top-bottom split layouts;
    • Center-symmetric layouts;
    • Irregular whitespace layouts.

    Only with enough samples across these layout types can a model keep its Chinese poster generation stable in real business use.


    4. Arbitrary-Aspect-Ratio Resolution: Bringing Output Closer to Real Business Needs

    Traditional image generation models usually assume a fixed resolution, such as a standard square, landscape, or portrait size. Posters, social media covers, e-commerce main images, and long detail pages, however, come in very different sizes. If the model only supports fixed ratios, real usage requires cropping, scaling, outpainting, or manual re-layout, which hurts both efficiency and output quality.

    4.1 Why Non-Standard Resolutions Matter

    Native support for arbitrary aspect ratios delivers at least three benefits:

    1. Lower post-processing cost: no generate-then-crop step, which reduces manual adjustment.
    2. Better visual completeness: cropping no longer cuts off the subject, text, or key information.
    3. Multi-terminal fit: the same creative can be used on mobile, desktop, feeds, and detail pages at different sizes.

    For posters, resolution is not only a dimension question but also a composition question. Portrait, square, and wide formats need different visual centers of gravity and element arrangements. If the model sees many aspect ratios during training, the output will follow design conventions more closely.

    4.2 What Variable-Resolution Training Requires

    To make a model natively support non-standard resolutions, the training strategy usually needs adjustment:

    • Multi-scale data mixing: mix samples with different aspect ratios and resolutions during training;
    • Resolution-aware positional encoding: let the model reason about positions across different spatial scales;
    • Dynamic noise initialization: generate latent representations at the scale of the target size;
    • Variable sampling strategy: adapt the sampling process to the target ratio at inference time;
    • Layout constraint input: pass the target size, safe areas, and whitespace regions in as conditions.

    Illustrative code:

    1. target_size = (1240, 1600) # example of a non-standard portrait size
    2. noise = init_latent(height=target_size[1], width=target_size[0])
    3. for step in sampler.steps:
    4. noise = denoise_step(
    5. latent=noise,
    6. condition=structured_prompt,
    7. layout_mask=layout_mask,
    8. resolution=target_size,
    9. step=step
    10. )
    11. image = decode(noise)

    Note that arbitrary resolution does not mean unlimited size. In production you still have to account for memory usage, inference latency, generation quality, and training stability. In practice, teams set a reasonable range and manage resolution tiers according to the business scenario.


    5. The Full Training Pipeline: Data, Pretraining, Fine-Tuning, Alignment

    The capability of a multimodal image generation model is not built in a single training stage. It is shaped by several stages working together.

    5.1 Data Preprocessing Sets the Ceiling

    High-quality data is the foundation of a poster generation model. A typical pipeline includes:

    1. Image-text pair collection and cleaning: remove low-quality, blurry, duplicated, and meaningless samples.
    2. Layout element extraction: detect regions such as headline, body copy, background, subject, and buttons.
    3. OCR text alignment: map the text inside an image back to the original text source.
    4. Copyright and compliance filtering: exclude infringing, sensitive, or non-compliant content.
    5. Quality scoring: filter samples by sharpness, composition, text readability, and aesthetics.

    For Chinese posters, data quality matters even more. The model has to learn not only what an image looks like, but also where text should appear, how the headline should be arranged, and how promotional information should be emphasized.

    5.2 Pretraining Builds the Visual and Semantic Foundation

    In pretraining, the model usually learns large-scale image-text correspondence and forms a basic visual understanding capability. The goal at this stage is not complex poster generation yet, but to establish:

    • Common objects, scenes, and styles;
    • Image-text semantic matching;
    • Color, composition, and lighting;
    • Weak associations between text and image regions.

    The more thorough the pretraining, the more stably fine-tuning and specialized capability training converge later.

    5.3 Supervised Fine-Tuning Teaches Instruction Following

    In supervised fine-tuning, the model learns to generate the target image from a user instruction. Training samples usually contain:

    • The user description;
    • The target poster image;
    • Structured layout information;
    • Text region annotations;
    • Style tags;
    • Quality scores.

    The core of this stage is moving the model from “can generate images” to “can generate images as requested”.

    5.4 Preference Alignment and Reinforcement Learning Improve Subjective Quality

    Image generation is strongly subjective. Different users disagree on what “beautiful”, “clear”, “high-impact”, or “on-brand” means. After supervised fine-tuning, preference optimization or reinforcement-learning-based alignment is therefore usually introduced as well.

    Common objectives include:

    • Improving image-text consistency;
    • Improving text readability;
    • Reducing visual clutter;
    • Improving compositional balance;
    • Raising overall aesthetics;
    • Lowering the risk of prohibited content.

    Evaluation at this stage cannot rely on a single metric. It should combine human evaluation, automatic metrics, and task success rate.


    6. Adapting to Domestic Compute: Stability and Efficiency Together

    Training multimodal models on domestic compute is not only about getting a job to run. It is about keeping training stable, throughput predictable, and resource utilization high. The work usually falls into four areas: framework adaptation, operator optimization, parallel strategy, and scheduling.

    6.1 Training Kits and Framework Adaptation

    Domestic compute platforms typically come with their own training frameworks and operator libraries. To run the model efficiently, the training team needs a complete kit covering data preprocessing, pretraining, fine-tuning, and alignment.

    Typical adaptation work includes:

    • Optimizing the data loading pipeline;
    • Modifying model structures for compatibility;
    • Filling in missing operators;
    • Adapting distributed communication;
    • Improving memory management;
    • Fixing training stability issues.

    6.2 Operator-Level Optimization for Compute Efficiency

    When training large models, a large share of the time goes into matrix operations, normalization, optimizer updates, and attention computation. Common optimization directions:

    • Using fused operators to reduce intermediate tensors;
    • Optimizing normalization modules such as RMSNorm and LayerNorm;
    • Accelerating optimizer components such as AdamW and EMA;
    • Using mixed precision to reduce memory pressure;
    • Improving memory access for long-sequence attention;
    • Removing unnecessary data copies and format conversions.

    Illustrative structure:

    1. Input tensor
    2. -> Linear layer
    3. -> Normalization
    4. -> Activation
    5. -> Residual connection
    6. -> Output tensor
    7. After optimization:
    8. -> Fused linear + normalization
    9. -> Fused activation + residual
    10. -> Fewer intermediate results written back

    6.3 Multi-Stream Parallelism with Communication-Compute Overlap

    In distributed training, communication overhead often limits overall throughput. On multi-card and multi-node setups, running computation and communication serially noticeably reduces resource utilization.

    Common strategies:

    • Split communication and computation into multiple streams;
    • Overlap communication with computation during the forward or backward pass;
    • Use pipeline parallelism to improve utilization at each stage;
    • Use multi-level pipeline dispatch to reduce scheduling bottlenecks;
    • Schedule gradient synchronization, parameter updates, and activation recomputation together.

    These optimizations do not change model logic, but they can improve training efficiency substantially.

    6.4 Training Stability and Observability

    Training on domestic compute also demands observability. Worth monitoring closely:

    • Whether the loss curve stays smooth;
    • Whether gradient norms become abnormal;
    • Whether memory usage spikes;
    • Whether communication shows long waits;
    • Whether data loading has become the bottleneck;
    • Whether a particular node fails repeatedly.

    On the platform side, logging services, monitoring and alerting, container platforms, and object storage can be combined to give training jobs unified scheduling and fault localization. If an organization uses general-purpose capabilities from Baidu AI Cloud, training data, model artifacts, logs, and monitoring metrics can also be brought into one governance system.


    7. Key Engineering Metrics for Chinese Poster Generation

    Whether a model is genuinely usable in production depends less on parameter count than on a set of engineering metrics.

    7.1 Text Accuracy

    For Chinese posters, incorrect text is a hard failure. Text accuracy should cover:

    • Whether characters are complete;
    • Whether strokes are missing;
    • Whether wrong characters appear;
    • Whether meaningless characters appear;
    • Whether long text gets truncated.

    7.2 Image-Text Consistency

    Image-text consistency measures whether the generated image matches the user description:

    • Whether the theme matches;
    • Whether the style matches;
    • Whether all elements are present;
    • Whether the copy is accurate;
    • Whether the colors meet the requirement.

    7.3 Layout Reasonableness

    Layout metrics typically include:

    • Whether the headline stands out;
    • Whether the subject is centered or follows the visual center of gravity;
    • Whether text overlaps the subject;
    • Whether whitespace is sufficient;
    • Whether the hierarchy is clear;
    • Whether the bottom button or caption area is well placed.

    7.4 Resolution Adaptation

    Arbitrary-aspect-ratio generation needs extra evaluation:

    • Whether portrait output is compositionally sound;
    • Whether square output balances information well;
    • Whether wide output shows stretching;
    • Whether long images show repetition or empty areas;
    • Whether non-standard sizes affect text sharpness.

    7.5 Safety and Compliance

    Poster generation must account for content safety:

    • Sensitive-word filtering;
    • Prohibited image detection;
    • Copyright risk flags;
    • Protection of minors;
    • Avoiding misleading or false advertising risk.

    8. A Typical Application Flow

    To put multimodal image generation into a real business system, a flow like this works well.

    8.1 Structuring the User Input

    User input is usually natural language, but the system needs to convert it into structured parameters. For example:

    1. {
    2. "scene": "new product promotion",
    3. "style": "fresh",
    4. "headline": "Autumn Refresh",
    5. "subtitle": "Great products, limited-time offer",
    6. "cta": "Shop now",
    7. "aspect_ratio": "portrait",
    8. "color_theme": ["orange", "cream"],
    9. "safety": true
    10. }

    Once structured, the system can more easily apply template control, run quality checks, and support later edits.

    8.2 Orchestrating the Generation Task

    The pipeline can be split into several tasks:

    1. Semantic understanding;
    2. Layout planning;
    3. Image generation;
    4. Text enhancement;
    5. Content safety review;
    6. Human review.

    For high-concurrency scenarios, a message queue can drive asynchronous task scheduling so that a single generation request does not block incoming requests.

    8.3 Deploying the Inference Service

    Image generation is compute-intensive, so the inference service needs attention to:

    • Pooled management of GPU and NPU resources;
    • Batching strategy;
    • Request queuing and timeout control;
    • Canary release of model versions;
    • Caching of inference results;
    • Retry on failure.

    8.4 Archiving Results and Iterating

    Generated results should not only be returned to the user; they should also be archived to object storage or a data warehouse for later evaluation. Worth saving:

    • The input prompt;
    • The structured parameters;
    • The generated image;
    • The model version;
    • The inference latency;
    • User feedback;
    • The safety review result.

    This data can then feed continuous training and quality optimization.


    9. Common Challenges and How to Address Them

    9.1 Long-Tail Chinese Characters Are Hard to Render

    Some characters have complex strokes or unusual structures and are prone to generation errors. Improvements come from expanding glyph data, adding glyph constraints, strengthening OCR alignment, and introducing text-region losses.

    9.2 Text Is Unreadable on Complex Backgrounds

    If the background is too busy, text becomes hard to read. Contrast control, text plates, outlines, and shadows can be added as design constraints during generation.

    9.3 Composition Breaks Down with Many Elements

    When the frame contains a subject, several text regions, and decorative elements, the model may crowd the image or leave too little whitespace. Layout templates, region masks, and layout constraints help.

    9.4 Insufficient Training Resources

    On domestic compute, memory, compute, and communication can all become limits. Data parallelism, tensor parallelism, pipeline parallelism, activation recomputation, and mixed-precision training all reduce resource pressure.

    9.5 Evaluation Is Highly Subjective

    Poster quality is hard to capture with one metric. A combination of automatic metrics, human scoring, and business conversion rate is the better approach, so that visual appeal is not pursued at the expense of actual usability.


    10. Where This Is Heading: From Generation Tool to Design Collaboration System

    The future of multimodal image generation is not just “type a sentence, get a picture”. It is moving toward editable, collaborative, closed-loop design systems. Likely directions:

    1. Stronger instruction following: accurately handling edit requests such as “make the headline larger”, “change the background to dark blue”, or “move the button to the bottom right”.
    2. Finer-grained editing: local inpainting, text replacement, subject movement, and style transfer.
    3. Better brand compliance: automatically following brand colors, fonts, logo usage rules, and visual style.
    4. Lower training cost: continuous cost reduction through domestic compute adaptation, data flywheels, and evaluation systems.
    5. Stronger safety and compliance: complete content safety mechanisms before, during, and after generation.

    For technical teams, training multimodal image generation models on domestic compute is not a simple hardware swap. It requires system-level design across data, architecture, training, operators, inference, evaluation, and business workflow. Only when model capability and the engineering pipeline are combined can complex scenarios such as Chinese posters, social media covers, and brand collateral be delivered reliably.