Training a Multimodal Image Generation Model End to End on Domestic AI Chips

    This article examines the technical value, engineering challenges, evaluation methods, and deployment considerations involved in training an image generation model entirely on domestic AI hardware. It covers model architecture, operator compatibility, mixed-precision stability, data quality, systematic evaluation, asynchronous inference services, and the progressive optimization of domestic hardware and software stacks. It also explains why production readiness depends not only on image quality, but also on reproducibility, stability, maintainability, compliance, and cost.

    1. Reconsidering the Value of Training Image Generation Models on Domestic Chips

    At first glance, the goal of a text-to-image model seems straightforward: a user provides a description in Chinese or English, and the system generates an image that matches the intended meaning. The output may take the form of a realistic image, illustration, concept design, poster draft, or another visual style.

    For enterprise research and development, however, the question of whether a model can generate an image is only the most basic evaluation criterion. More important questions include:

    • Can the model accurately understand complex text?
    • Are composition and visual quality stable across generations?
    • Does the model respond naturally to Chinese prompts?
    • Can training run continuously on the target computing platform?
    • Can the model be deployed in production at a reasonable cost?
    • Can the hardware and software stack be maintained and expanded over the long term?

    A conventional approach is to train a model on mature, general-purpose accelerators and then migrate it to domestic chips for inference. This can provide a relatively fast way to validate application performance. However, differences in operators, numerical precision, compilation, and performance tuning may emerge between the training, inference, and deployment stages.

    The significance of end-to-end training on domestic hardware is that data processing, model training, checkpoint storage, validation, evaluation, and inference deployment are completed as much as possible within the same type of domestic computing environment. The goal is not simply to replace one type of hardware with another. Rather, it is to verify whether the domestic hardware and software ecosystem can support the complete lifecycle of a complex generative model.

    2. Core Components of an Image Generation Model

    A relatively complete text-to-image system generally includes text encoding, image generation, and image decoding.

    Text Encoder

    The text encoder converts natural language into vector representations that the model can process. For example, given the prompt “a city street at night, wet pavement reflecting neon lights, cinematic composition,” the encoder must identify the subject, environment, time, material properties, and visual style.

    Image Generation Network

    Modern generation systems typically do not generate complete pixel images directly from text in a single step. Instead, they gradually construct an image representation in a compressed latent space. This reduces computational requirements and allows the model to progressively refine composition, color, and detail through multiple iterations.

    Image Decoder

    The image decoder converts the latent representation into a viewable image. Its quality directly affects the clarity, texture, edges, and color reproduction of the final output.

    A complete system also includes data cleaning, training orchestration, precision control, model checkpointing, and inference services. Problems in any of these components can reduce final quality. For example, inaccurate text annotations can impair semantic understanding, imbalanced data distributions can create stylistic bias, and unsuitable precision settings during inference can produce visible artifacts.

    3. Major Challenges in End-to-End Domestic Training

    3.1 Operator Compatibility and Computational Graph Stability

    Image generation models contain extensive matrix operations, normalization layers, attention mechanisms, activation functions, and tensor transformations. Support for these operators varies across hardware platforms. Problems are especially likely in dynamic-graph, mixed-precision, and large-scale distributed training scenarios.

    Common issues include missing operators, computational graphs that cannot be compiled, and inconsistent results during execution.

    Engineering validation should go beyond checking whether the model can start. It should also cover the following:

    • Whether forward computation remains stable
    • Whether backpropagation can continue over extended periods
    • Whether gradients become abnormal
    • Whether different input batches produce consistent results
    • Whether the training state is restored correctly after checkpoint recovery
    • Whether communication across multiple cards or devices is reliable

    A common strategy is to begin with a small dataset and a short training cycle to validate the computational graph. Batch size, resolution, and training duration can then be increased gradually. This narrows the scope of potential problems and avoids discovering low-level operator defects only after a long training run.

    3.2 Mixed Precision and Numerical Stability

    To reduce memory consumption and improve throughput, image generation models commonly use lower-precision computation while retaining higher-precision representations for selected operations. However, image generation is sensitive to numerical changes, particularly in attention computation, normalization layers, loss calculation, and noise scheduling.

    An unsuitable precision strategy may lead to:

    • Loss values suddenly becoming non-numeric
    • Large areas of generated images becoming a single solid color
    • Severe gradient fluctuations during the early stages of training
    • Significant differences between outputs generated on different devices
    • A decreasing training loss without any corresponding improvement in image quality

    Mixed precision should therefore not be understood as simply lowering the precision of every operation. A more appropriate approach is to apply different precision levels to different modules. Computationally intensive matrix operations can use lower precision, while numerically sensitive normalization, weight updates, and loss calculations retain higher precision. Gradient scaling, anomaly detection, and periodic sampling should also be used for monitoring.

    3.3 Data Quality Matters More Than Data Volume

    The capabilities of an image generation model depend heavily on its training data. A large dataset is not necessarily a high-quality dataset. If image-text alignment is inaccurate, the model may learn incorrect associations.

    Data processing typically includes the following tasks:

    • Removing duplicate or highly similar images
    • Filtering blurry, overexposed, or heavily compressed samples
    • Checking whether images match their text descriptions
    • Standardizing image dimensions, color spaces, and file formats
    • Cleaning noisy text annotations
    • Handling sensitive content and copyright risks in compliance with applicable requirements
    • Controlling the proportions of different styles, resolutions, and subjects

    Chinese-language scenarios require additional attention to omitted information, synonyms, colloquial expressions, and relationships between multiple entities. For example, “a cat is standing beside a table, with an open book on the table” and “beside the table is a cat, while an open book is placed on the table” have similar meanings. The model must still learn stable image-text alignment across different sentence structures.

    4. How to Evaluate the Actual User Experience

    A model should not be judged by one or two generated images. A more reliable approach is to build a test set that covers semantics, structure, quality, and stability.

    4.1 Basic Semantic Understanding

    Test simple objects, colors, quantities, and spatial relationships, such as:

    • A red cup on a wooden table
    • Two birds perched on a power line
    • A white vehicle driving on a rainy street

    These prompts can verify whether the model correctly understands the subject and its basic attributes.

    4.2 Complex Composition

    The next step is to introduce multiple objects, spatial relationships, and actions. For example, a prompt may require people, buildings, and vehicles to appear simultaneously while remaining in specified positions.

    The main evaluation points are whether the model omits subjects, confuses relationships, or changes the requested quantity.

    4.3 Text Rendering in Images

    Posters, interface mockups, and infographics often require text to appear inside the generated image. Evaluation should focus on whether characters are clear, content is complete, and the layout is reasonable.

    Image generation models generally remain more challenged by precise text rendering than by natural image generation. Overall visual quality is therefore not enough; text errors should be measured separately.

    4.4 Style Consistency

    Use the same subject with different style prompts to determine whether the model can preserve the subject while changing its visual expression. For example, the style can be changed from realistic photography to watercolor illustration and then to a concept design.

    If a style change is so strong that the subject is lost, the model’s conditional control is still unstable.

    4.5 Stability Across Multiple Generations

    Generate multiple images from the same prompt and observe whether composition, subject count, color, and detail remain within a reasonable range.

    Stability is especially important for enterprise applications. Real-world workflows typically require batch generation rather than an occasional image that happens to look good.

    5. Key Issues in Inference Deployment

    Completing training does not mean that the system is ready for use. Inference deployment must also address model loading time, memory consumption, concurrency, task queuing, and failure recovery.

    For an online service, the workflow can be divided into request reception, text preprocessing, task scheduling, model inference, result storage, and status reporting. Because generation tasks are usually computationally intensive, clients should not be required to wait continuously. An asynchronous task model can be used instead:

    1. Submit prompt
    2. ↓
    3. Add generation task to the queue
    4. ↓
    5. Inference service consumes the task
    6. ↓
    7. Save the generated result
    8. ↓
    9. Return task status and result address

    In production, general-purpose infrastructure such as result storage, message queues, container platforms, logging services, and monitoring and alerting systems can support this workflow. The specific architecture should be selected according to traffic volume, data sensitivity, and the deployment environment.

    The following runtime information should also be recorded:

    • Model version
    • Inference precision
    • Input prompt
    • Random seed
    • Image dimensions
    • Generation time
    • Failure reason
    • Device utilization and memory usage

    This information helps reproduce problems and compare changes between model versions.

    6. A Basic Approach to Hardware and Software Co-Optimization

    Domestic adaptation is not a one-time task. It is a continuous optimization process that should proceed in the following order: correctness first, stability next, and efficiency after that.

    The first phase validates functional correctness and ensures that the model can complete forward computation, backpropagation, and basic inference.

    The second phase improves runtime stability, with a focus on long-duration training, checkpoint recovery, communication across multiple devices, and exception retries.

    The third phase targets performance optimization, including operator fusion, memory reuse, batch scheduling, model partitioning, and adjustments to parallel execution strategies.

    The fourth phase improves service capabilities by adding access control, content review, rate limiting, audit logging, monitoring, and alerting.

    Performance should not be measured only by the latency of a single generation. A more complete set of metrics includes the number of images generated per unit of time, device utilization, peak memory usage, failure rate, long-term runtime stability, and cost per task.

    7. Implications for Developers and Enterprises

    Training an image generation model entirely on domestic chips validates more than the performance of a single model. It tests the completeness of the broader research and development system, including data engineering, algorithm development, compiler optimization, cluster scheduling, and service deployment.

    For developers, the focus should shift from “Can the model run?” to “Can it run stably, reproducibly, and maintainably?”

    For enterprise users, additional concerns include data compliance, content safety, cost control, and integration with business workflows.

    Overall, image generation on domestic hardware is no longer merely an algorithm experiment. Through a complete training pipeline, systematic evaluation, and engineering-oriented deployment, organizations can gradually build generative AI capabilities that are more autonomous, controllable, and sustainable.

    The quality of an actual implementation should be judged through multidimensional testing. It is important to assess not only whether the images look good, but also whether the semantics are accurate, the outputs are stable, the system is reliable, and the overall solution can genuinely support business scenarios.