Training Image Generation Models on Domestic AI Chips: A Full-Stack Practice and Evaluation

    This article examines the practical experience of training text-to-image generation models entirely on domestic AI chips, covering the full pipeline from data preprocessing and training to deployment. It discusses evaluation criteria beyond visual quality, key technical challenges such as operator adaptation, memory management, and distributed training stability, as well as methods for building repeatable test workflows and production-ready serving systems. The goal is to provide a balanced perspective on what full-stack domestic chip training means for controllability, reliability, and long-term iteration.

    Rethinking Full-Stack Domestic Training for Text-to-Image Models

    Text-to-image models typically take a natural language description as input and generate an image that matches the semantic requirements. The input can be in Chinese or English, while the output covers realistic photography, concept design, illustration, posters, product sketches, and more.

    If one only looks at the final images, it is tempting to regard such models as simple “prompt in, image out” tools. But for industrial deployment, the truly important factors are not just visual quality. They include whether the training process is controllable, whether the software and hardware environment is stable, whether the inference pipeline is complete, and whether the model can be iterated in computing environments of different scales.

    Traditional approaches often train models on general-purpose high-performance accelerators and then migrate the model to domestic chips for inference. This approach can verify model quality quickly, but using different computing architectures for training and deployment often leads to engineering issues such as inconsistent operator behavior, numerical precision differences, different memory management strategies, and non-reusable compilation optimizations.

    The value of full-stack domestic training lies in performing data preprocessing, model training, parameter saving, evaluation, and inference deployment all on the same type of domestic AI computing environment. This does not mean every step becomes inherently simpler. Rather, it means the entire system can be validated under a unified technology stack, making it more suitable for scenarios that require autonomous control, long-term maintenance, and local deployment.

    Evaluating Model Capabilities from Multiple Dimensions

    Assessing an image generation model should not rely on cherry-picking a few impressive samples. A more reasonable approach is to establish a multi-dimensional test set that evaluates the model’s comprehension, generation, and engineering capabilities.

    Chinese Prompt Understanding

    Chinese prompts often contain many modifying relationships, quantity relationships, and spatial relationships. For example:

    “At dusk, a pedestrian wearing a light-colored coat stands beside an old station, with three warm-yellow streetlights in the distance. The scene adopts a cinematic composition.”

    Such a description includes time, person, clothing, scene, quantity, and visual style information at once. The model must not only recognize keywords but also correctly understand the relationships between words. If the model only generates the station and the pedestrian but fails to accurately represent the number of streetlights, light direction, or object positions, it indicates there is room for improvement in complex semantic binding.

    Chinese generation also faces issues like word segmentation differences, complex long-sentence structures, and diverse cultural contexts. Therefore, practical evaluation should focus on whether the model can stably understand the relationships among subject, action, environment, material, viewpoint, and style, rather than simply testing simple noun combinations.

    Complex Composition and Spatial Relationships

    A good image generation model needs to handle information like “who is in what position, in what pose, and interacting with which objects.”

    Tests can be designed around the following categories:

    • Scenes with foreground, midground, and background elements simultaneously;
    • Multiple characters performing different actions;
    • Objects with occlusion, arrangement, or relative position relationships;
    • Clear overhead, side, or close-up viewpoints;
    • The same subject appearing repeatedly in different regions.

    These tasks test spatial reasoning much better than simply generating a single object. Common failure modes include unstable character counts, flipped left-right relationships, objects intersecting each other, actions that violate common sense, and background details being randomly replaced.

    Text and Symbol Generation

    Posters, packaging, interface sketches, and infographics require the model to generate legible text. Currently, image models widely suffer from text deformation, missing strokes, character confusion, and misaligned layout.

    Therefore, text generation capability should be evaluated separately, not mixed into the general art image score. During testing, the following aspects should be observed:

    • Whether short text retains character structure;
    • Whether mixed Chinese-English text is stable;
    • Whether numbers, dates, and symbols are error-prone;
    • Whether multi-line text has reasonable layout;
    • Whether text content matches the prompt.

    If the business scenario heavily depends on readable text, a combination of “image generation plus post-processing layout” is usually still recommended, rather than relying on the model to generate a final product in one shot.

    Style and Detail Representation

    Realistic images require attention to lighting, materials, perspective, and texture. Illustration images emphasize line consistency, color harmony, and composition. Design sketches require clear subject contours and well-defined structural relationships.

    Different styles have different evaluation criteria. For example, in realistic images, hands, eyes, reflections, and shadows are frequent defect areas. In illustrations, broken lines, local style drift, or insufficient color depth may occur. Tests should establish corresponding metrics based on the target task, and the same standard should not be used for all images.

    Technical Challenges in Full-Stack Training on Domestic Chips

    Operator Adaptation and Computation Graph Conversion

    Image generation models typically include modules such as attention computation, normalization, matrix operations, sampling, and image encoding/decoding. Different hardware environments may differ in operator support scope, input layouts, and data type requirements.

    During migration, simply copying a model file to another environment is not sufficient. Developers usually need to:

    • Check whether core operators have corresponding implementations;
    • Replace or combine unsupported operators;
    • Adjust tensor layouts and data types;
    • Verify compatibility between dynamic and static shapes;
    • Handle differences between training and inference computation;
    • Perform performance analysis and optimization on key operators.

    Among these, training places higher demands on operator completeness and numerical stability. Some operators may work for inference but are not suitable for long-duration backpropagation training.

    Memory and VRAM Management

    Training image generation models typically requires saving activations, gradients, optimizer states, and intermediate features. Memory pressure rises quickly as image resolution, batch size, and model scale increase.

    Common engineering measures include:

    • Using gradient accumulation to reduce per-batch memory requirements;
    • Using mixed precision to reduce memory usage;
    • Recomputing certain intermediate results;
    • Using staged training to control resource consumption;
    • Optimizing data loading, caching, and prefetching;
    • Partitioning model parameters, gradients, and optimizer states appropriately.

    It is important to note that lowering precision does not mean simply converting all computation to low precision. Parts of normalization, loss calculation, and weight update still require higher precision; otherwise, training oscillation, gradient overflow, or degraded generation quality may occur.

    Distributed Training Stability

    When a single computing device cannot meet training requirements, multi-device collaborative training becomes necessary. Communication efficiency, synchronization strategies, and fault recovery capabilities directly affect training speed.

    Typical problems in distributed training include:

    • Long communication wait times between nodes;
    • One device failure causing the entire task to abort;
    • Inconsistent computing speeds across devices;
    • Excessive checkpoint saving time;
    • State inconsistency after training recovery.

    Therefore, production systems should include periodic checkpoint saving, random state logging, training loss monitoring, and automatic anomaly detection. Simply getting the model to run does not mean the training pipeline is mature. Being able to run stably for days or longer is closer to production-level requirements.

    How to Conduct Practical Evaluations

    “Practical experience” should not simply mean entering a few prompts and showing images. A repeatable testing process should be established.

    First, prepare a fixed set of prompts covering people, landscapes, products, complex scenes, poster designs, and abstract concepts. Each prompt should generate multiple results to observe randomness and stability.

    Second, record the following information:

    • Input prompt;
    • Output resolution;
    • Generation time;
    • Sampling settings used;
    • Whether content deviated from the prompt;
    • Whether obvious structural errors exist;
    • Whether multiple retries were needed;
    • Differences in results across device environments.

    Finally, score results manually from five aspects: semantic alignment, image clarity, composition rationality, detail completeness, and result stability. For enterprise applications, additional dimensions such as compliance, safety, and sensitive content control should be included.

    From a general usability perspective, if a model trained through full-stack domestic chip training reaches mainstream quality, it usually already has good Chinese prompt response capability and basic style coverage. Users can generate human, scene, and design images through natural Chinese descriptions, which is suitable for creative sketches, content previews, and visual solution exploration.

    However, complex text layout, precise spatial relationships, multi-person interactions, and professional product structure remain common pain points. For tasks that require direct use in printing, advertising, or industrial design, generated results often still need manual selection, local repair, and post-generation layout.

    From Research Results to Actual Deployment

    Model quality is only part of deployment. Service capability must also be considered.

    For API design, it is recommended to structure request parameters, model version, random seed, generation results, and error messages. Generation tasks often take a long time, so asynchronous task patterns are preferred over long-lived connections.

    For resource scheduling, queue management can be based on image resolution, task priority, and concurrent requests. For peak-hour traffic, rate limiting and timeout policies should be set to prevent a few large tasks from affecting the overall service.

    For observability, the following should be logged:

    • Request volume and success rate;
    • Average generation time and tail latency;
    • Device utilization and memory usage;
    • Model loading time;
    • Types of failed tasks;
    • Quality changes between different versions.

    If the model is deployed in a container platform or a private computing environment, it should also be integrated with common infrastructure such as log services, monitoring alerts, object storage, and task queues, forming a complete chain from request intake to result delivery.

    How to Assess Its Technical Value

    The value of full-stack training on domestic chips should not be summarized simply as “generating better images” or “replacing certain hardware.” The more important value appears in three aspects.

    First, it validates the ability of domestic AI software and hardware stacks to support complex generative models. Completing training, inference, and deployment in a unified environment helps accumulate experience in operator adaptation, distributed communication, memory management, and model optimization.

    Second, it reduces dependence on a single computing ecosystem. For government, enterprise, research, and local deployment scenarios, software and hardware autonomy, supply chain stability, and data staying within the domain are often as important as image quality.

    Third, it provides an engineering foundation for subsequent model iterations. Only when the training pipeline truly works end-to-end can developers continuously perform data cleaning, model fine-tuning, quality evaluation, and inference optimization, rather than stopping at a one-time model migration.

    Overall, the practical experience of training image generation models on domestic chips should be viewed from a broad perspective: not just “are the images good looking,” but “does the model understand accurately, is training stable, is deployment reliable, and can the system evolve sustainably.” For ordinary users, it represents improved efficiency in Chinese creative and visual design. For developers, it is closer to a complete engineering validation. For enterprises, it relates to data security, infrastructure autonomy, and long-term operational costs.

    Therefore, the most reasonable way to judge whether such a model is mature is not to rely on a few demonstration samples, but to conduct comprehensive evaluation using standardized prompts, repeated generation tests, resource monitoring, fault recovery, and real business workflows. Only when model quality and engineering reliability are balanced can it move from technical demonstration to sustainable application.