Training a Text-to-Image Model End to End on Domestic AI Chips

    This article examines how to build a production-ready text-to-image system entirely on a domestic AI hardware and software stack. It covers data governance, training stability, compiler and operator support, inference deployment, Chinese prompt alignment, evaluation methodology, and practical measures for safety, reproducibility, and long-term maintenance.

    Multimodal image generation has moved beyond the stage of simply producing images. For developers and enterprises, the real challenge is making the entire pipeline scalable, reproducible, and deployable. The key differentiators are often not the visual quality of a single sample, but whether the system can provide reliable data governance, stable training, low inference latency, accurate Chinese semantic alignment, and an independently evolvable hardware and software stack.

    When both training and inference run on the same domestic computing and compiler ecosystem, the model becomes more than a demonstration of visual quality. It becomes an auditable, portable, and maintainable technical asset.

    1. The Core Challenge: Text-to-Image Generation Is a Full-Stack Capability

    Functionally, text-to-image generation must support Chinese and English prompts, multiple styles such as photorealistic and illustrative generation, and integration with broader multimodal workflows, including understanding, prompt rewriting, review, and post-processing.

    However, evaluating only pixel quality can obscure the engineering value of a solution. Many systems are trained on general-purpose accelerators and later adapted for inference on other chips. Running the complete workflow—including data processing, pretraining or fine-tuning, and inference deployment—on a domestic AI hardware and software stack requires all of the following to work together:

    • Operator coverage
    • Communication topology
    • Mixed-precision execution
    • Checkpoint recovery
    • Compiler optimization

    Performance must also reach a level suitable for practical use. This changes the problem from a competition over model architectures into a competition over end-to-end system capabilities.

    2. Key Components of End-to-End Training on Domestic Hardware

    2.1 Data and Preprocessing

    Image-text alignment datasets require deduplication, quality filtering, safety review, and normalization of Chinese descriptions. The preprocessing pipeline must also maintain stable throughput on a domestic cluster, covering image decoding, resizing, augmentation, sharded storage, and integrity checks.

    A useful practice is to make data processing reproducible and replayable. Record file hashes and dataset versions so that data alignment does not change unexpectedly when the workload is moved to a different set of machines.

    2.2 Training Stability

    Diffusion-based and autoregressive generation models are sensitive to memory capacity, communication efficiency, and numerical precision. Important areas to monitor include:

    • Gradient clipping and loss spikes
    • Pipeline bubbles introduced by ZeRO-style sharding and pipeline parallelism
    • Checkpoint consistency
    • Precision differences between compiled operators and handwritten kernels

    In practice, fix the random seeds, record the loss at every step, and save periodic samples. Build a training health dashboard instead of relying only on final metrics such as FID.

    2.3 Compiler Optimization and Operator Gaps

    New architectures frequently expose unsupported operators, particularly in attention variants, conditional injection, and VAE encoding or decoding.

    A practical strategy is to:

    1. Rewrite the implementation using operators that have already been validated.
    2. Fuse operations in performance-critical paths.
    3. Use a small reproducibility set to compare logits and pixel-level errors against a reference implementation.
    4. Increase the batch size only after the small-scale comparison is consistent.

    This process helps separate compiler or operator issues from model and data issues before scaling up training.

    2.4 Inference and Service Deployment

    Text-to-image latency is affected by the number of denoising steps, image resolution, and batch size. The deployment layer should standardize the following components:

    • Request scheduling
    • Timeouts and retries
    • Watermarking and content safety
    • Integration with object storage
    • Logging, monitoring, and alerting

    On a domestic software and hardware stack, dynamic-shape execution and fallback paths for failed graph capture require additional validation. These paths are essential for preventing isolated compiler or runtime failures from affecting the entire service.

    3. Capability Boundaries and Prompt Engineering

    Chinese prompts are particularly sensitive to spatial relationships, quantities, and style-related terms. A structured prompt should generally specify:

    • Subject
    • Scene
    • Composition
    • Lighting
    • Style
    • Negative requirements

    For example, a diagram in the style of a children’s science book should explicitly describe features such as a cross-sectional view, labels, simple color blocks, and the absence of realistic skin texture. A cinematic still should instead emphasize focal length, depth of field, and material properties.

    Evaluation should not rely on a single image. For the same prompt, generate samples with multiple seeds and different step counts. Check whether the model fails on:

    • Text rendering
    • Fingers and hands
    • Symmetrical objects
    • Counting small objects

    When photorealistic and illustrative characteristics are combined, negative prompts can help isolate conflicting features such as photographic noise and oil-paint brushwork.

    4. How to Evaluate a Genuine SOTA-Level Experience

    Visual appeal is not the same as a meaningful metric. A practical evaluation framework should be divided into several layers:

    • Semantic alignment: Does the subject appear, are its attributes correct, and does the model understand Chinese proper nouns?
    • Structural correctness: Are perspective, occlusion, and text legibility accurate?
    • Style controllability: Does changing a style term remain stable under the same seed?
    • Engineering performance: What are the per-image latency, concurrency, failure rate, and peak memory usage?

    Build an internal evaluation set using brand-independent scenarios, such as technical diagrams, portraits, product sketches, and poster layouts. Keep sampling configurations fixed to support regression testing. Carefully selected images from social media should not be used as a substitute for a repeatable evaluation set.

    5. Deployment Recommendations and Risk Control

    Image generation should be treated as one stage in a broader pipeline:

    Prompt rewriting with a multimodal understanding model → image generation → super-resolution or restoration → safety review → storage

    Content safety and copyright compliance must be addressed early rather than inferred from whether an image merely appears acceptable.

    For critical use cases such as educational diagrams and industrial sketches, constraint templates and post-processing validation can help reduce hallucinations. OCR and object detection can be used to verify generated text and required objects.

    To manage cost, apply resolution tiers and adaptive step counts. Frequently used templates can also cache intermediate features. At the team level, maintain a failure-case library and turn recurring problems—such as malformed hands, unreadable text, and incorrect multi-object counting—into continuous regression tests.

    6. Conclusion

    Training and deploying a usable text-to-image model entirely on domestic AI chips and their supporting software stack is valuable not simply because it produces another image-generation website. Its main benefits are controllability, reproducibility, and the ability to build and maintain the system independently.

    Developers should focus on dataset versioning, training observability, complete operator coverage, and the construction of reliable evaluation sets. A strong visual result becomes production capability only when it is backed by reproducible configurations, monitoring, and a complete engineering workflow.