Training a Text-to-Image Model Entirely on Domestic Chips: What Is the Real-World Experience?

    Text-to-image models trained entirely on domestically developed hardware are attracting attention, but sample images alone cannot determine whether they are ready for production. This article examines the technical challenges of full-stack training, presents a reproducible evaluation framework, and discusses the engineering practices required to deploy these models as reliable image-generation services. It also explains what “entirely domestic” actually means and where current limitations are most likely to appear.

    Open-source image-generation models are being released at a rapid pace. Against this backdrop, a model that is “trained entirely on domestic chips while reaching mainstream performance” naturally deserves attention. The question that follows is usually straightforward: how does it perform in real-world use?

    A few sample images are not enough to provide a reliable answer. Text-to-image quality depends on training data, conditioning alignment, and sampling schedules. The significance of training entirely on domestic hardware also goes far beyond replacing one type of accelerator with another.

    This article sets aside branding claims and examines three practical areas: the real barriers to full-stack training, reproducible methods for evaluating user experience, and the engineering work required to deploy inference in production. The goal is to determine whether this class of model can move beyond demonstrations and support real applications.

    1. What Determines Image Quality? Generation Quality Is Not the Same as Model Quality

    Before evaluating a text-to-image model, it is important to understand the independent components that influence its output. Separating these components is essential for fair comparison.

    • Text encoding and multimodal alignment: How the model interprets prompts determines its ability to follow semantic instructions. Counting objects, such as “three white pigeons,” spatial relationships, such as “a backpack to the left of the cup,” and attribute binding, such as “a boy wearing a red hat,” are particularly prone to errors.
    • Generation backbone: Diffusion models generate images through iterative denoising, while autoregressive models generate images piece by piece using discrete visual tokens. They make different trade-offs in speed, controllability, and detail.
    • Sampling scheduler and guidance strength: Changing the sampler, denoising steps, or prompt-guidance scale can produce a larger visual difference than switching between two different models.
    • Training-data distribution: Coverage of English and Chinese prompts, image quality, and aesthetic filtering directly affect the consistency of the generated style.
    • Post-processing capabilities: Resolution and aspect-ratio support, super-resolution, and local inpainting are engineering features, but they have a major impact on the production experience.

    Therefore, comparisons made without fixing the prompt, sampling parameters, and random seed are difficult to reproduce and offer limited value.

    2. What Does “Trained Entirely on Domestic Hardware” Actually Mean?

    The phrase “entirely domestic” can refer to different levels of the technology stack. These levels should be distinguished in technical discussions:

    • Level 1: Inference only on domestic chips. Training still depends on overseas hardware and software ecosystems. This is currently the most common arrangement and is also the one most likely to be misunderstood as full-stack localization.
    • Level 2: Fine-tuning or distillation on domestic chips. The backbone model was trained using overseas hardware, while domestic hardware is used only for part of the post-training process.
    • Level 3: Pretraining on domestic hardware. Pretraining is completed on domestic accelerators, but operators, parallelism strategies, and training frameworks still require deep adaptation to the domestic software stack.
    • Level 4: End-to-end domestic training and deployment. Data cleaning and filtering, image-text feature extraction, pretraining, post-training alignment, evaluation, and inference deployment are all completed within a domestic hardware and software stack.

    Only Level 4 represents a genuinely end-to-end domestic workflow. Its main value lies in completing long-running pretraining, not in performing a single inference request. The greatest challenges are concentrated in the software stack required to support that process.

    3. Why Is a Full-Stack Closed Loop Difficult?

    3.1 Operator Coverage and Numerical Consistency

    Operators such as attention and normalization can be highly sensitive to numerical precision in diffusion models. When an operator lacks a native implementation, it may need to be assembled from multiple basic operators. This can lead to three problems:

    • Higher peak memory usage
    • Lower execution speed
    • Greater accumulated numerical error

    When low-precision accumulation is used as the foundation of training, errors may become amplified during the later stages and appear as abnormal loss values. These issues can also be difficult to reproduce.

    A safer approach is to first align operator-level numerical errors on a single device before enabling multi-device parallelism.

    3.2 Parallelism Strategies and Collective Communication

    Training generally requires a combination of data parallelism, tensor or pipeline parallelism, and activation recomputation. The text encoder and diffusion backbone can differ significantly in parameter count and computational demand, so their partitioning strategies may need to be designed separately.

    The bandwidth available for collective communication primitives such as all-reduce and all-gather varies across device interconnect topologies. That difference directly affects whether the cluster can achieve high accelerator utilization.

    3.3 The Data Pipeline

    The data pipeline for image generation is often more demanding than that of language models:

    • Image-text features may need to be extracted and cached in advance.
    • Variable resolutions require bucketing or unified resizing.
    • Deduplication, aesthetic scoring, and content filtering must operate as a complete pipeline.

    If data delivery is slower than training, the accelerators will spend time waiting rather than computing.

    3.4 Stability During Long-Running Training

    Pretraining can run for weeks. On clusters with thousands of devices or more, individual failures are almost inevitable. Checkpoint recovery, slow-node detection, and automated fault recovery are therefore foundational infrastructure rather than optional features.

    4. How to Evaluate Real-World Experience: Build a Reproducible Evaluation Matrix

    The question “How does it perform in practice?” must be answered through reproducible evaluation rather than a small collection of sample images. A useful evaluation should cover the following six dimensions:

    1. Semantic adherence: Check whether the model correctly handles object counts, attribute binding, and spatial relationships.
    2. Chinese text rendering: For posters, signs, and similar use cases, evaluate whether Chinese characters are clear and properly formed. Chinese character rendering is often more demanding than English text rendering because it depends heavily on training-data coverage.
    3. Composition and aesthetics: Assess whether the subject is prominent, perspective is reasonable, and lighting appears natural.
    4. Consistency: Determine whether the same character maintains a stable appearance across different scenes.
    5. Style reproduction: Check whether the requested style, such as photorealism, illustration, or line-art sketching, is followed accurately.
    6. Content safety: Evaluate whether the model generates sensitive or potentially infringing content, and whether watermarks and compliance labels can be added effectively.

    The key to an evaluation script is to fix the parameters and random seeds:

    1. # Example code: fixed seeds and sampling parameters for reproducible evaluation
    2. PROMPTS = [
    3. "An orange cat sitting on a wooden windowsill while it rains outside, cinematic lighting",
    4. "A dark blue technology-themed poster with the Chinese title “云端训练” clearly readable at the top",
    5. # ... organize the remaining prompts by evaluation dimension
    6. ]
    7. def eval_one(gen, prompt, seed=20260101, steps=30, guidance=6.0):
    8. gen.set_seed(seed)
    9. # Generate each prompt with multiple fixed seeds to avoid bias from a single result
    10. results = []
    11. for s in range(5):
    12. img = gen(prompt, seed=seed + s, num_steps=steps, guidance_scale=guidance)
    13. scores = {
    14. "semantic": semantic_aligner.score(prompt, img), # Text-image alignment score
    15. "text_ocr": ocr_recall(img, expected_text="云端训练"), # Text-rendering recall
    16. "aesthetic": aesthetic_model.predict(img), # Aesthetic score
    17. "latency_s": gen.last_latency,
    18. }
    19. results.append((img, scores))
    20. return results

    Automated metrics should be used only for initial filtering. Text-image alignment scores may not be sensitive enough to capture fine-grained semantic errors, while image-quality metrics can be strongly affected by resolution and composition. Final conclusions should combine multi-person preference testing or pairwise comparisons with dimension-specific scoring.

    Performance must also be quantified. End-to-end image-generation latency consists of three stages: text encoding, iterative denoising, and image decoding. The decoding stage is often overlooked. Measurements should include single-device throughput, peak memory usage, and time to first image under different resolutions, step counts, and guidance strengths. These metrics are more informative than sample images when determining whether a model is suitable for production.

    5. Production Inference: Turning Image Generation into a Reliable Service

    A usable model is only the starting point. Engineering determines the upper limit of user experience and controls operating cost. Key considerations include the following:

    • Memory usage and quantization: BF16 or FP16 weights are common baselines. Low-bit quantization can significantly reduce memory and bandwidth pressure, but different quantization methods can affect edge details and text rendering in different ways. They should be validated on the evaluation set before deployment.
    • Cache reuse: Cache text-encoding results by prompt hash. When prompts share the same prefix, the corresponding prefix encoding can potentially be reused.
    • Batching and graph optimization: Dynamic batching combines concurrent requests for execution. Continuous batching allows new requests to enter between batch iterations, improving device utilization. Graph optimization and memory reuse in the inference framework can further reduce decoding latency.
    • Asynchronous long-running jobs: High-resolution and batch-generation tasks should be decoupled through a message queue to prevent synchronous API timeouts. Queue depth can serve as a signal for automatic scaling.
    • Service and storage: Run model packages on a container platform, and store weights and generated artifacts in object storage with on-demand mounting and traceable versions.
    • Observability: Record each request’s prompt, seed, step count, and latency in the logging service so that issues can be reproduced. Monitoring and alerting should cover queue backlog, memory utilization, timeout rates, and failure rates.
    • Prompt enhancement: Large-model platforms such as Baidu AI Cloud Qianfan can structure and expand a user’s description, extract the subject, style, and parameters, and then pass the refined prompt to the image model. This is often more stable than sending a short, unstructured description directly to the image model. Specific capabilities and service terms should be verified in the official documentation.
    • Scheduled batch generation: Separating overnight offline generation from daytime online inference is an effective way to control costs.

    6. Pre-Launch Checklist

    • Parameter versioning: Lock the model version, sampler, step count, guidance scale, and random-seed strategy.
    • Style assets: When brand-consistent styles are required, consider parameter-efficient fine-tuning methods such as low-rank adaptation. Use a small set of properly authorized materials for lightweight adaptation, and separately verify its impact on Chinese text rendering.
    • Compliance review: Training data should have traceable authorization. Generated content must comply with applicable requirements for labeling, watermarking, and content review. Review both prompt inputs and image outputs, and retain generation parameters for later auditing.
    • Fallback planning: Define primary and backup models as well as timeout and retry strategies to prevent service interruptions caused by a single dependency.
    • Continuous evaluation: Maintain a regression evaluation set and rerun it after every model or inference-stack upgrade to prevent quality and performance regressions.

    Conclusion

    From a technical perspective, the scarce capability is not whether a single generated image looks impressive. It is the ability to complete a full-stack loop covering data cleaning, image-text feature extraction, pretraining, post-training, evaluation, and inference deployment.

    Reasonable expectations for the current user experience are as follows:

    • Common photorealistic and illustration scenarios can be handled consistently.
    • Complex semantics in long Chinese prompts and small Chinese text rendering remain common weaknesses.
    • The trade-off between inference cost and image quality must be tuned for each business scenario.

    For developers, the right starting point is to establish an evaluation baseline with fixed seeds and dimension-specific scoring before debating which model is better. Only then can conclusions be reproducible and transferable. Models trained entirely on domestic hardware and software stacks also provide an additional deployment path for image-generation applications built on those ecosystems.