Training Text-to-Image Models End-to-End on Domestic AI Chips: An Engineering Practice

    A practical guide to building a text-to-image model across the full domestic AI stack—data processing, training, and inference. It explains what "full-stack localization" actually means, why getting a model to run is different from getting it to run efficiently, and how to evaluate an alternative-hardware solution using end-to-end training time, scaling efficiency, and convergence consistency rather than single-card throughput. You'll find concrete checklists for operator coverage, mixed precision, loss scaling, and communication overhead, a troubleshooting table for common failure modes, and advice on baselining, layered verification, and observability—plus the boundaries that determine when the effort is worthwhile.

    1. What “full-stack localization” really means

    Text-to-image (T2I) models do one thing directly: given a Chinese or English prompt, they output images in styles ranging from photorealistic to illustration to design sketches. Judging them only by “does the output look good” misses the engineering work happening behind the scenes.

    The common industry path is to train on mature general-purpose accelerators first, then migrate to domestic chips for inference. This “train elsewhere, infer locally” pattern only makes the deployment stage self-reliant. The training stage—the most compute-intensive part and the one most dependent on the low-level software ecosystem—still runs on someone else’s foundation. Full-stack localization means that data processing, model training, and inference deployment all run on a domestic AI software/hardware stack. The goal is then to specify the model parameter count, data version, hardware model, software stack version, and evaluation metrics, and use measured results to judge whether the approach comes close to mainstream industry levels. The point here is not to assume it “will definitely match,” but to define a reproducible verification baseline.

    Around this topic, developers tend to ask: How do you actually do full-stack localization? Which metrics matter when choosing a domestic card? What pitfalls show up in practice? What should you watch for during migration? This article answers those “how-to, how-to-choose, what-to-avoid” questions layer by layer.

    2. Why “runs” and “runs well” are two different things

    Getting a model to “run” on domestic chips is not hard. What is hard is running the entire training cycle efficiently, stably, and at scale. Several typical engineering barriers stand in the way:

    • Operator coverage: Diffusion models use many custom operators (attention variants, upsampling, timestep embeddings, and so on). When the low-level operator library is missing or incomplete, you have to write or port them by hand, which directly affects both convergence and performance.
    • Framework adaptation: Mainstream training frameworks support domestic hardware backends to varying degrees. Compatibility issues can surface in graph lowering, memory management, automatic differentiation, and elsewhere.
    • Collective communication efficiency: Multi-GPU, multi-node training relies on primitives like AllReduce and AllGather. The maturity of the communication library determines whether scaling efficiency improves roughly linearly as you add cards.
    • Numerical precision alignment: Under mixed precision, low-precision numerical stability behaves differently across hardware, so you need to validate it layer by layer.

    Because of this, evaluating a full-stack localization solution cannot rest on single-card throughput alone. You have to look at three combined metrics: end-to-end training time, scaling efficiency, and convergence consistency.

    3. Breaking down the key stages of full-stack autonomous training

    3.1 Data engineering: the underestimated first gate

    A large model’s success depends heavily on its data. Text-to-image work needs huge volumes of high-quality text-image pairs. Beyond cleaning, deduplication, labeling, and resolution normalization, you also have to handle compliance and quality upfront:

    • Licensing and compliance: Confirm the licensing scope of your data sources. Filter out infringing images, images containing clear human faces/portraits, and samples touching on sensitive content. Keep licensing and cleaning records for auditing.
    • Dedup threshold: Use the cosine similarity of text-image embeddings as the criterion for semantic-level deduplication. A good starting threshold is around 0.92, calibrated on a small sample. Too high and deduplication is incomplete; too low and you delete valid data by mistake.
    • Quality spot checks: Sample by data source in layers (e.g., 100–200 out of every 10,000), and manually verify text-image consistency, sharpness, and labeling accuracy to produce a quantifiable quality report.

    On the engineering side, it helps to decouple preprocessing from storage:

    1. # Example: data pipeline stage responsibilities; all functions must be implemented for your own data format
    2. pipeline = [
    3. decode_and_resize(), # Input: raw image bytes; Output: tensor normalized to a fixed resolution
    4. text_image_filter(), # Input: (image, prompt); Output: boolean mask of samples that are text-image consistent and free of sensitive content
    5. dedup_by_embedding(), # Input: sample embedding matrix; Output: deduplicated sample indices (threshold ~0.92)
    6. tokenize_and_pack(), # Input: text-image pairs; Output: fixed-length packed training tensors
    7. ]
    8. # Persist preprocessing outputs to object storage; stream-read during training to avoid recomputation

    Persisting preprocessing outputs to object storage and streaming them on demand during training significantly reduces repeated computation and I/O jitter.

    3.2 Training: balancing operators, communication, and precision

    Training is the core of full-stack autonomy. In practice, optimize around three points and provide verifiable checks:

    • Operator completion and fusion: For missing operators, prefer native implementations. Use operator fusion (combining normalization + activation + bias addition) to cut down round trips to memory.
    • Distributed parallelism: Combine data parallelism, tensor parallelism, and pipeline parallelism according to memory and bandwidth constraints. When memory is tight, enable gradient checkpointing and optimizer-state sharding.
    • Mixed precision and stability: Decide explicitly between BF16 and FP16. BF16 has a wider dynamic range and is usually more stable; FP16 needs loss scaling to work well.

    A checklist you can actually run through:

    • Mixed-precision type: State BF16/FP16 and where it’s enabled;
    • Loss scaling initial value: FP16 commonly starts at 65536, with dynamic scaling governed by whether inf/nan appears;
    • Gradient clipping threshold: Often set to a global norm of 1.0, adjusted based on convergence;
    • Communication time share: The fraction of total step time spent in communication like AllReduce—generally keep it under 20%. A higher share means scaling efficiency is being limited;
    • Throughput and convergence baselines: Record single-card tokens/s, N-card tokens/s, and the number of steps needed to reach the same loss/metric. Measure linearity with scaling efficiency (N-card throughput / single-card throughput / N).

    The training skeleton looks roughly like this (pseudo-code, abstracting away specific hardware interfaces):

    1. for step, batch in enumerate(dataloader):
    2. with autocast(): # Mixed-precision context, specify dtype=BF16/FP16
    3. loss = model(batch)
    4. loss.backward()
    5. clip_grad_norm_(model, 1.0) # Gradient clipping
    6. optimizer.step() # Includes learning-rate scheduling
    7. if step % log_interval == 0:
    8. log_metrics(step, loss) # Hook into logging/monitoring

    The key is that training logic—optimizer state, learning-rate scheduling, gradient clipping—must stay consistent and hardware-independent. Only then can you compare results between the domestic stack and a general-purpose stack and verify that convergence lines up.

    3.3 Inference deployment: from training results to a stable service

    Autonomous inference focuses on graph compilation, operator fusion, quantization, and batch scheduling:

    • Use graph optimization to cut scheduling overhead and raise single-card throughput;
    • Apply INT8 quantization only when the precision loss is controlled: fix a calibration set that covers typical prompts/styles (e.g., 500–1000 samples), evaluate the deviation before and after quantization on metrics like CLIP similarity and FID, and set an acceptable ceiling (e.g., FID rising by no more than 5% relative to FP16);
    • Define a fallback strategy: when quantization precision exceeds the threshold or an anomaly triggers, automatically fall back to the FP16 version to keep the service available;
    • Combine dynamic batching with cache reuse to improve concurrency, and monitor three things: concurrency, P99 latency, and throughput.

    For serving, you can reuse common cloud capabilities like container platforms, logging, and monitoring/alerting, packaging the model as an elastically scalable service rather than binding it to proprietary interfaces.

    Troubleshooting table for four typical problems

    Problem Typical symptom Verification method Suggested handling
    Missing operator Unsupported-operator error or abnormal results Compare single-operator output against the general-purpose stack Native implementation or porting; lock it in with unit tests
    Communication bottleneck Multi-card throughput doesn’t scale linearly with card count Measure communication time share and scaling efficiency Optimize the communication library, adjust parallel partitioning, overlap compute and communication
    Precision drift Sudden loss spikes, nan/inf appearing Monitor numerics layer by layer, compare against baseline loss Check loss scaling and gradient clipping; switch to BF16 if needed
    High inference latency High P99 latency, low throughput Load-test concurrency/latency/throughput Graph optimization, quantization, tune dynamic batching

    4. Practical advice for developers

    1. Baseline first, then optimize: Lock in a baseline on a mature stack. The baseline checklist should include a fixed random seed, a fixed data version number, recorded training steps and loss curves, evaluation metrics, and a complete reproduction config (hardware model, software stack version, hyperparameters).
    2. Verify in layers: Validate correctness and scaling efficiency layer by layer—single operator → single card → single machine multi-card → multi-machine multi-card—so problems don’t only surface at very large scale.
    3. Build observability: Feed training metrics, memory usage, and communication time into logging and monitoring to form a traceable tuning loop.
    4. Iterate hardware and software together: Operator and framework issues should feed back into the low-level software stack, forming a positive loop of “discover—fix—lock in.”

    Common mistakes to avoid: Don’t look only at single-card throughput while ignoring scaling efficiency; don’t skip numerical alignment and compare results directly; don’t overlook multi-node communication and storage I/O jitter; don’t turn on quantization without a calibration set and fallback strategy.

    5. Applicability and conclusion

    To be clear, the approach in this article suits teams that already have some training scale and the ability to adapt at the low level. If you just want to quickly validate a demo idea, getting inference running on a mature stack is usually more efficient, and there’s no need to invest in a full-stack localization overhaul from the start.

    The value of full-stack localized training is not in producing yet another text-to-image tool, but in verifying that a domestic AI software/hardware stack can support end-to-end multimodal large-model training. For developers and enterprises, the focus should shift from “comparing output quality” to “assessing stack maturity”: Is operator coverage complete? Is scaling efficiency linear? Is numerics stable? Is deployment elastic? Understanding these engineering boundaries and verification standards is what lets you make rational technology-selection and investment decisions on a self-reliance path.