Full-Stack Text-to-Image Model Training on Domestic AI Chips: Engineering Practices

    This article explains what full-stack training on domestic AI chips means, why training text-to-image models is harder than serving them, and how engineering teams can adapt operators, compilation, numerical precision, parallelism, memory, long-running training, data pipelines, evaluation, inference, and cloud operations. It includes practical checks, code examples, rollout steps, and troubleshooting guidance.

    Full-Stack Text-to-Image Model Training on Domestic AI Chips: Engineering Practices

    1. First, clarify what counts as “full-stack domestic-chip training”

    In industry discussions, people often say a model “runs on domestic chips.” In practice, this claim has at least three layers of meaning, and the engineering effort behind them is very different.

    • Layer 1: Inference migration. The model is trained on a general-purpose accelerator cluster, and the weights are then moved to domestic accelerators for serving. This path mainly tests operator coverage and framework adaptation on the inference side, and the work is often measured in weeks.
    • Layer 2: Training availability. Domestic chips can run the full training task. The loss can converge normally, and samples can be generated, but the setup may rely on many operator fallbacks or partially fall back to host-side computation. Scaling efficiency may still show a clear gap.
    • Layer 3: Full-stack training. Data processing, model definition and compilation, distributed parallelism, collective communication, operator implementation, precision strategy, fault recovery, evaluation, and deployment all form a closed loop on domestic hardware and software. The final generation quality and training efficiency can be comparable to mainstream industry levels.

    Why is training much harder than inference? It can be understood from four dimensions:

    1. Operator scale doubles. Inference has only one forward pass. Training includes forward, backward, and optimizer update. It must save activations, build backward computation graphs, and maintain optimizer states such as momentum. Modules such as variational autoencoders, noise schedules, and attention variants all need high-quality backward implementations. If any one part lacks support, the whole training path may fail.
    2. Errors are amplified. In inference, a small error from a single operator usually only affects the current output. During training, every step’s gradient carries tiny deviations, and these deviations accumulate over tens of thousands of iterations. The final result may be slower convergence or even divergence.
    3. Shapes are highly dynamic. Text-to-image training commonly uses multiple resolutions and aspect ratios. Sequence length changes roughly with the square of resolution, which puts pressure on static graph compilation and operator fusion.
    4. Communication patterns are complex. Data parallel gradient synchronization, tensor or sequence parallel activation splitting and aggregation, and pipeline parallel point-to-point transfer between adjacent stages all depend on the collective communication library and cluster interconnect quality. The maturity of domestic chip clusters in this area varies, and this often becomes the first scaling bottleneck.

    Therefore, the difficulty of “full-stack domestic-chip training” is not a single operator, but the coordination of operators, compilers, communication libraries, frameworks, and training strategies. It is more accurate to view it as joint systems engineering than as a simple chip replacement.

    2. Four key areas for training adaptation

    2.1 Operators and graph compilation: eliminate implicit fallbacks first

    A common situation when adapting to new hardware platforms is “it runs, but it is very slow.” The cause is often not the model itself, but a number of operators that lack native implementations. At runtime, they automatically fall back to host-side execution or low-efficiency implementations, and each call involves data movement.

    A practical investigation process can be fixed into three steps:

    1. Full scan. Run several training steps with the platform’s profiling tools and export operator timing and fallback lists.
    2. Triage by priority. Rank fallback operators by time share. Prioritize writing or auto-tuning operators with high time share. Operators with low time share and low call frequency can be accepted temporarily.
    3. Build a gate. Add “no fallback for critical operators” to continuous integration checks so that later iterations do not silently introduce new fallback paths.
    1. # Illustration: check operator fallbacks and hotspots before training
    2. report = profiler.profile(train_step, warmup=3, steps=10)
    3. fallback = [op for op in report.ops if op.is_fallback] # Operators executed on the host
    4. hot = report.top_by("self_time", k=20) # Operators with the highest self time
    5. print(fallback)
    6. for op in hot:
    7. print(f"{op.name:40s} {op.share:.1%}")

    Graph compilation also matters. Full-graph mode can reduce memory access and scheduling overhead through operator fusion, but once an unsupported shape or operator appears, the whole graph may fall back. A more stable approach is partitioned compilation: let the core backbone, such as the diffusion Transformer backbone and attention modules, use full-graph optimization, while keeping peripheral data preprocessing and sampling scheduling dynamic.

    2.2 Numerical precision: turn “looks the same” into “provably the same”

    1. Layer-by-layer numerical comparison. Use the same weights and the same batch, run the reference implementation and the domestic-stack implementation, compare outputs layer by layer, and form a staged check from “operator → module → whole network.”

    1. # Illustration: compare forward outputs layer by layer
    2. def compare_forward(ref_model, cand_model, batch):
    3. with no_grad():
    4. for name, (a, b) in zip_layer_outputs(ref_model, cand_model, batch):
    5. a, b = a.float(), b.float()
    6. max_abs = (a - b).abs().max()
    7. mean_rel = (a - b).abs().mean() / (a.abs().mean() + 1e-6)
    8. flag = "OK" if max_abs < 1e-2 else "CHECK"
    9. print(f"{name:40s} max_abs={max_abs:.4f} mean_rel={mean_rel:.5f} {flag}")

    The acceptance criteria should be scenario-specific:

    • Pure linear layers usually have small relative errors in low precision.
    • Normalization layers and softmax involve many accumulations, so errors are easily amplified.
    • If the error jumps by orders of magnitude at certain layers, the issue can usually be localized to a specific implementation.

    2. Single-step gradient comparison. Under identical random seeds, data order, and learning rate, run one optimizer step on each side and compare loss, global gradient norm, and parameter update magnitude. This confirms that the backward pass and update stage are also consistent.

    1. # Illustration: compare a single training step
    2. def compare_step(ref, cand, batch):
    3. result = {}
    4. for tag, model in (("ref", ref), ("cand", cand)):
    5. loss = model.training_step(batch)
    6. grad_norm = global_grad_norm(model.parameters())
    7. result[tag] = (loss, grad_norm)
    8. print(result)
    9. assert abs(result["ref"][0] - result["cand"][0]) < 1e-3 # Loss deviation
    10. assert abs(result["ref"][1] / result["cand"][1] - 1) < 1e-2 # Gradient norm deviation

    3. Long-horizon convergence comparison. Consistency in one step does not guarantee consistency over tens of thousands of steps. It is recommended to train for several thousand steps on both sides and compare loss curves, gradient norms, learning rate schedules, and final generation quality. In low-precision training, pay special attention to gradient noise caused by insufficient accumulation precision. If necessary, enable loss scaling, or force sensitive modules such as normalization and softmax to run in higher precision.

    4. Reproducibility management. Fix random seeds, data augmentation, and sampling order. Clearly define which operations allow nondeterminism and which must be deterministic. Also archive the hardware version, driver version, and compiler options for each experiment. Otherwise, when results become abnormal, it is difficult to determine whether the issue is caused by the model or the environment.

    2.3 Parallelism and communication: make scaling efficiency measurable

    Single-node multi-card deployment is only the starting point. What truly tests platform maturity is multi-node scaling efficiency. It is useful to build a “parallel strategy comparison table” that clarifies the hardware requirements of each partitioning method:

    Parallel strategy What is partitioned Main communication Key hardware requirements
    Data parallel Batch samples Gradient AllReduce Parameter-plane bandwidth, communication library stability
    Tensor parallel Single-layer weight matrices AllReduce / ReduceScatter High-speed inter-card interconnect, low latency
    Pipeline parallel Model layer segments Point-to-point between adjacent stages Link stability, bubble control
    Sequence parallel Sequence dimension AllGather / ReduceScatter Large-scale collective communication efficiency
    Parameter sharding (optimizer state/gradient sharding) Optimizer state, gradients ReduceScatter + AllGather Balance between memory capacity and communication

    Several practical rules are worth following:

    • Overlap communication and computation. Gradient bucket size and overlap strategy have a large impact on end-to-end throughput. They should be tuned on the target hardware, rather than reusing default values from an old platform.
    • Watch slow nodes. In large-scale training, a few nodes may slow down global synchronization because of link degradation or cooling problems. It is useful to collect percentile statistics for single-step latency, such as P50 and P95, and automatically isolate nodes that consistently exceed thresholds.
    • Pay attention to the partitioning dimension for tensor parallelism. In text-to-image models, text conditions are usually broadcast to spatial dimensions with relatively small sequence lengths. If partitioned along the wrong dimension, additional rearrangement and communication overhead will be introduced. In multi-resolution training, the spatial dimension is not fixed, so sequence parallelism also needs to work with resolution bucketing.

    2.4 Memory and long-term stability: treat one training run as a long-term service

    For diffusion models, training memory pressure mainly comes from four sources: backbone parameters and optimizer states, activations needed for backward propagation, intermediate features from the variational autoencoder encoder/decoder, and temporary usage from the text encoder. Optimization methods include:

    • Activation recomputation: trade about 20% to 30% extra computation for a significant reduction in activation memory;
    • Parameter and state sharding: partition optimizer state across the data parallel group;
    • Freezing and sharding the variational autoencoder: disable gradients during encoding/decoding and limit its peak memory;
    • Operator-level memory reuse: avoid fragmentation caused by repeated allocation and release of temporary tensors.

    Long-term stability also requires engineering safeguards, because a full training run may last for weeks:

    1. Frequent checkpoints. Save checkpoints by both step count and time interval. Use “write to a temporary file first, then atomically rename” to avoid corrupted files if the process is interrupted. Important checkpoints should also be backed up off-site.
    2. Full recovery state. Do not save only weights. Also save optimizer state, learning rate scheduler, random number state, and data sampling position.
    3. Automatic fault tolerance. When a node exits unexpectedly, reschedule automatically and resume training. Failed tasks should be retried automatically and the failure reason should be recorded. Abnormal frequency should be included in alerts.
    1. # Illustration: atomic checkpoint saving and full recovery
    2. def save_checkpoint(step, model, optimizer, scheduler, dataloader):
    3. state = {
    4. "step": step,
    5. "model": model.state_dict(),
    6. "optimizer": optimizer.state_dict(),
    7. "scheduler": scheduler.state_dict(),
    8. "data_state": dataloader.state_dict(), # Ensure the data order is consistent after recovery
    9. "rng": save_rng_state(),
    10. }
    11. tmp = f"ckpt/step_{step:07d}.tmp"
    12. atomic_write(tmp, state)
    13. rename(tmp, f"ckpt/step_{step:07d}.pt") # Atomic replacement to avoid partial files
    14. def resume(path):
    15. state = atomic_load(path)
    16. model.load_state_dict(state["model"])
    17. optimizer.load_state_dict(state["optimizer"])
    18. scheduler.load_state_dict(state["scheduler"])
    19. dataloader.load_state_dict(state["data_state"])
    20. load_rng_state(state["rng"])
    21. return state["step"] # Continue training from this step

    3. Data and training engineering for text-to-image models

    Beyond general training practices, text-to-image models have several engineering points that are easy to underestimate.

    1. Data quality deserves more investment than raw data scale. The upper limit of generation quality is largely determined by image-text pair quality. Noisy captions, style mismatch, low resolution, and duplicate samples can directly push the model in the wrong direction. A typical data pipeline includes:

    • Source compliance and licensing checks;
    • Coarse filtering based on aesthetic scores, sharpness, and watermark detection;
    • Near-duplicate removal;
    • Rewriting or supplementing captions with multimodal models;
    • Distribution statistics by language, domain, and style.

    2. Chinese capability needs dedicated work. Understanding Chinese prompts and rendering Chinese characters, especially complex characters with many strokes, is a high-frequency pain point in Chinese generation scenarios. It needs separate optimization in areas such as Chinese image-text pair ratios, Chinese re-captioning corpora, and layout/glyph supervision data. It should not be assumed that English capability will automatically transfer.

    3. Resolution bucketing and feature precomputation. Training at a single fixed resolution wastes compute and weakens compositional diversity. A more common approach is to bucket samples by several aspect ratios and batch samples within the same bucket. In addition, variational autoencoder encoding results and text encoding results can be precomputed and cached before training. During training, they can be read directly, avoiding repeated forward passes in every training step. The tradeoff is that online image augmentation becomes more difficult, so teams need to balance this based on the task.

    1. # Illustration: resolution bucketing + offline conditional feature caching
    2. BUCKETS = [(1024, 1024), (1152, 896), (896, 1152), (1280, 768), (768, 1280)]
    3. def assign_bucket(w, h):
    4. ratio = w / h
    5. return min(BUCKETS, key=lambda b: abs(b[0] / b[1] - ratio))
    6. def precompute(sample):
    7. latent = vae_encode(sample["image"]) # Run only once
    8. cond = text_encoder(sample["caption"])
    9. key = f"{sample['id']}.npz"
    10. object_store.put(key, save(latent, cond, assign_bucket(*sample["size"])))

    4. The tradeoff between training objective and guidance. Training objectives usually use denoising regression forms such as noise prediction or velocity prediction. At inference time, classifier-free guidance (CFG) amplifies the difference between conditional and unconditional predictions to improve prompt adherence, but it multiplies sampling steps and compute. A common compromise is to train with fewer steps and distill a fast model for the default mode, while keeping a multi-step high-quality mode for professional scenarios.

    4. Evaluation system: turn “looks good” into measurable metrics

    Evaluation of generative models is easily reduced to subjective judgment. It is recommended to build at least three layers:

    • Distribution-level metrics: such as distribution-distance metrics like Fréchet distance and classification accuracy lift. They help judge overall quality and diversity, but they must be tied to a fixed reference set and resolution. Otherwise, results are not comparable.
    • Capability-level breakdowns: prompt adherence, composition reasonableness, object count and relationship correctness, text rendering accuracy, fragile structures such as hands, and style consistency. Each category should have its own test set.
    • Human evaluation: blind tests with multiple reviewers, pairwise comparisons, and win-rate statistics, covering typical scenarios such as photorealistic images, illustrations, design sketches, and poster layouts.

    In addition, a regression gate should be established: every time the model or inference configuration changes, batch generate on a fixed prompt set and set thresholds for key subcategories. This prevents fixing A while breaking B.

    5. From training to inference: extend full-stack optimization to serving

    Finishing training is only the midpoint. To serve the model to business at low cost, there is also room for optimization on the inference side:

    • Step compression: consistency- or distillation-based methods can reduce sampling from dozens of steps to a single digit. This is usually the highest-return single optimization.
    • Low-precision inference: use low-precision weights and operators when quality is acceptable. Evaluate errors layer by layer, with special attention to normalization and attention.
    • Compilation and operator fusion: redo operator fusion and memory planning for the target hardware.
    • Batching and caching: combine requests, deduplicate prompts, and cache hits for identical seeds and parameters.
    • Asynchronous processing and queuing: generation tasks naturally take a long time. Decoupling requests from execution with a task queue, plus priority and timeout policies, is usually more stable than synchronous interfaces.
    1. # Illustration: step-by-step sampling with CFG
    2. def sample_image(prompt, n_steps=28, cfg=5.0, seed=0):
    3. cond = text_encoder(prompt)
    4. uncond = text_encoder("")
    5. z = randn_latent(shape_of(n_steps), generator=seeded_generator(seed))
    6. sigmas = build_sigma_schedule(n_steps)
    7. for i, s in enumerate(sigmas):
    8. z_in = concat([z, z]) # Combine conditional and unconditional into one batch
    9. v = denoiser(z_in, s, cond=concat([cond, uncond]))
    10. v_cond, v_uncond = split(v)
    11. v = v_uncond + cfg * (v_cond - v_uncond) # Classifier-free guidance
    12. z = solver_step(z, v, s, sigmas[i + 1], order=2)
    13. return vae_decode(z)

    6. How general cloud capabilities support these tasks

    Full-stack domestic training does not require enterprises to build their own data centers. General cloud platform capabilities can already cover most of the required stages:

    • Object storage: store image-text datasets, cached latents and conditional features, and training checkpoints;
    • Container platforms: standardize training and inference images, enabling reproducible environments and elastic scaling;
    • Message queues / task queues: decouple generation requests from execution and support queuing and retries for long-running tasks;
    • Logging services: aggregate training logs, operator fallback lists, and inference call chains;
    • Monitoring and alerting: track chip utilization, memory, temperature, communication bandwidth, single-step latency, and task success rate.

    Among these, putting “operator fallback count,” “scaling efficiency,” and “single-step latency percentiles” into a unified monitoring dashboard is one of the biggest operational differences between domestic training environments and ordinary training environments.

    Advancing in stages can keep risk within an acceptable range:

    1. Freeze the baseline: fix model structure, data mixture, and hyperparameters in the existing environment to form a comparison baseline.
    2. Single-node comparison: complete operator fallback cleanup and layer-by-layer/single-step numerical comparisons to ensure “same input, same result.”
    3. Small-cluster scaling: scale from 8 cards to dozens of cards, draw the scaling efficiency curve, and locate communication bottlenecks.
    4. Short convergence validation: train for several thousand steps, compare loss and sampling quality, and confirm there is no long-horizon drift.
    5. Full training and inference optimization: enable long-term stability mechanisms, and start step compression and low-precision inference evaluation in parallel.
    6. Launch and regression: establish metric gates and gradual rollout to form a closed loop.

    8. Quick reference for common issues

    Symptom Common causes Direction to investigate
    Training works but speed is far below expectations Operator fallbacks, communication not overlapped Rank and address the fallback list, tune communication buckets
    Loss is normal early but fluctuates later Low-precision accumulation error, missing gradient clipping Increase precision of sensitive modules, enable loss scaling
    Loss jumps after resuming training Optimizer state or data position not restored Save full state, persist data position
    Scaling returns diminish after reaching a certain card count Collective communication or slow nodes Adjust parallel partitioning, isolate slow nodes
    Specific Chinese prompts produce poor quality Insufficient Chinese corpus ratio Add Chinese re-captioning and glyph supervision data

    9. Conclusion

    The answer to “Can domestic chips train text-to-image models?” does not depend on a single demo image. It depends on whether five metrics can be sustained: operator coverage, numerical consistency, scaling efficiency, long-term stability, and inference cost. By breaking adaptation work into measurable, regression-testable engineering processes, and combining them with general cloud capabilities such as storage, containers, queues, and monitoring, most teams can make the transition from inference migration to full-stack training at a controlled cost.