A Practical Guide to Training and Deploying a Multimodal Generation Model Entirely on Domestic AI Chips

    This article provides an engineering-focused analysis of a multimodal generation model trained entirely on domestic AI chips. It covers how to evaluate model behavior, adapt operators and mixed-precision training, run distributed and fault-tolerant pretraining, build a reproducible inference pipeline, fine-tune with low-rank adapters, set up quantitative and human evaluations, address compliance risks, and harden the service with cloud infrastructure. The goal is to help teams decide whether such a model is usable in real business scenarios and how to maintain a stable path from training to production.

    1. First, Clarify What You Are Evaluating

    When evaluating an open-source multimodal generation model, the easiest mistake is to treat it as a black box that takes a prompt and outputs an image. Technically, such a model usually combines three parts: a text encoder maps natural language into a semantic space; the denoising backbone iteratively removes noise in the latent space; and the decoder reconstructs the latent representation into pixel images. The denoising backbone determines the upper bound of generation quality, the text encoder affects semantic alignment, and the decoder influences detail clarity and inference cost.

    Therefore, the question “how good is the model in practice” should be broken into more specific engineering questions:

    • Semantic alignment: complex sentences, multiple subjects, spatial relations (“on the left”, “behind”), and attribute binding (“red hat”).
    • Controllability: support for image size and aspect ratio, reproducible seeds, reference-image conditioning, and instruction-based editing.
    • Localization: understanding of Chinese prompts and accuracy of rendered text in images.
    • Engineering usability: memory usage, single-card throughput, batch efficiency, and whether training and inference can run stably on domestic accelerators.

    Only by breaking the experience into these dimensions can teams judge whether a model is actually usable in their own scenarios.

    2. What “Trained Entirely on Domestic AI Chips” Means in Engineering

    A model pre-trained from scratch on domestic AI accelerators has more significance than merely “running once”. It means the full toolchain has been tested under real conditions, which is directly relevant to teams seeking controllable compute. Four challenges are especially important.

    Operator Coverage and Numerical Alignment

    Mainstream diffusion models use many operators, including convolution, group normalization, attention, interpolated upsampling, and sinusoidal positional encodings. Heterogeneous backends often have missing operators, different numerical implementations, or performance degradation. A practical approach is layer-by-layer numerical alignment: use the same weights and inputs, run forward passes on the reference backend and the target backend, compare the relative error of each layer’s output tensor, locate the layers where errors grow, and reduce the gap through custom operators, equivalent replacements, or higher-precision computation in those layers.

    Numerical Stability and Mixed-Precision Strategy

    Diffusion training is sensitive to numerical ranges. Some attention and normalization steps can overflow or produce abnormal gradients under pure half precision. A common engineering combination is to use low precision for the main computation to improve throughput, keep single precision for critical reductions and loss calculation, and manage optimizer states and gradient accumulation with separate precision policies.

    Distributed Training and Fault Tolerance

    Pre-training an image generation model with more than one billion parameters usually requires a hybrid strategy of data parallelism, tensor parallelism, pipeline parallelism, and memory optimizations such as optimizer-state sharding. Gradient accumulation is also used to scale the global batch size to a level suitable for stable training. Periodic checkpointing and resumable training are essential. Otherwise, a single communication hiccup or node failure can waste a large amount of compute.

    Communication Libraries and Collective Communication Tuning

    When cross-device communication becomes the bottleneck, the biggest gains often come from changing the parallelism layout, overlapping communication with computation, and improving topology-aware collective communication, rather than simply adding more nodes.

    3. Inference: How to Run It in Practice

    Once you have the weights, the first step is not to write business code, but to establish a minimal reproducible inference pipeline.

    1. # Illustrative pseudocode. Use the actual framework interfaces.
    2. backend = detect_accelerator() # Automatically detect the available accelerator type
    3. dev = build_device(backend) # Unified device abstraction
    4. set_default_dtype("bfloat16") # Set the default precision
    5. pipe = load_pipeline(model_dir, device=dev, dtype=DTYPE)
    6. if oom_risk(pipe):
    7. pipe.enable_memory_saving("sequential_offload") # Move less active stages when memory is tight
    8. image = pipe(
    9. prompt=USER_PROMPT,
    10. negative_prompt=DEFAULT_NEGATIVE,
    11. height=H, width=W,
    12. num_inference_steps=STEPS,
    13. guidance_scale=CFG,
    14. generator=make_generator(seed=SEED), # Fixed seed for reproducibility
    15. )

    In practice, four parameter groups need careful tuning:

    Parameter Role Tuning advice
    Sampling steps Number of denoising iterations Increase moderately for quality-sensitive cases; for high-concurrency scenarios, consider distilled models
    Guidance strength How strongly the prompt is enforced Too high causes oversaturation and artifacts; too low causes semantic drift
    Resolution and aspect ratio Composition and memory overhead Memory and latency grow non-linearly; generate in tiers
    Random seed Reproducibility Business systems should record it for troubleshooting and A/B comparisons

    Memory optimization is usually ordered from lowest to highest cost: low-precision weights, decoder tiling, attention-memory reuse, and staged offloading of inactive modules to host memory. The first two can be enabled first because they usually cause little or no quality loss.

    Throughput optimization can come from static graph compilation and operator fusion, dynamic batching that merges multiple requests into one denoising iteration, and quantized inference. Note that quantization can affect image quality more noticeably than in classification models. Always compare before and after using a fixed evaluation set instead of deploying based on intuition alone.

    4. Customization: Low-Rank Adapter Fine-Tuning

    A general model often cannot cover enterprise brand visual standards and domain-specific styles. Full-parameter fine-tuning is expensive and prone to catastrophic forgetting, so low-rank adaptation is more practical. Freeze the text encoder and decoder, and inject trainable low-rank matrices only into the attention and feed-forward layers of the denoising backbone.

    1. # Illustrative pseudocode
    2. model = load_base_model(base_dir)
    3. freeze(model.text_encoder)
    4. freeze(model.vae)
    5. inject_low_rank_adapters(
    6. model.denoise_net, rank=16, alpha=16,
    7. targets=["attn_qkv", "attn_out", "ffn"]
    8. )
    9. train_one_step(
    10. model, dataloader=build_dataloader(image_caption_pairs),
    11. lr=1e-4, loss="flow_matching_or_denoising", precision="bf16"
    12. )

    Three data-side lessons:

    • Image-text pairs must be cleaned. Samples where the caption does not match the image should be removed; wrong labels directly damage fine-tuning results.
    • Scale and consistency matter more than raw quantity. A few hundred style-consistent, carefully curated images often outperform tens of thousands of messy images.
    • Preserve general capability. Mixing a certain proportion of general samples into domain data helps mitigate style collapse.

    5. Evaluation: Replace “Feels Good” with Data

    Without objective evaluation, model iteration degenerates into subjective debate. A three-layer evaluation system is recommended:

    1. Automated metrics: use distribution-distance metrics to measure how close generated images are to real images statistically, and use image-text similarity models to measure semantic consistency between prompts and outputs. These metrics are suitable for tracking changes in the same model over time, but should not be used directly to compare different models.
    2. Structured test sets: create 50–100 fixed prompts for each category, including attribute binding, quantity relations, spatial relations, text rendering, hand and human-body structure, and complex materials. Run the full set after every change to maintain a comparable baseline.
    3. Human preference evaluation: use randomized blind tests with two independent raters scoring image quality, semantic alignment, and usability, then calculate an inter-rater agreement coefficient as a final decision input.

    Inference performance should also be observed independently: first-token latency, end-to-end latency per image, peak accelerator memory usage, and queue length under concurrency. These metrics should be connected to logging and alerting services to create a dashboard that covers both quality and performance.

    6. Compliance and Safety Before Launch

    Risks in image generation businesses concentrate in three areas: copyright and licensing status of training corpora; violation risks in generated content, including portraits, violence, and content involving minors; and trust issues caused by deepfakes.

    Engineering should provide: bidirectional filtering of prompts and negative prompts; multi-level review of generated results, including sensitive-feature detection and human sampling; implicit watermarks and metadata provenance marks; and auditable retention of requests and outputs. Retention policies must align with data compliance requirements, clearly defining retention periods, masking rules, and access permissions.

    7. Cloud Landing: Use General Capabilities to Fill Engineering Gaps

    Getting the model running is only the starting point. In the service stage, teams can use general cloud capabilities to reduce operational burden: object storage for model weights and generated images, container platforms for elastic scaling and canary releases of inference services, message queues for peak shaving and asynchronous image generation, and logging plus monitoring and alerting for observability. For teams using Baidu AI Cloud, its general cloud infrastructure can cover these needs, so teams do not have to build a complete operations stack themselves. Note that versions, drivers, and compilation toolchains for the heterogeneous accelerator adaptation layer should be pinned in images to avoid environment drift that leads to “works locally, fails online”.

    8. Summary: An Actionable Rollout Checklist

    For teams planning to build image-generation capabilities on domestic compute, proceed in this order:

    • Verify end-to-end: with fixed weights, fixed prompts, and fixed seeds, first get a reproducible minimal inference pipeline working.
    • Evaluate cost and benefit: define tolerance for latency and cost, then decide sampling steps and quantization strategy.
    • Establish baselines: build structured test sets and automated evaluation scripts, and run every change against the baseline.
    • Fine-tune only when needed: introduce low-rank adaptation fine-tuning only when general capabilities cannot meet visual standards.
    • Harden for production: build content safety, watermark provenance, monitoring and alerting, and elastic scaling together.
    • Keep aligning upstream: follow updates to operators and compilation toolchains, and regularly regression-test numerical consistency and performance.

    The real value of “trained entirely on domestic AI chips” is not a leaderboard ranking. It is that the full chain, from operator adaptation and distributed training to inference deployment, can be closed on self-controlled compute. For engineering teams, turning this experience into their own adaptation layer, evaluation sets, and operations standards is the sustainable technical asset.