Text-to-Image Generation on Domestically Produced Chips: Technical Workflow and Evaluation Methods

    Text-to-image models are often judged by sample outputs, but production use requires validating the full training-to-deployment loop: stable training, checkpointing, evaluation, and serving on domestic hardware and software. This article outlines the basic structure of text-to-image tasks, common chip adaptation issues such as operator accuracy, memory pressure, and multi-card communication, plus data compliance and a practical image-quality evaluation protocol. It also provides a deployment checklist, monitoring thresholds, and five criteria for deciding whether a domestic chip solution is usable.

    1. Define the Evaluation Target First

    The value of a text-to-image model is not reflected only in sample images. Teams need to determine three things: whether training can run stably to completion, whether existing resources can support continuous iteration, and whether the service can remain reliable after launch. If only inference adaptation is completed, it shows only that the hardware can run the model. Whether training, checkpoint saving, evaluation, and deployment can form a closed loop in the same domestic hardware and software environment must be verified separately. Therefore, this article divides evaluation into two parts: whether the pipeline is connected, and whether generation quality meets the bar.

    2. Basic Structure of a Text-to-Image Task

    The model input is a natural-language prompt, and the output is one or more images. Prompts often include subject, count, position, style, lighting, material, and aspect ratio. A usable system usually includes the following stages.

    Text encoding: Convert the prompt into a semantic representation. “A cat wearing a hat sits by the window” and “A cat by the window is wearing a hat” use similar words, but the correspondence between subject and attribute is different. The encoder needs to distinguish the two.

    Condition alignment: Map concepts in the text to visual representations. When alignment is insufficient, common symptoms include missing prompt elements, incorrect subject count, and mismatched color or material.

    Generation backbone: Most mainstream methods refine images progressively over multiple iterations. Computation is concentrated in matrix operations, normalization, and attention calculation. This part has the highest requirements for chip compute and memory bandwidth.

    Image decoding: Restore the internal representation to pixels. Decoding usually requires less computation than the backbone, but it directly affects clarity and per-image latency.

    3. Three High-Frequency Issues in Domestic Chip Adaptation

    3.1 Operator Adaptation: A Reproducible Troubleshooting Process

    The following uses attention operators as an example. Specific values need to be measured in your own environment and cannot be directly reused.

    Step 1: Fix the random seed, run forward computation in FP32 and FP16 on the same batch of data, record the maximum absolute error and relative error layer by layer, and identify the layer where the error first becomes clearly amplified.

    Step 2: If the error is concentrated in scaling and summation before softmax, check whether the operator implementation on the target hardware performs row-wise numerical stabilization, such as subtracting the row maximum. Also check whether tensor layout causes frequent transposes.

    Step 3: Compare three implementations: framework-native operators, vendor-provided fused operators, and retaining FP32 accumulation in key layers. Record error and per-step throughput for each implementation.

    Step 4: Decide which implementation to adopt according to pre-agreed acceptance criteria. For example, layer-wise relative error should not exceed a team-defined threshold, and per-step training throughput should be no less than 90% of the target. Thresholds need to be determined based on task type and accuracy requirements.

    Dynamic shape compilation overhead also deserves separate checking. If requests at different resolutions frequently trigger recompilation, bucket resolutions to reduce compilation count.

    3.2 Memory Pressure: Enable Optimization Strategies in Order

    The recommended order is: mixed precision, gradient checkpointing, gradient accumulation, optimizer state sharding, and data prefetching and cache tuning. Each time an option is enabled, record peak memory, per-step time, and the loss curve. If an option slows training only to save memory, or affects convergence, it should be identified promptly.

    3.3 Multi-Card Communication: Calculate the Ratio First, Then Tune the Parallel Strategy

    Before adding more cards, measure per-step computation time T_comp and communication time T_comm, and calculate the communication ratio T_comm / (T_comp + T_comm). Based on engineering experience, if this ratio remains above about 30%, prioritize checking the following items instead of directly adding devices:

    • Whether the inter-card topology matches expectations and whether cards within the same node use high-bandwidth interconnect;
    • Whether gradient bucket size is reasonable;
    • Whether communication can overlap with backward computation;
    • Whether data loading blocks training;
    • Whether checkpoint saving synchronously blocks the training process.

    Resume-training validation also needs to be included in testing. Interrupt the process mid-training, restore from the latest checkpoint, and confirm that the loss curve is continuous, the learning rate schedule is correct, and the data iteration position has no duplication or omission.

    4. Data and Compliance: Work to Complete Before Training

    Data processing usually includes the following steps: remove images that are too low-resolution, corrupted, or duplicated; filter noisy text and clearly incorrect annotations; unify image size, color space, and file format; clean and normalize text descriptions, including tokenization and proper nouns where needed; establish reliable image-text correspondence; and evaluate data distribution based on sampling results to confirm that various scenarios are sufficiently covered.

    Compliance work needs to leave traceable records.

    Authorization records: For each data source, register the license agreement, scope of permission, and acquisition date. Data beyond the permitted scope must not enter the training set.

    Sensitive content filtering: Use dual filtering with a rule base and a classification model, and perform periodic sampling reviews of filtering results.

    Sampling review records can use the following fields: batch number, sample count, issue type, number of nonconforming samples, disposition, reviewer, and review date.

    5. Practical Experience Evaluation: An Executable Scoring Method

    5.1 Test Set

    Prepare 300 prompts covering six categories, 50 prompts each: subject count, attribute binding, spatial relations, mixed Chinese-English and colloquial expressions, long text, and negation descriptions.

    5.2 Sampling and Recording

    Use 4 fixed random seeds for each prompt, generating 1,200 images in total. At the same time, record model version, chip model, driver version, framework version, sampling steps, and resolution. This information is the basis for later reproduction.

    5.3 Scoring Rules

    Semantic adherence: Use a binary judgment. If subject, count, color, and position are all correct, score 1; otherwise score 0.

    Image quality: Three annotators score independently on a 1-to-5 scale, and the median is taken. Hide model information during annotation to avoid bias.

    Style consistency: For 4 images with the same style requirement, judge whether style features remain consistent and calculate the consistency rate.

    Same-prompt stability: Calculate the proportion of 4 images that simultaneously satisfy “semantic adherence correct” and “image quality not lower than 4 points.”

    5.4 Acceptance Thresholds

    The following thresholds are suggested starting points and should be adjusted according to business requirements: semantic adherence accuracy not lower than 85%, image quality median not lower than 3.5 points, and same-prompt stability not lower than 75%.

    Inter-annotator agreement also needs to be checked. If the Fleiss kappa of the three annotators is lower than 0.6, revise the annotation instructions first, then use the current results.

    5.5 Efficiency Metrics

    Also record the median and P95 of single-image time, peak memory, and success rate under concurrent requests.

    5.6 Reproduction Requirements

    Fix the seed and configuration file, and rerun the same script on the same machine model. If the core metric difference between two runs is within 2 percentage points, it can be considered reproducible. If it exceeds this range, investigate environment differences first.

    6. Deployment: Checklist from Conversion to Launch

    • After weight conversion, compare outputs before and after conversion using 100 fixed prompts, and measure the difference with PSNR or perceptual similarity such as LPIPS. If it is below the team-defined threshold, roll back to the previous precision configuration.
    • Single-card inference: record first-image latency and steady-state latency. After continuous requests, observe whether memory continues to grow; if it grows, investigate memory leaks.
    • Batch and concurrency: ramp up at 1, 2, 4, and 8 concurrent requests, record P50, P95, and error rate, and find the performance knee. Set the concurrency cap to about 80% of the knee.
    • Serviceization: set a request queue length cap, per-request timeout (for example, 60 seconds, adjustable by resolution), per-card isolation, and provide a health check endpoint.
    • Content safety: block sensitive prompts at the request entry. Generated results pass through an image moderation model; hits are not returned and are written to logs. Changes to moderation rules need version records.
    • Rollback conditions: automatically throttle when the error rate exceeds 2% for 5 consecutive minutes, or when P95 exceeds 1.5 times the target value. If the anomaly persists for 10 minutes and is still not recovered, roll back to the previous model version.
    • Failure drills: simulate a card failure and a process crash at least once a month, record recovery time, and compare it with the target recovery time.

    Recommended monitoring metrics:

    Metric Collection method Alert threshold Action
    Request success rate Gateway log statistics Below 98% Check model process and card status; roll back if necessary
    P95 latency Server-side instrumentation Exceeds 1.5x target value Throttle; check concurrency and queue
    Memory usage Device driver interface Continuous growth for more than 30 minutes Investigate leaks; restart instance
    Queue length Server-side metrics Exceeds 80% of cap Scale out or reduce concurrency
    Chip utilization Device monitoring Below 20% or near 100% for a long time Check data supply or batch settings

    7. How to Judge Whether a Domestic Solution Is Usable

    Check the following five items one by one: whether the pipeline is connected, with data-to-deployment completed in the target environment; whether evaluation metrics meet agreed thresholds; whether training can resume after interruption and whether the loss curve is continuous; whether deployment can roll back and whether recovery time is acceptable; and whether data authorization and review records are traceable.

    Note that a full-stack domestic deployment does not automatically mean the model quality is superior. Conclusions apply only to the model size, resolution, and task type tested in this run. For other scenarios, evaluation should be re-run.