Training Multimodal Image-Generation Models on Domestic AI Accelerators: Engineering Reliable Text Rendering

    This article examines the engineering path for training multimodal image-generation models on domestic AI accelerators, with a focus on overcoming text-rendering limitations. It covers chip adaptation, mixed precision, distributed parallelism, memory and I/O optimization, training stability, data construction, structured prompts, evaluation metrics, inference optimization, cloud service architecture, application scenarios, and deployment recommendations.

    Introduction: From Visually Appealing to Textually Correct

    Generating an image that also contains readable text has long been a deceptively simple but difficult requirement in generative AI. Early text-to-image models often produced attractive compositions and colors, but struggled with text: incomplete strokes, merged characters, repeated content, and incorrect mixed Chinese-English layouts. For practical products such as posters, covers, and knowledge diagrams, a single incorrect character can significantly reduce perceived quality.

    By 2026, this situation is changing. Text rendering in multimodal image-generation models has improved substantially. Posters, hand-copied reports, infographic pages, and illustrated titles are reaching a usable level. Leading results on visual text-generation and long-text-rendering benchmarks are increasingly coming from these models. At the same time, domestic AI accelerators are becoming capable of supporting full pretraining for this class of models. Training multimodal models on domestic compute is moving from small-scale validation to large-scale, full-parameter training.

    For developers and enterprises, three engineering questions matter:

    • How can domestic AI accelerators support efficient and stable multimodal training?
    • How can text rendering be quantified, optimized, and accepted as a hard requirement?
    • How can models serve generation workloads at scale with low unit cost?

    This article addresses these questions.

    1. Why Text Rendering Is the Boundary of Multimodal Image Generation

    1.1 Text Rendering as a Precision Benchmark

    Natural images tolerate small local errors. A finger joint that is slightly off may not be noticed. Text is different because it is a discrete symbol system. A small stroke difference can change the meaning entirely. Generated text quality therefore becomes a natural test of whether a model can consistently understand and generate content. The model must understand the text, place it correctly in the layout, and reconstruct the glyph shape stroke by stroke. Any failure is visible in the final image.

    1.2 The Last Mile of Image Generation

    Posters, covers, and knowledge diagrams share one characteristic: image plus text. The image attracts attention; the text carries information. If the text is wrong, the image usually cannot be used directly and requires manual retouching. This removes much of the automation value of a generative workflow. It also explains why recent evaluation sets focus on complex visual text generation and long-text rendering, pushing models from merely good-looking outputs toward usable outputs.

    1.3 Additional Difficulty for Chinese Text

    Chinese text rendering is harder than English text rendering for three main reasons:

    • High glyph complexity: thousands of commonly used Chinese characters have dense stroke structures, and similar glyphs can be hard to distinguish.
    • Strong layout constraints: vertical text, artistic fonts, calligraphic styles, outlined titles, and mixed Chinese-English layouts all increase the difficulty.
    • Deep context dependence: the model must understand the hierarchy among titles, subtitles, and body text, and generate the right content in the right place.
    Dimension English text rendering Chinese text rendering
    Glyph variety Few letters; high composability Large character set; sufficient coverage required
    Similar-character confusion Relatively limited Frequent, such as pairs with small visual differences
    Layout forms Mostly horizontal Horizontal, vertical, and artistic layouts coexist
    Evaluation focus Spelling correctness Glyph accuracy, semantic plausibility, layout hierarchy

    1.4 Multi-Size Output Is a Basic Application Requirement

    Social media, e-commerce, and office documents require different image sizes. Whether a model can natively generate common resolutions and aspect ratios directly affects whether content production can be automated. If the output must be cropped or stretched after generation, text can be cut off or distorted.

    2. Training Multimodal Models on Domestic AI Accelerators: Challenges and Engineering Responses

    Training multimodal generation models on domestic AI accelerators is not a single performance problem. It is a system-level engineering challenge involving hardware characteristics, framework ecosystem, model structure, and training scale.

    2.1 Chip Adaptation: Operator Coverage and Numerical Consistency

    Typical multimodal generation models include text encoders, image or joint image-text encoders, diffusion backbones, and variational autoencoder (VAE) decoders. Core operators such as attention, convolution, normalization, and activation functions may not be implemented exactly like mainstream hardware. Three issues need attention:

    1. Operator coverage and performance alignment: custom or fused operators may need dedicated adaptation; otherwise they can become bottlenecks.
    2. Numerical precision differences: accumulation order and rounding behavior may differ across hardware, causing subtle shifts in loss curves.
    3. Fusion strategy differences: attention is often implemented through fused kernels, and different implementations involve trade-offs in memory usage and compute efficiency.

    Recommended practices:

    • Run small-scale parity checks before training: use the same data and configuration for several hundred steps, and compare loss curves, gradient norms, and key intermediate tensors.
    • Establish an operator performance baseline: identify unoptimized operators and fill gaps with graph compilation, operator fusion, or custom kernels.
    • Track hardware and framework evolution so adaptation remains sustainable rather than a one-time rewrite.

    2.2 Mixed Precision: Balancing Memory and Throughput

    Mixed-precision training is a fundamental way to balance memory usage and throughput. BF16 has the same exponent range as FP32 and is less likely to overflow. FP16 has more mantissa bits and slightly higher precision, but requires loss scaling to prevent gradient underflow. For diffusion models, noise schedules involve different timesteps, and gradient magnitudes in backpropagation can vary widely. The precision strategy should be designed in layers:

    1. # Illustrative code: mixed-precision training loop on domestic AI accelerators (pseudocode)
    2. scaler = GradScaler(enabled=use_fp16) # Enable dynamic loss scaling for FP16 to prevent gradient underflow
    3. model.train()
    4. for step, batch in enumerate(dataloader):
    5. images = batch["image"] # Normalized image tensors
    6. texts = batch["text"] # Text descriptions aligned with each image
    7. with autocast(dtype=bf16 if use_bf16 else fp16):
    8. # 1) Forward pass: add noise to clean images and predict noise or velocity
    9. pred = model(images, texts)
    10. # 2) Loss: weight by timestep to balance gradient magnitudes under different noise levels
    11. loss = timestep_weighted_loss(pred, images, texts)
    12. scaler.scale(loss).backward()
    13. # Gradient clipping: limit gradient norms to reduce spikes from long sequences and complex structures
    14. scaler.unscale_(optimizer)
    15. clip_grad_norm_(model.parameters(), max_norm=1.0)
    16. scaler.step(optimizer)
    17. scaler.update()
    18. optimizer.zero_grad(set_to_none=True)
    19. if step % log_interval == 0:
    20. logger.info({
    21. "step": step,
    22. "loss": loss.item(),
    23. "grad_norm": grad_norm,
    24. "lr": scheduler.get_last_lr()[0],
    25. })

    Additional details on domestic AI accelerators:

    • Keep normalization statistics at sufficient precision.
    • Use FP32 accumulation for operations such as squares and square roots in the loss function when necessary.
    • Freeze the text encoder first and fine-tune it later to reduce interference between precision stages.

    2.3 Distributed Parallelism: Data, Sequence, and Pipeline Strategies

    Large-scale training usually combines multiple parallelism strategies. Table 2-1 compares common options. The final choice depends on model structure, memory capacity, and cluster interconnect bandwidth.

    Table 2-1 Comparison of Common Parallelism Strategies

    Parallelism What is split Main bottleneck addressed Major communication cost
    Data parallelism (DP/DDP) Training samples Model fits on a single card Gradient synchronization
    Tensor parallelism (TP) Weight matrices within a layer Single-layer parameters are too large Frequent layer-level communication; requires high-bandwidth interconnect
    Pipeline parallelism (PP) Model layers Many layers or insufficient memory Pipeline bubbles
    Sequence/context parallelism Sequence dimension High resolution or long sequences Attention communication
    FSDP/ZeRO-style sharding Parameters, gradients, optimizer states Optimizer state consumes memory Parameters are gathered and released as needed

    Multimodal generation training has two special characteristics:

    • High-resolution training amplifies activation memory usage: memory consumption grows quickly with resolution in pixel space or high-resolution latent space.
    • VAE decoding is expensive: if loss is computed in pixel space at every step, decoding can limit overall throughput.

    Practical options include:

    • Latent-space diffusion: run diffusion in a compressed latent space to reduce compute.
    • Segmented loss: apply the main loss in latent space and an auxiliary loss in pixel space at a certain ratio, preserving detail constraints while controlling decoding frequency.
    • Text-encoding cache: cache text-encoding results offline and read them directly during training to avoid repeated computation.

    Parallelism also needs to follow cluster interconnect topology. Tensor parallelism requires high bandwidth and is best placed within a node’s high-speed interconnect domain. Data parallelism and pipeline parallelism can scale across nodes.

    2.4 Memory and I/O Pressure in Long-Sequence Training

    When generating dense text, the model repeatedly relies on attention over local regions. Under high-resolution long-sequence training, memory usage is highly sensitive to activations. Common optimizations include:

    • Gradient checkpointing: trade compute for memory by storing only some intermediate activations and recomputing them during backpropagation.
    • Memory-efficient attention: reduce explicit storage of intermediate attention matrices.
    • Optimizer state sharding: split parameters, gradients, and optimizer states across devices.
    • Gradient accumulation: simulate a larger batch when memory is limited while maintaining stable training dynamics.
    • Mixed data loading: sample different data types, such as image-text pairs and synthetic layout data, according to a chosen ratio.

    There is also an I/O bottleneck. Image-text pairs are high-resolution and large in number. If cropping, scaling, and encoding are performed on the fly at every step, data loading can become the bottleneck. Common approaches include:

    • Moving preprocessing and augmentation offline, so training reads tensors directly.
    • Using local NVMe storage to cache hot data.
    • Persisting intermediate products such as text encodings and latent variables to object storage, with the data loader reading them in parallel across processes.

    2.5 Training Stability: The Engineering Baseline for Long Training Runs

    Training runs involving trillions of tokens and lasting weeks or months make stability as important as performance:

    • Elastic fault tolerance: use elastic training capabilities to rebuild communication groups after node failures or resource changes and recover from the nearest checkpoint.
    • Asynchronous checkpointing: save checkpoints in background threads to avoid blocking training for long periods.
    • Fine-grained monitoring: collect GPU utilization, memory usage, communication wait time, and data-loading latency to locate bottlenecks.
    • Anomaly detection: alert automatically on sudden gradient norm spikes, NaN loss, or slow compute nodes, and isolate faulty nodes quickly.

    In public cloud environments, these capabilities are often combined:

    • Elastic containers or container services can host distributed training jobs.
    • Object storage can hold checkpoints and datasets.
    • Message queues can smooth bursts of generation tasks.
    • Logging and monitoring services provide observability and timely response.

    3. Teaching the Model to Render Text Correctly

    Improvement in text rendering usually comes from a combination of data, training, and alignment, not a single trick.

    3.1 Data Is the Primary Factor for Text Rendering

    1. Real image-text pairs: e-commerce product images, posters, book covers, and slides naturally contain paired text and images and are high-quality data sources.
    2. Synthetic data for long-tail coverage: rendering titles, body text, and labels with vector fonts and composing them according to layout templates gives precise control over text content, position, font size, and color. Synthetic data may lack photographic realism, but it can provide many samples with accurate glyphs and well-structured layouts.
    3. Cross-language balance: mixed Chinese-English samples should avoid an overly high proportion of one language, which can degrade performance for the other.

    Example synthetic layout specification:

    1. {
    2. "canvas": {"width": 1080, "height": 1440},
    3. "background": {"type": "gradient", "colors": ["#FFD9E8", "#FFF7D6"]},
    4. "elements": [
    5. {"type": "title", "text": "Spring Travel Guide", "font": "handwriting", "size": 96,
    6. "bbox": [60, 80, 960, 220], "color": "#D92B4B", "stroke": {"width": 4, "color": "#FFFFFF"}},
    7. {"type": "subtitle", "text": "3 days 2 nights · about 800 yuan per person", "font": "sans", "size": 48,
    8. "bbox": [60, 240, 700, 310]},
    9. {"type": "bullet", "items": ["Route planning", "Accommodation", "Food list"], "size": 40,
    10. "bbox": [60, 340, 960, 560]}
    11. ]
    12. }
    1. # Illustrative code: layout composition pipeline (pseudocode)
    2. def render_sample(spec, background_pool):
    3. canvas = load_random_background(background_pool, size=spec["canvas"])
    4. for element in spec["elements"]:
    5. draw_element(canvas, element) # Render text elements under layout constraints for precise glyph and position control
    6. return {
    7. "image": canvas,
    8. "text": collect_text(spec), # Record all text that should appear in the image as supervision labels
    9. "layout": spec["elements"], # Record element positions for layout consistency evaluation
    10. }

    The final result depends on the ratio of synthetic to real data and how closely synthetic styles match the real distribution. Multiple controlled experiments are usually needed.

    3.2 Staged Training: General, Specialized, Then Aligned

    Stage Goal Data focus Key practices
    Pretraining Build general image-text alignment Large-scale image-text pairs Latent-space diffusion, text conditioning
    Text-focused continued training Improve glyph and layout reconstruction Image-text pairs plus synthetic layout data More high-resolution samples, higher weight on text regions
    Instruction tuning and preference alignment Align outputs with human aesthetics and usability standards Human-curated high-quality samples and pairwise preference data Reference-model regularization, learnable reward models

    “Text-region weighting” means assigning higher loss weight to text-dense regions so the model prioritizes correcting glyph errors.

    3.3 Structured Prompts: Reducing Uncertainty Through Input Constraints

    Even with strong text-rendering ability, structured prompts can improve output quality. The more clearly the model knows what to write, where to place it, and what style to use, the more stable the result. A useful structure includes four elements: scene, text content, layout, and style constraints.

    1. [Scene] A summer island travel scene, with a person seen from behind looking at the sea
    2. [Text content] Title: Summer Island Travel Guide; Subtitle: 6 experiences · budget checklist
    3. [Layout] Horizontal title area at the top, person and sea view in the middle, three-column key points at the bottom
    4. [Style] Bright colors, bold title, clear and undistorted text

    By contrast, a vague prompt such as “generate a travel poster” leaves layout, text, and style choices to the model and usually leads to higher uncertainty.

    4. Evaluation: Making Text Correctness Measurable

    4.1 Common Metrics

    • Word/character accuracy: the match rate between generated text and expected text; the most direct metric.
    • Normalized Edit Distance (NED): ranges from 0 to 1 and is computed as 1 minus the ratio of edit distance to reference text length. It reflects recognition errors, missing characters, and extra characters, and is more sensitive for long text.
    • Layout consistency: checks text position, font hierarchy, and overlap between text and images, usually with human evaluation or layout-detection models.
    • Semantic plausibility: checks whether the generated text matches prompt intent, for example whether a title summarizes the theme and whether bullet points correspond to body content.

    4.2 Scenario-Based Evaluation Matrix

    Table 4-1 Scenario-Based Evaluation Matrix for Text Rendering

    Scenario Key focus Recommended metrics Common failure modes
    Short titles Glyph accuracy and visual prominence Character accuracy Missing strokes, distorted artistic fonts
    Long text Completeness and contextual coherence NED, human rating Missing lines, repetition, broken sentences
    Mixed Chinese-English Correct language switching Per-language accuracy Letter distortion, pinyin-like output
    Complex backgrounds Contrast between text and background Human evaluation plus contrast detection Text blends into the background
    Vertical or artistic text Direction and stroke continuity Human evaluation Merged strokes, incorrect orientation

    4.3 Evaluation Pipeline

    1. # Illustrative code: automated text-rendering evaluation pipeline (pseudocode)
    2. def evaluate_text_rendering(model, prompts, evaluator):
    3. results = []
    4. for item in prompts:
    5. image = model.generate(item["prompt"], size=item["size"])
    6. # 1) Text recognition: obtain detected text sequences from the image
    7. recognized = ocr_recognize(image)
    8. # 2) Metric calculation: word accuracy and normalized edit distance
    9. acc = word_accuracy(recognized, item["target_text"])
    10. ned = normalized_edit_distance(recognized, item["target_text"])
    11. # 3) Layout and aesthetics: check whether text obscures the main subject and whether composition is balanced
    12. layout_ok = layout_consistency(image, item["layout_hint"])
    13. results.append({
    14. "case": item["id"], "word_acc": acc, "ned": ned,
    15. "layout_ok": layout_ok, "image": image,
    16. })
    17. return aggregate_report(results) # Produce scenario summaries and failure samples

    Human evaluation remains necessary. Automated metrics can detect wrong text, but they struggle to judge whether an image looks good or whether information hierarchy is clear. Combining automated metrics with human spot checks forms a complete quality loop.

    5. Inference: Low Unit Cost and Scaled Serving

    5.1 Common Inference Optimizations

    Technique Benefit Considerations
    Few-step sampling and distillation Reduces denoising steps and generation time Check whether text clarity degrades
    Resolution tiering Generate a low-resolution draft first, then upscale by stage Upscaling can affect edge sharpness of text
    Batching and dynamic batching Improves hardware utilization Control queueing latency
    Quantization and operator fusion Reduces memory and compute cost Evaluate accuracy loss
    Cache reuse Return cached results for identical prompts and parameters Set reasonable expiration policies
    Predictive instance scaling Scale out and in with traffic Combine with cold-start optimization

    5.2 Cloud Service Architecture

    When packaging the model as an API service, a layered architecture is recommended: access layer, inference layer, and storage layer.

    1. [Client / business system]
    2. │ HTTPS
    3. ▼
    4. [Access layer: authentication / rate limiting / parameter validation]
    5. │
    6. ▼
    7. [Inference layer: containerized model service with elastic scaling and multi-instance load balancing]
    8. │
    9. ├── Text encoding (cacheable)
    10. ├── Diffusion denoising sampling
    11. └── Decoding and post-processing (size cropping, watermarking, etc.)
    12. │
    13. ▼
    14. [Storage layer: object storage for images, logging for calls and latency]
    15. │
    16. ▼
    17. [Message queue: asynchronous task peak shaving] + [Monitoring and alerting: failure rate / latency / cost]

    Example request-handling logic:

    1. def handle_generation_request(req, client):
    2. # 1) Parameter validation: check size, aspect ratio, and count
    3. validate(req)
    4. # 2) Call the model service and obtain image bytes
    5. image_bytes = client.generate(
    6. prompt=req["prompt"],
    7. size=req["size"],
    8. n=req.get("n", 1),
    9. )
    10. # 3) Store the image in object storage and return an accessible URL
    11. key = f"generated/{req['biz_id']}/{uuid4().hex}.png"
    12. storage.put(key, image_bytes)
    13. return {"url": storage.public_url(key), "key": key}

    Pricing should balance predictability and fairness. Per-image pricing is simple and suits high-frequency, small-size scenarios. Time-based or compute-based pricing better matches actual resource consumption. A common approach is to price different resolution and step tiers separately, helping users choose an appropriate trade-off between cost and quality.

    6. Typical Application Scenarios

    1. Social media content production: cover images, educational infographics, and topic posters that need prominent text and eye-catching colors.
    2. E-commerce and marketing materials: product images, promotional banners, and campaign headers that require accurate text and clear selling points.
    3. Education and knowledge sharing: hand-copied reports, knowledge cards, and long infographics that need structured text and clear hierarchy.
    4. Office and document visuals: slide covers and report illustrations that emphasize consistent style and editability.
    5. Batch production on content platforms: combining template systems and prompt engineering to generate multiple material variants from one request, followed by human selection and refinement.

    Model outputs are probabilistic. When sensitive elements are involved, such as brand logos, personal portraits, or copyrighted fonts, content-safety review and human verification should be part of the workflow.

    7. Deployment Recommendations: From Evaluation to Production

    1. Define metrics by scenario: first clarify the business tolerance for text accuracy, then choose evaluation sets and thresholds.
    2. Define the solution by domestic compute: validate operator coverage, mixed-precision strategy, parallelism, and fault tolerance early to avoid rework during training.
    3. Define the ceiling by data: combine synthetic and real data, and focus on long text, mixed Chinese-English text, and complex layouts.
    4. Define acceptance by evaluation: combine automated metrics with human spot checks, build a regression set of failure samples, and rerun it after model or prompt changes.
    5. Define cost by service: choose pricing based on traffic patterns, and use batching, caching, and elastic scaling to bring unit cost into an acceptable range.
    6. Define boundaries by safety: establish review, watermarking, and traceability mechanisms for generated content, and clarify responsibility boundaries.

    Conclusion

    Moving from “can generate images” to “can generate usable images” requires multimodal generation models to turn text from decoration into a trustworthy information carrier. Training on domestic AI accelerators means this path no longer depends on a single hardware ecosystem. Meanwhile, engineering improvements in text rendering, layout consistency, and multi-size output determine whether these models can enter production workflows.

    For developers, understanding the coupling among training adaptation, data construction, evaluation metrics, and inference cost is more valuable than chasing a single benchmark score. When models, compute, data, and engineering pipelines form a closed loop, multimodal generation can move from demonstration to productivity.