Text Rendering in Multimodal Image Generation: Engineering Practices on Homegrown AI Chips

    Text rendering is one of the most visible and failure-prone capabilities in multimodal image generation. This article explains why glyph accuracy, text integrity, layout balance, and long-text retention break down, then presents an engineering workflow covering quantitative evaluation, structured rendering specifications, a generate-verify-repaint loop, resolution planning, and efficient training and inference on homegrown AI chips. It also discusses text-specific data pipelines, numerical stability, inference optimization, cost controls, deployment scenarios, acceptance checks, and current capability boundaries.

    1. Problem Definition: Why the Image Looks Great but the Text Is a Mess

    In multimodal image generation, text rendering is one of the easiest capabilities to notice and one of the hardest to get right. The root cause is a natural impedance mismatch between two representations:

    • Compression on the semantic side: Text encoders output high-dimensional semantic vectors. They are good at expressing what something means, not at specifying which pixel should contain which stroke.
    • Continuity on the visual side: Diffusion-style models iteratively denoise in a continuous latent space. Their optimization target is global perception and distribution matching; it does not inherently include a hard constraint that every glyph must have complete strokes.
    • Competition for capacity: When an image contains dozens of Chinese characters, the model must simultaneously handle composition, lighting, materials, perspective, and glyph rendering. The attention budget is significantly diluted, and small-text regions tend to fail first.

    Typical failure modes fall into four categories, which should be checked separately during acceptance testing:

    1. Glyph errors: missing strokes, extra strokes, misplaced components, or substitution with visually similar characters.
    2. Text fragmentation: words split apart, character order scrambled, or the same sentence repeated.
    3. Layout imbalance: text covering the subject, cramped line spacing, or confusing whitespace and hierarchy.
    4. Long-text degradation: longer sentences become blurrier, and the latter part is noticeably worse than the beginning.

    Understanding these four failure modes is a prerequisite for all later optimization work, because different failures require different fixes. Glyph errors mainly depend on model capability and specialized data. Layout imbalance mainly depends on prompts and layout constraints. Long-text degradation often requires segmented generation or local repainting.

    2. Evaluation Criteria: What Counts as Correctly Rendered Text

    To determine whether a model is production-ready, visual judgment is not enough. A quantifiable, regression-friendly evaluation process is necessary. The industry already has relatively mature benchmarks and metric systems for complex visual text generation and long-text rendering. The core idea is to feed generated results back into a recognition stage and form a closed loop:

    1. generated image → text detection/recognition → compare with intended copy → compute metrics → aggregate score

    Common metrics and their purposes:

    Metric Calculation What it primarily indicates
    Text accuracy Number of fully matched target text spans / total spans Whether characters match exactly
    Normalized edit distance (NED) 1 − (edit distance / reference text length) Degree of near-but-not-exact correctness
    Long-text retention Tail-paragraph accuracy / head-paragraph accuracy Degree of degradation over long sequences
    Layout compliance Images satisfying hierarchy and whitespace rules / total images Usability of the layout

    Three engineering points need attention:

    • Tokenization granularity: For mixed Chinese-English content, counting by word versus by character can produce very different conclusions. For Chinese, use characters as the base unit; for English, use words. Then report separate Chinese-English scores and an average.
    • Recognizer bias: The recognition stage itself can make mistakes, especially on artistic text, gradient text, and rotated text. Manually inspect 10%–20% of samples to calibrate automatic metrics.
    • Regression baseline: Whenever the model version, prompt template, or inference parameters change, run the same fixed sample set. Avoid decisions driven by one-off visual impressions.

    3. Turning Fuzzy Requirements into Controllable Rendering Specifications

    In practice, most failures are not simply caused by an incapable model. They are caused by prompts that leave critical information unspecified. A high-quality text-rendering prompt is essentially a structured layout requirements form. It should consistently include the following fields:

    • Layout type: cover, poster, infographic, card, receipt, screenshot.
    • Target copy: the complete, accurate, directly quotable text content, listed item by item.
    • Hierarchy: font size and weight order for the main title, subtitle, body text, and corner labels.
    • Language and character count: Chinese-English mixing ratio, maximum characters per line, total number of lines.
    • Visual subject: people, scenes, products, diagram elements.
    • Style and color palette: primary color, secondary color, texture, reference style description.
    • Negative constraints: no watermarks, no extra text, no distorted characters.

    Illustrative code (a framework-agnostic prompt construction template that only expresses the organization):

    1. def build_render_spec(layout, texts, style, negative):
    2. spec = {
    3. "layout": layout, # e.g., cover / poster / infographic
    4. "texts": texts, # e.g., {"title": "...", "subtitle": "..."}
    5. "hierarchy": ["title", "subtitle", "body"],
    6. "language": "zh-en",
    7. "max_chars_per_line": 12,
    8. "style": style, # palette, texture, composition preference
    9. "negative": negative, # distorted characters, watermarks, extra text, etc.
    10. }
    11. return render_template(spec) # assemble the final prompt

    A concrete prompt for reference:

    1. Generate a social media cover in a vertical format.
    2. Main title (extra-large, centered slightly above the middle, high-contrast stroke): "The Complete Thailand Travel Guide"
    3. Subtitle (medium size, below the main title): "7 days and 6 nights: itinerary + budget checklist"
    4. Scene: in the lower right, a person seen from behind; the background shows an island and a long-tail boat; the sky occupies the upper third.
    5. Palette: high-saturation cyan-blue contrasting with warm orange; rounded sans-serif typeface.
    6. Prohibited: any text other than the copy above, watermarks, logos, distorted characters, or Chinese characters with missing strokes.

    A few rules of thumb:

    • Provide the copy exactly as it should appear. Do not write “a title about Thailand travel information or similar.” Fuzzy descriptions inevitably lead to glyph drift.
    • The longer the copy, the more it should be segmented. Split a long passage into a main title, subtitle, and bullet points, and constrain the position of each.
    • Define layout before adjusting style. Images with layout errors cannot be rescued through style adjustments.
    • Use regional inpainting. Redrawing only the local area containing the text is more stable and cheaper than regenerating the whole image.

    4. From Lucky Draws to a Pipeline: A Generate-Verify-Repaint Loop

    Production environments cannot rely on luck. A reliable approach turns a single generation into a pipeline with automatic validation: generate a rough draft, recognize and compare the text, perform targeted repainting when quality is below threshold, and release the image when it passes.

    1. def render_with_verification(spec, max_rounds=3, threshold=0.93):
    2. image = generate(spec)
    3. for i in range(max_rounds):
    4. texts = extract_text(image) # detection + recognition
    5. ned = compute_ned(texts, spec["texts"]) # normalized edit distance
    6. if ned >= threshold:
    7. return image, {"rounds": i + 1, "score": ned}
    8. # Repaint only the lowest-scoring text region and preserve the remaining pixels
    9. image = inpaint_region(image, region=locate_worst(texts, spec))
    10. return image, {"rounds": max_rounds, "score": ned, "need_manual": True}

    Supporting engineering details:

    • Tiered models: A draft model handles composition and layout, while a refinement model handles text and details. Call the refinement model only for samples that pass layout validation. This can significantly reduce average unit cost.
    • Result caching: Cache final images with a key based on the hash of “prompt + specification parameters + model version.” Image-text assets are often repeated, and the hit rate is usually substantial.
    • Asynchronous queues: Generation is a long-running task. Use a message queue for decoupling. The API should return only a task identifier, and the frontend should poll or receive a callback for the result.
    • Observability: Write generation latency, repainting rounds, and NED scores to a logging service. Use monitoring and alerting to detect degraded versions. Store final images and task metadata in object storage for traceability and reprocessing.

    5. Resolution and Aspect Ratio: Physical Constraints on Text Clarity

    Multimodal generation models typically support outputs from 1024×1024 to 2048×2048 across multiple aspect ratios. This is important for social media, e-commerce, and office materials. However, higher resolution is not always better; tradeoffs are required:

    • Font size scales with resolution: A 10-point character may occupy only about 10 pixels of height on a 1024-wide image, making it almost impossible to restore stroke details. On a 2048-wide image, at least an identifiable skeleton becomes possible. A practical approach is to first set a lower bound for the pixel height corresponding to the smallest font (body text is recommended to be no lower than 20–24 pixels), then infer the required resolution.
    • Resolution-bucket training: During training, samples are usually drawn from several fixed-ratio buckets. Arbitrary ratios at inference time require extrapolation capability from positional encodings or sampling grids. Extreme aspect ratios outside the training distribution, such as a 1:5 banner, should be used cautiously.
    • Nonlinear growth in compute and memory: Attention and convolution costs rise rapidly with resolution. Doubling the long edge costs much more than a linear increase. Production systems often use a combination of “one high-resolution refinement + multiple low-resolution candidates.”

    6. Efficient Training and Inference on Homegrown AI Chips

    One notable industry trend is that leading multimodal models may now be trained entirely on non-mainstream accelerators. The engineering value of this is greater than its symbolic significance: it means supply-chain diversification, and it also means teams must build a “hardware-agnostic” capability stack. The key transferable stages are below.

    6.1 Operator and Graph Compilation Adaptation

    Performance bottlenecks on heterogeneous hardware are often not determined by peak compute, but by operator coverage. Practical points:

    1. Build an operator coverage checklist. Compare the model computation graph layer by layer, mark missing operators, slow operators, and numerically unstable operators. Prefer equivalent mathematical transformations, such as splitting complex fused operators into basic operators or using attention reordering to reduce dependence on special kernels.
    2. Use graph compilation and automatic operator generation to reduce the manual effort of per-operator hand tuning.
    3. Focus optimization on memory-bound stages such as normalization, elementwise operations, and upsampling. These operators account for a large share of vision models and easily become bottlenecks.

    6.2 Parallelism and Memory Strategies

    For diffusion-style multimodal models, training memory is dominated by activations. Common combined strategies include:

    • Data parallelism + parameter sharding: Use ZeRO-style sharding ideas to shard optimizer states, gradients, and parameters, reducing per-GPU memory pressure to an acceptable range.
    • Tensor parallelism and pipeline parallelism: Enable these only when a single layer cannot fit on one device. Communication overhead is high, so they must match the high-speed interconnect topology.
    • Activation recomputation: Trade a small amount of compute for memory. This is the default option at higher resolutions.
    • Mixed precision: Use low-precision computation in forward and backward passes while retaining high-precision master weights and loss scaling. Visual generative models are sensitive to numeric ranges, so monitor gradient norms and loss spikes carefully.

    6.3 Data Pipeline: Text-Rendering-Specific Data

    Text capability depends heavily on data construction, not purely on model architecture:

    • Image-text pair sources: web pages with images and text, document screenshots, posters and receipts, and book covers. These naturally carry strong supervision signals for “text and imagery coexisting.”
    • Weak-label augmentation: Use optical character recognition to annotate existing images with text boxes and textual content, forming low-cost region-level supervision.
    • Mixing ratio and curriculum: First learn composition and semantics from large-scale general image-text pairs, then increase the proportion of “samples containing text” for a specialization stage, and finally use manually selected high-quality layout data for preference optimization. Staging is more stable than one-time mixed training and makes regressions easier to locate.
    • Difficulty stratification: Grade samples by text amount, font size, language, and whether artistic text is present. Organize training rounds in increasing order of difficulty.

    6.4 Stability and Precision Governance

    A common pitfall in non-mainstream hardware ecosystems is “it runs, but it does not converge.” Four types of guardrails are recommended:

    • Run small-scale, short-duration validation early in training. Confirm that the loss curve has the same shape as a known baseline before scaling up.
    • Fix random seeds and data order to make failures reproducible.
    • Keep high-precision backup checkpoints and EMA weights so that degradation can be rolled back quickly.
    • For low-precision quantization on the inference side, perform layer-wise error analysis. Attention layers and the first and last layers are usually the most sensitive; keep them at higher precision.

    6.5 Inference Optimization Checklist

    Text rendering is a detail-sensitive task, so inference optimization must not sacrifice high-frequency details:

    • Few-step sampling and distillation: Serving a few-step student model distilled from a teacher model is the most effective way to reduce cost, but text accuracy must be regressed separately.
    • Service-side batching: Dynamically merge requests to improve throughput. At the same time, rate-limit long-tail high-resolution requests separately to avoid slowing the queue.
    • Quantization and operator fusion: Use low-bit quantization for linear layers and fuse normalization and elementwise operators to reduce kernel launch overhead.
    • Elastic scaling: Run stateless inference replicas on a container platform and autoscale based on queue depth. Use monitoring, alerting, and logging to identify hotspots.
    1. # General inference service configuration skeleton (illustrative; no vendor-specific interfaces)
    2. RENDER_CONFIG = {
    3. "model": "text_render_v1",
    4. "resolution": (1536, 1536),
    5. "steps": 28,
    6. "guidance": 6.5,
    7. "sampler": "few-step distilled sampler",
    8. "batch": {"min": 1, "max": 8, "timeout_ms": 20},
    9. "verify": {"enabled": True, "threshold": 0.93, "max_rounds": 2},
    10. "cache": {"enabled": True, "ttl_hours": 168},
    11. }

    6.6 Cost Perspective: What Determines the Cost per Image

    The compute required for a single generation is approximately proportional to “model size × sampling steps × pixel count.” Therefore, the correct order for cost reduction is:

    1. Increase the cache hit rate (almost zero risk).
    2. Use tiered models so that most requests are served by the draft model.
    3. Use distillation to reduce sampling steps, and regress text metrics.
    4. Only then consider lowering resolution or compressing precision.

    The reason for this order is that the first two steps do not affect quality, while the latter two trade quality for cost. They should be applied only after a quality baseline has been established.

    7. Deployment Scenarios and Acceptance Checklist

    High-value scenarios are concentrated in materials where text must be correct: social media covers and infographics, e-commerce hero images and promotional posters, event and recruiting materials, educational diagrams and process cards, and receipt- or form-style mockups. Before launch, check each item:

    • The target copy is provided character by character, with no fuzzy phrasing such as “or similar” or “etc.”
    • The pixel height corresponding to the smallest font meets the lower bound.
    • For mixed Chinese-English content, metrics are reported separately for each language.
    • The automatic validation chain (recognition + NED + threshold) is integrated; failed images are repainted automatically.
    • Long copy is split into main title, subtitle, and bullet points to avoid an overly long single section.
    • A fixed regression sample set has been established and can be compared across version changes.
    • Caching, asynchronous queues, rate limiting, and monitoring/alerting are all in place.
    • Content safety review and copyright compliance processes are applied both before and after generation.

    8. Capability Boundaries and Risks

    Objectively, current technology still has clear boundaries. Engineering teams should design mitigation paths in advance:

    • Very long text and dense layouts: Full-page documents and complex tables are still difficult to generate in one pass. Split them into blocks and composite them, or use a hybrid approach of “layout engine + generated assets.”
    • Low-resource languages and special fonts: Coverage is determined by the training-data distribution. Languages other than Chinese, English, Japanese, and Korean require separate evaluation.
    • Controllability limits: Precise coordinate-level layout, such as a mandatory 48-pixel margin, is not suitable for generative models. It should be handled by deterministic layout tools.
    • Compliance requirements: Brand identifiers, human likenesses, and copyrighted character designs should not be generated freely. Generated content should retain metadata for auditing.

    9. Conclusion

    The maturation of text-rendering capability moves image generation from a visual asset toy toward a material production tool. At the same time, making the training stack work on homegrown chips significantly reduces supply-chain risk for this capability path. For developers, the real deployment requirements are not mysterious: quantify evaluation criteria, structure requirements, turn generation into a pipeline with validation, and prioritize cost optimization by “zero quality loss first.” When all four are in place, upper-layer applications can produce usable results stably and predictably regardless of how the underlying acceleration hardware changes. On cloud platforms, object storage, container services, message queues, logging, and monitoring/alerting are generic capabilities that can serve as standard building blocks for training and inference systems.