LLM Application Evolution: Engineering Multimodal Generation from Chat Interfaces

    This article explores how LLM applications are evolving from text-only chat interfaces toward multimodal generation workflows. It explains why this shift changes user experience, business processes, and cost structures, and breaks down the engineering work needed across base-model capabilities, modality expansion, asynchronous task design, GPU resource scheduling, cost control, Chinese-language text rendering, video consistency, real-time audio/video interaction, quality evaluation, safety compliance, and deployment planning.

    1. Why multimodal generation has become a core requirement for LLM applications

    Early LLM applications commonly existed as chat interfaces, with capabilities limited to text Q&A and text creation. Once the language capabilities of foundation models began to converge, the user-perceivable difference moved quickly to another dimension: whether a single interaction can deliver an artifact that is immediately usable. A usable poster, a ten-second video clip, or a video call with visible facial expressions can create a much stronger experience impact than another block of text.

    This shift also changes three things:

    1. User experience model: from “Q&A” to an iterative loop of “generate — preview — revise — regenerate.” The number of interactions increases, but the value delivered in a single interaction can also be higher.
    2. Business process model: content production changes from multi-stage human collaboration to an instruction-driven automated pipeline, shortening the path from idea to finished asset.
    3. Cost structure model: text chat is a low-compute-density workload, while image generation—and especially video generation—is compute-heavy. Per-request costs can differ by several orders of magnitude, so billing, rate limiting, and queueing mechanisms must be redesigned.

    Therefore, multimodal generation capability is not simply about connecting a few more models. It requires a broad restructuring from interface design to scheduling systems.

    2. Three main tracks in capability evolution

    2.1 Foundation model capabilities: from chatting to planning and tool use

    The foundation of multimodal applications is still the base language model, but the focus of demand has changed. What really determines the ceiling of an application is not conversational fluency alone, but several engineering-friendly capabilities:

    • Instruction following and constraint adherence: whether the model can reliably produce text that conforms to a specified structure, which directly affects downstream parsing success rates.
    • Structured output: obtaining verifiable JSON results through schema constraints is a prerequisite for integrating a model into business systems.
    • Tool calling and planning: abstracting actions such as “generate image,” “generate video,” and “query data” into tools, and letting the model orchestrate them, is a practical way to handle complex tasks.
    • Context length and stability: long context is not only about a larger window; it also requires reliable information recall from the middle of long inputs.

    2.2 Modality expansion: text → image → video → real-time audio/video

    This path is not simply about adding modalities. Each stage introduces new technical difficulties:

    Stage Core method family New major challenge
    Text-to-image Diffusion models + image-text alignment encoders Composition control, text rendering, aesthetic quality
    Image-to-video / text-to-video Diffusion with a temporal dimension, spatio-temporal attention Temporal consistency, motion plausibility, compute amplification
    Real-time audio/video interaction Multimodal understanding + streaming recognition and synthesis End-to-end latency, interruption handling, transport jitter

    It is worth emphasizing that image-to-video is usually more controllable than pure text-to-video. Once the first frame is given, subject identity, composition, and color tone are anchored, and the model mainly needs to solve how the scene should move. In product design, making image-to-video the primary path and text-to-video a secondary entry point often yields more reliable usable results than the reverse.

    2.3 Endpoint forms: distributing the same capability across multiple surfaces

    Capabilities ultimately reach users through mobile apps, lightweight mini-apps, web interfaces, and other entry points. Multi-surface consistency does not mean identical implementation. The real differences come from three constraints:

    • Network conditions: mobile networks are often unstable, with frequent switching. Long-running tasks must support disconnection recovery and result retrieval.
    • Compute location: heavy generation tasks will almost always run in the cloud; edge devices are better suited for preprocessing such as noise reduction, beauty enhancement, and frame cropping, as well as lightweight inference.
    • Interaction modes: touchscreens are suitable for parameterized editing, while voice and cameras are suitable for continuous natural interaction. Their input collection and state management are fundamentally different.

    3. Engineering breakdown of generation capabilities

    3.1 Task model: from synchronous requests to asynchronous jobs

    Generating a video can take tens of seconds or even minutes. Carrying that work over a synchronous HTTP long connection is neither robust nor economical. The common industry approach is a three-stage pattern: submit — query — callback. The client submits a task and receives an identifier, then polls or receives status updates through a message channel. Artifacts are delivered via signed links on object storage.

    1. # Illustrative code: interface design for asynchronous generation tasks (framework-agnostic)
    2. @app.post("/v1/generation/tasks")
    3. def create_task(req: TaskRequest, user = Depends(auth)):
    4. # Idempotency key: avoid duplicate tasks and duplicate billing when users repeat clicks or clients retry
    5. key = f"{user.id}:{req.idempotency_key}"
    6. if cached := cache.get(key):
    7. return cached
    8. if not quota_check(user, cost=ESTIMATE_COST[req.modality]):
    9. raise HTTPException(429, "rate_limited")
    10. task = Task(
    11. id=uuid4().hex,
    12. owner=user.id,
    13. payload=req.model_dump(),
    14. status="queued",
    15. created_at=now(),
    16. )
    17. db.insert(task)
    18. cache.setex(key, 3600, {"task_id": task.id, "status": "queued"})
    19. queue.publish("gen.default", task.id, priority=req.priority)
    20. return {"task_id": task.id, "status": task.status}
    21. @app.get("/v1/generation/tasks/{tid}")
    22. def get_task(tid: str, user = Depends(auth)):
    23. task = db.get(tid)
    24. if task is None or task.owner != user.id:
    25. raise HTTPException(404) # Avoid probing others' resources through task IDs
    26. return {
    27. "status": task.status, # queued / running / succeeded / failed
    28. "progress": task.progress,
    29. "artifacts": task.artifacts if task.status == "succeeded" else [],
    30. "error": task.error_code, # Expose a stable error code externally, not a stack trace
    31. }

    Several points in this design are easy to overlook but critical:

    • Idempotency key: retries caused by network jitter are normal. Without it, a single user action can create three generation tasks and consume three quotas.
    • Explicit state machine: at minimum, it should include queued, running, succeeded, and failed states. Failures should distinguish between retryable cases such as timeouts and insufficient resources, and non-retryable cases such as invalid input or invalid parameters.
    • Artifact delivery: avoid routing large files through the application server. Use object storage with temporary signed links, and configure lifecycle policies for automatic cleanup.
    • Resumability: long-running tasks should save intermediate artifacts, such as keyframes or draft-resolution results, so that failed jobs can resume from the failure point instead of being fully recomputed.

    3.2 Resource scheduling: GPU memory is the real scarce resource

    Capacity planning for generation services is very different from that for conventional web services. CPU utilization does not accurately reflect the true pressure on GPU-backed services. Instead, the system should focus on:

    • GPU memory usage: resolution, frame count, duration, and batch size together determine peak GPU memory consumption, and the relationship is often superlinear. Out-of-memory failures are a primary failure mode, so request-level resource estimation and admission control are necessary.
    • Queueing time: what users perceive as “slow” often comes from queueing rather than the computation itself. Tracking queueing time as a first-class metric is necessary to locate problems.
    • Multi-level queues: mixing interactive requests, which require low latency and small batches and can be preempted, with batch production requests that can tolerate longer waits will degrade both. They should be physically isolated into different resource pools.

    For scaling strategies, queue depth is a better trigger than CPU utilization, and scale-down should be much slower than scale-up to avoid oscillation:

    1. autoscaling:
    2. metric: queue_depth_per_replica
    3. target: 8
    4. min_replicas: 2
    5. max_replicas: 40
    6. scale_up_cooldown: 30s # Scale up quickly to preserve latency
    7. scale_down_cooldown: 300s # Scale down slowly to avoid oscillation when traffic returns
    8. admission:
    9. reject_when_queue_depth: 200
    10. on_reject: return_queued_with_eta # Provide an estimated wait time instead of returning a hard error during overload

    During overload, queuing requests and returning an estimated wait time is usually better for user retention than immediately returning failures. However, a queue limit is still required; otherwise, backlog can cause every request to time out.

    3.3 Four levers for cost control

    1. Two-stage generation: first generate a low-resolution draft for users to confirm composition and semantics, then render a high-definition final result after confirmation. This usually eliminates a large share of wasted compute.
    2. Result reuse caching: use a hash of the normalized prompt, key parameters, and model version as the cache key. The random seed must be included in the key; otherwise, results may be reused incorrectly.
    3. Resolution and duration tiers: tie cost to user-visible value. Drafts can be free, while final renders consume quota. The pricing structure should remain consistent with the technical structure.
    4. Tiered storage and transcoding: retain original high-bitrate artifacts for a short period, then transcode, compress, or move them to lower-cost storage to prevent object storage costs from growing linearly over time.

    4. Two challenges specific to Chinese-language scenarios

    4.1 Rendering Chinese characters inside images

    Chinese characters have dense strokes and limited glyph space. When diffusion models generate text inside images, they are prone to producing structurally corrupted pseudo-Chinese characters. This is a notable disadvantage compared with Latin-script scenarios. Several mitigation strategies can be combined:

    • Glyph conditioning: render target text into a glyph raster image as an additional control condition, so the model traces the glyph rather than relying on memory.
    • Layout region constraints: use layout control to specify text regions and ensure each character receives enough pixel budget. Too small a font size is a common cause of failure.
    • Character-aware evaluation and training signals: introduce explicit supervision of glyph structure at the data and loss-function levels.
    • Product-side fallback: after generation, use text recognition to validate the output. If validation fails, retry. If multiple attempts fail, fall back to a layered approach that combines generated graphics with vector glyph compositing.

    The last point is especially important: a closed loop of generation + validation + fallback is more robust than pursuing a high single-pass success rate alone.

    1. def render_with_text_guard(image_bytes, text_items, max_retry=2):
    2. for attempt in range(max_retry + 1):
    3. img = generate(image_bytes)
    4. ocr = text_detection_service.detect(img)
    5. if all(ocr.contains(item.text, min_conf=0.90) for item in text_items):
    6. return img, {"fallback": False, "attempt": attempt + 1}
    7. # Recognition failed in the previous attempt; strengthen constraints and retry
    8. image_bytes = strengthen_text_condition(text_items)
    9. # Fallback: keep the generated image and composite real glyphs as a text layer to ensure usability
    10. out = composite_text_layer(base=img, text_items=text_items)
    11. return out, {"fallback": True, "attempt": max_retry + 1}

    4.2 Temporal consistency in video generation

    The core quality problems in video generation are subject drift, brightness flicker, and identity inconsistency. Common engineering strategies include:

    • Keyframe anchoring: provide a first frame for image-to-video, or first and last frames, and let the model interpolate the intermediate frames. This significantly improves controllability.
    • Reference conditions: use reference images to constrain the appearance of people or objects, reducing cross-frame drift.
    • Segmented generation plus stitching: generate short segments separately, then stitch them with transitions or frame interpolation. This trades duration for stability.
    • Motion amplitude as a tunable parameter: small-motion results have a much higher success rate than large-motion results. Products should provide motion-intensity levels rather than asking users to freely describe motion.

    5. Pipeline design for real-time audio/video interaction

    Conversational interaction with a camera and microphone is essentially a chain that connects four capabilities into a low-latency pipeline: streaming recognition → multimodal understanding and dialogue → streaming speech synthesis → real-time encoding and transmission. Latency accumulates across each stage, and synchronous waiting in any stage can directly ruin the experience.

    1. Camera / microphone
    2. → on-device preprocessing (echo cancellation, noise reduction, automatic gain control)
    3. → streaming transport (jitter buffering, packet loss compensation, adaptive bitrate)
    4. → multimodal input understanding (audio frames + video keyframes + text)
    5. → conversational model (streaming token output, first-token priority)
    6. → streaming speech synthesis (sentence-level chunking, first-packet priority)
    7. → encoding and push stream → playback

    Key practices include:

    • First-packet latency first: perceived quality is mainly determined by the interval between when the user stops speaking and when they hear the first word, not overall throughput. It is better to sacrifice voice timbre consistency than to delay the first sound.
    • Sentence-level synthesis chunking: trigger synthesis when the model reaches the first sentence-ending punctuation, rather than waiting for the entire response to be generated.
    • Interruption support: use voice activity detection to determine when the user starts speaking, immediately stop the current playback, and reset the context. This is essential for natural conversation; without it, users may feel that the system is reading from a script.
    • Multi-turn context feedback: inject the multimodal understanding results from the previous turn, such as what the user showed and what the environment looked like, into the next context. This makes the interaction feel as though the system can see and hear.
    • Device-side fallback: under weak network conditions, reduce video resolution and frame rate to prioritize continuous audio. Audio interruption damages the experience much more than a blurry image.

    6. Quality evaluation: the most difficult part of generation applications

    There is no single metric that fully captures generation quality. A layered approach is needed:

    1. Automated metrics: image-text semantic consistency, such as scores from image-text similarity models; text recognition accuracy within images; frame-to-frame consistency; aesthetic scores. It is important to recognize that these are proxy metrics and only have limited correlation with subjective user satisfaction.
    2. Human evaluation: pairwise comparison, where annotators choose whether A is better than B, roughly equivalent, or B is better than A, is more stable than absolute scoring. At the same time, annotator agreement should be monitored; low agreement indicates ambiguity in the evaluation criteria.
    3. Online behavioral metrics: first-attempt generation success rate, regeneration rate, result adoption and download rate, secondary editing rate, and cost per task. The regeneration rate is the most sensitive signal for exposing quality problems.
    4. Fixed regression sets: maintain a set of prompts covering typical intents, combined with fixed parameters. Any model version or sampling parameter change should run through this set.

    One common methodological mistake is to bundle changes to model version, sampling parameters, and prompt templates into the same release, making it impossible to attribute quality changes. They should be managed as independently switchable experimental factors.

    7. Engineering touchpoints for safety and compliance

    Generative capabilities have a significantly larger risk surface than text-only conversation. Multiple defense points are needed along the pipeline:

    • Input side: detect prohibited content, guard against prompt injection, and validate external inputs such as reference images.
    • Generation side: identify and avoid requests involving real people’s likenesses or protected identifiers.
    • Output side: review images as a whole, sample frames for video review, and review audio after transcription. Frame sampling density should match video length.
    • Traceability: store artifacts together with generation parameters, model versions, review conclusions, and requesting entities. Retention periods should be configured according to applicable regulations.
    • Content labeling: add explicit prompts and implicit identifiers, such as watermarks or metadata, to generated content. Identifiers should withstand common transcoding and cropping.
    • User rights channels: provide entry points for content deletion and data clearing, especially for biometric data such as faces and voiceprints.

    Review should not serve only as post-hoc blocking. Review results should feed back into quality signals. An abnormal increase in block rate often indicates that prompt constraints have failed or that the upstream model version has regressed.

    8. Deployment checklist

    • Are long-running capabilities asynchronous, with idempotent interfaces and explicit state machines?
    • Is there a two-stage design of “draft preview → high-definition finalization”?
    • Is autoscaling based on queue depth rather than CPU utilization, and is scale-down sufficiently gradual?
    • Are artifacts delivered through object storage with signed links and lifecycle cleanup?
    • Are failures classified as retryable or non-retryable, with user-visible retry and compensation paths?
    • Is a fixed regression set in place, and can model and parameter changes be attributed independently?
    • Are generated content pieces identifiable, traceable, and deletable?
    • Is there a budget gate and an over-limit degradation strategy for per-task cost?
    • For critical quality points such as in-image text, is there a closed loop of generation + validation + fallback?

    9. Conclusion

    The evolution from conversation to multimodal generation may appear to be a continuously growing list of model capabilities. In practice, it is a shift in engineering paradigm: from web services that are low-latency, stateless, and synchronous to production systems that are long-running, resource-heavy, asynchronous, and require quality evaluation. Foundation model capability iterations determine the upper bound of what is possible, while queue orchestration, GPU scheduling, cost control, and quality fallback determine whether these capabilities can be delivered reliably to users. The dividing line for multimodal applications often depends not on whether the model is chosen correctly, but on whether these engineering details are treated seriously.