Text-Only LLMs Can Generate Video: A Code-Driven Rendering Pipeline

1. The Core Problem: Can Video Be Generated from Text Alone?

1.1 A common misconception

When people first try to produce video with an LLM, the instinct is to look for a multimodal model that “can see”: hand it a reference image or a sample clip, let it interpret them, then generate. If only a text-only model is available, the usual conclusion is that the job can’t be done.

That conclusion conflates two separate things: the boundary of the model’s capability, and the level of abstraction at which the task is defined. It is true that a text-only model cannot understand pixel content, and there is no way around that. But many stages of video production are not about understanding a picture at all — they are about defining one.

1.2 Video is a decomposable, deterministic structure

Structurally, a video is remarkably regular: a set of frames laid out at a fixed frame rate, plus one or more audio tracks that change over time. The picture inside each frame can be broken down further:

• Background layer: solid color, gradient, image, or animated texture;
• Content layer: titles, body copy, charts, icons, photos;
• Effects layer: masks, blur, glow, grain;
• Transition layer: fades, slide transitions, zoom transitions.

Every layer has an explicit position, size, color, and opacity, plus its own function of time. What we call “animation” is really nothing more than “a set of parameters expressed as functions of time.” And the way you express a function is text — precisely the input/output format a large model handles best.

1.3 The core idea: have the model write a program, not paint a picture

This leads to the central proposal of this article: instead of asking the model to draw frames, ask it to write a program that decides every pixel at render time.

Throughout the whole process the model never has to “see” a single frame. It only has to describe precisely that the title slides in from the left at frame 0, starts fading out at frame 45, and cuts to the next scene at frame 90. Deterministic rendering turns those descriptions into pixels. A text-only model thus stops being the odd one out and becomes a core producer on the pipeline.

2. Solution Overview: A Four-Stage Pipeline

The pipeline breaks into four sequential stages plus one parallel audio channel:

Stage	Input	Handled by	Output
Script	Topic, audience, source copy	Text LLM	Storyboard script (structured JSON)
Description	Storyboard, layout templates	Text LLM	Declarative scene component code
Rendering	Component code, audio durations	Render engine (headless browser + video encoder)	Video file with no audio track
Post-processing	Video, audio, subtitles	Editing/compositing tools	Final cut

The audio channel runs in parallel with the script stage: as soon as the narration copy for the storyboard is finalized, it goes to a text-to-speech service, and the real duration of each narration clip is fed back to the rendering stage to determine the frame count of each shot.

The data flow looks like this:

Creative intent
  └─▶ Storyboard (JSON)
        ├─▶ Scene component code ──▶ Frame-by-frame rendering ──▶ Silent video ─┐
        └─▶ Narration text ────────▶ Text-to-speech ──▶ Audio + timestamps ────┤
                                                                                 ▼
                                        Mixing, subtitle burn-in, concatenation ──▶ Final cut

The reason for layering is that a large model is most error-prone when it has to emit one long chained artifact in a single pass: if the storyboard drifts, all the code and rendering downstream are wasted. Collapsing each layer’s output into a verifiable structure lets errors be caught before they propagate.

3. Key Design One: Build a Programmatic Model of Video

3.1 Time must be controlled, not elapsed

The first principle of programmatic rendering: every visual change must be a function of the current frame, never of wall-clock time.

That means avoiding three categories of code:

• Timer-driven animation (setInterval, setTimeout and similar);
• Gradients and transitions that depend on the real clock;
• Uncontrolled randomness.

The renderer’s job is to “seek the timeline to frame i and ask the page to draw the state of that frame.” Only purely functional rendering guarantees that seeking to any frame yields one uniquely determined result. Otherwise, frame-skipping rendering produces discontinuous animation, and parallel rendering produces fragments that contradict each other.

A minimal working scene component looks like this:

// Declarative scene component: every visual state is a function of the current frame
type SceneProps = {
  title: string;
  subtitle: string;
  durationInFrames: number; // derived from the narration audio duration
};
const FPS = 30;
function SceneTitle({ title, subtitle, durationInFrames }: SceneProps) {
  const t = useTimeline();                 // { frame, fps }, provided by the render runtime
  const p = clamp01(t.frame / 20);         // enter progress 0 → 1, done within 20 frames
  const q = clamp01((t.frame - (durationInFrames - 15)) / 15); // exit progress
  return (
    <div style={{ width: 1920, height: 1080, background: '#0B1020' }}>
      <h1 style={{
        fontSize: 96,
        color: '#FFFFFF',
        opacity: p * (1 - q),
        transform: `translateX(${(1 - p) * -200 - q * 200}px)`,
      }}>{title}</h1>
      <p style={{ fontSize: 40, color: '#AAB4C8', opacity: clamp01((t.frame - 25) / 20) * (1 - q) }}>
        {subtitle}
      </p>
    </div>
  );
}

3.2 A shot is a scene; a timeline is a sequence

Cut the whole video into a number of scenes, each an independent, parameterized component. A scene has its own inputs (copy, images, duration) and its own output (the number of frames it occupies), and an upper-level timeline container stitches them together in order.

This decomposition buys three things:

1. Localized changes: editing the copy of shot 3 only triggers a re-render of that shot, with no knock-on effect on the rest of the video;
2. Parallel-friendly: scenes share no state, so they can be distributed across multiple machines and rendered simultaneously;
3. High reuse: the same layout component with different copy is a new shot, so the template library gains value with every use.

3.3 Assets and fonts must be controlled

Model-generated code has a recurring hazard: it casually references remote image URLs or absolute local file paths. The former makes rendering depend on external network access and results non-reproducible; the latter breaks outright on a different machine.

The correct approach is to maintain an asset manifest: every image, icon, and font is uploaded and registered in advance, and code may only reference the logical name from that manifest, which the build stage maps to the real address. Fonts matter especially — if the rendering environment has no specified font, it silently falls back to the system default, and the layout ends up looking “slightly off” in a way that is very hard to track down.

4. Key Design Two: Feed the Runtime Contract to the Model

A model usually fails to write directly runnable code not because it isn’t smart enough, but because it doesn’t know the rules of the target runtime. The point of the prompt is therefore not “please write better code” but to state the contract clearly.

4.1 The contract checklist

A proper rendering contract should include at least:

• Canvas size and frame rate (e.g. 1920×1080, 30fps);
• The names, signatures, and semantics of available built-in components and hooks (timeline container, image component, narration subtitle component, etc.);
• Styling conventions and unit definitions (what pixels, percentages, and rem each mean);
• The convention for duration units (frames throughout, or seconds converted consistently);
• An explicit prohibition list (no network requests, no real time, no randomness, no external scripts);
• Known layout constraints (safe-area margins, minimum font size, maximum characters per screen).

The contract should be maintained as a versioned file and injected with every request. When the render runtime is upgraded, the contract is updated in lockstep so the model isn’t still writing code to old rules.

4.2 A structured intermediate format: storyboard first

More robust than generating code directly is having the model first produce a verifiable structured storyboard:

{
  "fps": 30,
  "canvas": { "width": 1920, "height": 1080 },
  "scenes": [
    {
      "id": "intro",
      "template": "center-title",
      "narration": "Three minutes on how large models take part in video production.",
      "durationInFrames": 150,
      "content": { "title": "From copy to final cut", "subtitle": "A reusable pipeline" }
    },
    {
      "id": "point-1",
      "template": "bullet-left",
      "narration": "Video is a sequence of frames, and animation is a function of parameters over time.",
      "durationInFrames": 180,
      "content": { "bullets": ["Video = frame sequence + audio track", "Animation = parameters as a function of time"] }
    }
  ]
}

This intermediate format has three kinds of value:

• Cheap error correction: missing fields, overlong copy, or a zero duration are caught by JSON Schema validation in seconds, far cheaper than discovering them after a full render;
• Reviewable and diffable: the storyboard goes into version control, so what changed is obvious at a glance;
• Decoupled generation and rendering: the storyboard can be edited without the model, and code generation can be skipped entirely by handing it to a generic renderer that maps the template field to a layout automatically.

4.3 A feedback loop: turning “invisible” into “verifiable”

A text-only model cannot see the picture, but that doesn’t make iteration impossible. A workable loop:

1. After generating code, run static checks first: syntax parsing, type checking, removing unused imports;
2. Test-render three sampled frames (first, middle, last) and stitch them into a contact sheet;
3. Have a vision-capable model or a human produce a structured diagnosis of that contact sheet: is text overflowing, are elements overlapping, is contrast sufficient;
4. Feed the diagnosis back to the text model together with the renderer’s error logs so it can fix the code.

The key is the division of labor: visual understanding belongs to the vision stage, while the creation of the picture stays with the text model. This neither oversteps the model’s own capability boundary nor leaves the missing feedback signal unaddressed.

4.4 Rules of thumb for text overflow

Chinese characters are visually “fuller” than monospaced Latin characters, so layout estimates need to be more conservative. The rules of thumb commonly used in engineering:

• On a 1080p canvas, a 96px headline should be kept to 12 Chinese characters per line;
• At 40px body size on a 1920px width, no more than 22 Chinese characters per line;
• No more than 5 bullet points per screen, each no longer than 18 characters; beyond that, split into two shots.

Putting these numbers into the contract is far cheaper than reworking frame by frame afterwards.

5. Key Design Three: Voice First, Then Shot Durations

5.1 Get the order wrong and everything downstream is wrong

A frequent mistake is to render the picture to estimated durations first and add the voiceover afterwards. The result is either narration cut off mid-sentence when the picture changes, or the picture lingering long enough to create an awkward dead pause.

The correct order is reversed:

1. Finalize the narration text in the storyboard;
2. Call text-to-speech segment by segment to obtain each audio file and its real duration;
3. Derive each shot’s frame count with frames = ceil(audioDuration × fps) + breathing frames;
4. Only then proceed to code generation and rendering.

With this order the picture rhythm follows the voice rhythm naturally, and audio-video sync is guaranteed structurally rather than forced into place by trial-and-error alignment in post.

5.2 Phoneme-level timestamps driving subtitles and highlights

If the text-to-speech service can provide word-level or phoneme-level timestamps, two things can be done to noticeably improve the viewing experience:

• Karaoke-style subtitles that highlight word by word look clearly better than a whole sentence appearing at once;
• Key numbers and terms can pop in at the moment they are spoken, aligning “auditory information” with “visual information” in time.

Without timestamps, the fallback is to synthesize and concatenate sentence by sentence, using each sentence’s boundary time as the subtitle granularity.

5.3 Basic mixing conventions

• Loudness: keep enough level difference between voice and background music, prioritizing voice intelligibility; normalize the overall loudness of the final cut to the target platform’s convention (commonly in the -16 LUFS to -14 LUFS range), leaving headroom at the peak so transcoding doesn’t clip;
• Transitions: add a 10–30 ms audio fade in/out at each shot change to eliminate pops;
• Compression and noise reduction: apply light compression to the narration so volume is more even across separately recorded segments.

6. Engineering: How to Build the Rendering Pipeline

6.1 Two forms of render engine

Form one: headless browser, screenshot per frame. The renderer seeks the timeline to frame i, the page reads the state for that frame and paints it, and a screenshot yields one image. Once all frames are collected, they go to a video encoder to be composited into a standard video file. The advantage is true WYSIWYG: CSS layout, font rendering, and SVG vector graphics can all be reused directly. The downside is that throughput is limited by the per-frame cost of browser screenshots.

Form two: pure code drawing of frames. A graphics library generates each frame’s pixel data directly in memory, then hands it to the encoder. The advantage is speed, no browser overhead, and easy parallelism. The downside is that you have to implement text layout, line wrapping, and ellipsis truncation yourself — and Chinese typography is especially easy to get wrong.

A common combination in practice: use form one for previews and drafts to validate the look quickly, and form two for high-volume production to maximize throughput. Both share the same storyboard contract, which keeps the visuals consistent.

6.2 Parallelism, caching, and idempotency

• Sharded parallelism: shard by scene, with each shard an independent render task; once all shards finish, concatenate them in timeline order.
• Content-addressed caching: the cache key is determined jointly by “storyboard content hash + component code hash + asset hash + runtime version + render parameters.” If the key is unchanged, reuse the artifact directly; change one word and only one shot re-renders.
• Idempotent retries: a task table records status, retry count, and error summary. Render tasks may be re-run any number of times and must produce identical results — which is exactly why the purely functional rendering requirement in section 3.1 is non-negotiable.
• Contract validation up front: a task must pass JSON Schema validation and asset-existence checks before entering the render queue. Letting an invalid task fail on a render machine is the most wasteful way for it to fail.

6.3 Distributed capacity

Rendering is a typical stateless, CPU-intensive workload and scales out naturally. On Baidu AI Cloud, you can assemble it from existing general-purpose building blocks:

• Run render workers on a container platform, with automatic scaling based on queue backlog;
• Store frame sequences, audio, and final cuts in object storage, using lifecycle policies to clean up intermediate artifacts;
• Use a message queue for task distribution and priority control, isolating long jobs into a dedicated channel;
• Collect renderer output and failure stack traces with a logging service, and watch queue backlog, failure rate, and P95 render duration with monitoring and alerting.

The team can then focus purely on storyboard templates and prompt engineering instead of building its own scheduling substrate.

6.4 Points to tune render parameters

• Higher concurrency is not always better. Browser rendering has memory peaks, and over-provisioning gets processes force-terminated, which actually lowers overall throughput; after load testing to find the throughput inflection point, use about 70% of it as the default.
• Using lossy compression for intermediate frames and a higher bitrate for the final cut significantly reduces disk usage and I/O pressure.
• Prefer hardware encoding; if the GPU is also serving inference workloads, partition VRAM and time slices between the two so they don’t contend.

7. Post-Processing: Extending the Capability Boundary with Tool Calls

The render engine is only responsible for the picture. The remaining work — editing and compositing, subtitle burn-in, stitching intros and outros, extracting cover frames — should be wrapped as a set of parameterized video-editing tools and exposed to an agent through the model’s structured tool calling (function calling).

7.1 Four principles for tool design

1. Narrow, enumerated parameters: if you can accept transition: "fade" | "slide-left" | "none", don’t accept free text; it shrinks the model’s room to err;
2. Declarative description rather than command concatenation: have the model output a structured object describing “which clips are on the timeline and their in/out points,” not a shell command string — the latter carries command-injection risk and is more error-prone;
3. Validate side effects up front: output paths must fall inside a controlled directory, with no relative-path escapes; asset references must come from the registered manifest;
4. Explicit timeout and retry policy: video processing is a long-running task, so the API layer should return a task ID immediately, with status retrieved via a query endpoint or message notification rather than blocking synchronously.

7.2 Keep orchestration logic in code

Worth emphasizing: the model decides what to do; code ensures it is done correctly. Constraints such as clip order, duration validity, and audio track alignment should be implemented as deterministic validation code — don’t count on the model never making a mistake.

8. Quality Assurance: How to Validate a Video You Can’t See

8.1 Three layers of checks

• L1 structural: JSON Schema validation, static code checks, asset and font existence checks. Failure means the task is rejected before entering the render queue.
• L2 visual: sample the first, middle, and last frame of each shot into a contact sheet, and review it manually or with a vision model for cropped text, overlapping elements, and adequate contrast.
• L3 playback: after downloading the final cut, actually play it and verify that total duration matches the audio duration, that there are no black frames or audio pops, and that subtitles align with the voice.

8.2 Common problems checklist

Symptom	Typical root cause	Fix
Text is cropped	No wrapping or truncation handling	Limit character count in the contract; add uniform ellipsis logic in the renderer
Font suddenly changes	Render machine lacks the specified font	Bundle fonts with the project; validate at build time
Animation skips or is discontinuous	Real time or timer-driven code	Enforce purely frame-functional code; block it in static checks
Audio and video out of sync	Rendering before voiceover	Drive shot frame counts from audio duration
Color tone differs across shots	Different color spaces or compression settings	Unify render and encoding parameters

8.3 Versioning and reproducibility

Put the storyboard, component code, asset manifest, and runtime version all under version control, and record the corresponding build number and parameters in the final cut’s metadata. Being able to reproduce a problem with one click is the most concrete advantage this route has over “black-box generation.”

9. Cost Structure and When to Use It

9.1 What makes up the cost

The pipeline’s cost breaks down roughly into four parts:

• Inference cost: billed per input/output token, directly related to storyboard count, contract length, and number of retries;
• Render cost: CPU/GPU time, proportional to final duration, resolution, and frame rate, and inversely proportional to cache hit rate;
• Audio cost: billed per synthesized character or per duration;
• Storage and delivery cost: intermediate frames are the big storage consumer, so a lifecycle policy is mandatory.

For optimization priority, start with caching and storyboard template reuse — the biggest payoff with no quality loss. Only then consider lowering draft resolution for fast previews and rendering the final version at full quality once approved.

9.2 Suitable and unsuitable cases

Where this route excels:

• Data visualization and report-style broadcasts;
• Course explanations, knowledge explainers, and product feature demos;
• High-volume production of multilingual versions (swap the copy, reuse the layout);
• Marketing material that must strictly follow brand guidelines.

Where it doesn’t fit:

• Live-action footage, real scenes, and complex physical effects;
• Shots needing cinematic quality and highly realistic camera language.

Those cases are better served by multimodal video generation models or professional 3D rendering pipelines.

10. Comparison with the Multimodal Generation Route

Dimension	Code-driven rendering	Multimodal video generation
Controllability	High: every pixel is determined by parameters	Medium: prompts influence the result, but there is randomness
Text and charts	Precise, crisp, editable	Error-prone; small text tends to break down
Photorealistic imagery	Depends on assets; can’t conjure imagery from nothing	A strength
Batch consistency	High: template reuse	Low: every run differs
Unit cost	Dominated by render time; cacheable	Dominated by inference time; hard to cache
Iteration	Change parameters to change the result; versionable	Regenerate, with an uncontrollable result

The two are not substitutes. A mature approach is hybrid: use multimodal generation for live-action-feeling clips as “shot material,” use code-driven rendering for subtitles, charts, transitions, intros and outros, and the overall packaging, then assemble the final cut through a deterministic compositing process.

11. Recommended Rollout Path

1. Get a minimal loop working: one template, three narration segments, one final cut, to validate that audio-driven timing and audio-video sync hold up;
2. Build up a template library: abstract high-frequency layouts into parameterized scene components driven by the storyboard contract, gradually reducing the share of code the model generates;
3. Add quality gates: wire the L1–L3 checks into the pipeline so that failure at any layer blocks release;
4. Observe and tune: use monitoring data to find the render throughput inflection point, and cache hit rate to measure template reuse effectiveness;
5. Extend the capability boundary: integrate more video-editing tools and audio capabilities on demand, keeping tool parameters narrow and validation strict.

Conclusion

The claim that a text-only model can’t make video only holds if you insist it generate pixels directly. Once video is re-modeled as “a sequence of frames on a timeline plus deterministic functions of parameters,” the model’s strengths come into play: writing code, following structured contracts, and generating at scale according to rules.

The real value of this route is not replacing multimodal generation but filling in its weakest spots — precise text, controllable layout, reproducible processes, and estimable cost. Deciding which parts of the picture should be determined by parameters and which by a generation model is the first step in designing this kind of system, and the most important one.