Generate Videos with Code: A Declarative Rendering Approach and AI-Assisted Practice

1. From Editing Video to Writing Video: A Shift in Production Paradigm

Traditional video production depends on timeline-based editing software: import assets, drag them into place, add effects, export. That workflow is highly efficient for handcrafting a single clip, but the moment the requirement becomes “generate ten thousand videos from one layout,” three weaknesses surface immediately: repetitive work cannot be reused, the process is not data-driven, and the output cannot be version controlled.

Generating video with code solves a different class of problem. It treats video as the output of a program rather than the product of manual operations. That shift brings several direct benefits:

• Different cost structure: rendering runs locally or on your own servers, so there is no subscription pressure from per-seat or per-render-minute pricing.
• Versionable: the picture is source code, so it can be managed, reviewed, and rolled back in a code repository; changing one number does not require reopening a project file.
• Data-driven: product names, prices, customer names, scores, and dates can be injected as parameters, letting one template produce thousands of differentiated videos in bulk.
• Automatable: rendering is a process invocation, so it fits naturally into task queues, pipelines, and scheduled jobs.

The trade-off is equally clear: this approach is a poor fit for freeform creative editing, and a strong fit for scenarios where the layout is fixed and the content varies.

2. The Core Abstraction: Video as a Pure Function from Frame to Picture

Understanding this approach requires accepting one abstraction: a video is an ordered sequence of a finite number of frames, and the drawing result of each frame is determined solely by “the current frame number plus the input parameters.”

Expressed as pseudocode:

// video = f(frame, props) -> picture
function Scene({ frame, props }) {
  const t = frame / props.fps;        // current time in seconds
  const progress = clamp(t / props.duration, 0, 1);
  return renderLayout(progress, props.text);
}

This abstraction imposes three engineering constraints that are also the foundation of everything that follows:

1. No internal state. A component must not retain mutable state between two invocations. The render process is split across multiple processes and may execute frames out of order or repeatedly; any state leakage produces inconsistent frames.
2. Determinism. The same input must produce the same pixels. This rules out unseeded randomness, direct reads of the system clock, and reliance on asynchronous race conditions.
3. Explicit time. Every animation must be derived from frame, not triggered by a “play for a while, then fire” timer.

3. Technology Stack Layers: Who Is Responsible for What

Solutions of this kind usually consist of four layers, and understanding the layering significantly reduces debugging cost:

Layer	Responsibility	Focus
Scene description	Writing DOM/SVG/Canvas structures with a component-based frontend framework	Layout, fonts, graphics, styling
Time driving	Converting frame numbers into time, progress, and eased values	Frame rate, duration, interpolation curves
Render capture	Screenshotting frame by frame with a headless browser to guarantee pixel consistency	Asset waiting, device pixel ratio, font loading
Encoding and packaging	Encoding the frame sequence into a video file, or muxing an audio track	Codec parameters, color space, bitrate

The key point: the picture is described with web technology, rendering relies on a headless browser, and output relies on an encoder. Using web technology means nearly all web animation capabilities can be reused directly — CSS transitions, SVG paths, per-pixel Canvas drawing, WebGL shaders, and web font typography.

4. Setting Up the Project: From Scaffolding to a Runnable Preview

The standard starting point is to create a project with the scaffolding provided by the ecosystem. The interactive setup typically asks about the template type, whether style support is needed, and whether to install the specification/instruction files aimed at AI coding assistants. The last item is often overlooked, yet it has a large impact on the AI-assisted quality discussed later.

Once dependencies are installed, running the local development command opens a visual debugging interface with a typical layout:

• Left: a list of renderable units (Compositions), each corresponding to an entry component.
• Center: a live preview window where you can drag to jump to any frame.
• Bottom: a timeline with frame-by-frame stepping controls.
• Export entry point: triggers a local render and writes out a video file.

The most important capability during development is frame-level debugging: jump to the offending frame and inspect that frame’s DOM and styles directly, instead of replaying repeatedly and waiting for the problem to reproduce.

5. Organizing Scenes and Designing Parameters

Define each video as a render unit with metadata, where the metadata decides “how to render”:

{
  "id": "new-year-countdown",
  "fps": 30,
  "durationInFrames": 300,
  "width": 1920,
  "height": 1080,
  "defaultProps": { "year": 2026, "from": 10, "theme": "confetti" }
}

Design points:

• Express duration in frames rather than seconds. 300 frames is 10 seconds at 30fps and 5 seconds at 60fps; using frames as the unit avoids duration drift when the frame rate changes.
• The outer container only orchestrates; child components do the drawing. Splitting a countdown into a background layer, a number layer, a particle layer, and a caption layer makes each piece independently replaceable and reusable.
• Converge parameters into a single props object. For bulk production, that object maps directly to one data record that can be read from a spreadsheet, a database, or a message queue.
• Fixed dimensions and fixed layout. All outputs from one template must share the same layout, so that validation, review, and sampled frame QA can be automated.

6. Implementing Animation: Interpolation, Easing, and Time Offsets

Animation is essentially a “mapping from progress to numeric value.” Three common techniques:

1) Linear normalization plus an easing curve

const linear = (frame, start, end) => clamp((frame - start) / (end - start), 0, 1);
const easeOutCubic = p => 1 - Math.pow(1 - p, 3);
const easeOutBack  = p => 1 + 2.70158 * Math.pow(p - 1, 3) + 1.70158 * Math.pow(p - 1, 2);
const scale = 0.6 + 0.4 * easeOutBack(linear(frame, startFrame, startFrame + 12));

The overshoot segment of easeOutBack is precisely where the “bounce” comes from, and it is more controllable than hand-written keyframes.

2) Sequencing and time offsets

Duplicating one element several times and adding a fixed offset to each copy’s start time produces staggered, wave, and waterfall effects:

const items = Array.from({ length: 20 }, (_, i) => (
  <Confetti key={i} delay={i * 3} seed={1000 + i} />
));

3) Deterministic pseudo-randomness

Falling, flickering, and particle effects need to look random, but rendering must be reproducible, so you need a pseudo-random number generator with its own seed, sampling by “seed + index”:

function mulberry32(seed) {
  return function () {
    seed |= 0; seed = seed + 0x6D2B79F5 | 0;
    let t = Math.imul(seed ^ seed >>> 15, 1 | seed);
    t = t + Math.imul(t ^ t >>> 7, 61 | t) ^ t;
    return ((t ^ t >>> 14) >>> 0) / 4294967296;
  };
}
// Each particle's position is uniquely determined by seed + index,
// so rendering any single frame in isolation yields an identical result.

7. AI-Assisted Coding: Give Your Coding Assistant a Specification

Having an AI coding assistant produce scene components directly from natural language is currently the highest-leverage path. A sample prompt can be written as explicit acceptance criteria:

Build a New Year countdown scene, 30fps, 300 frames total:
1. The number counts down from 10 to 0, switching once per second.
2. Each number switch uses a scale-bounce entrance (overshoot no more than 15%), with an entrance duration of 12 frames.
3. The background is falling confetti, 80 particles, using seeded pseudo-randomness to guarantee frame-level determinism.
4. When the countdown reaches 0, fade in the "Happy New Year" caption and hold it until frame 300.
5. All animation must be derived from `frame`; no timers, no reading the system time.
6. Use a common system sans-serif font to avoid rendering inconsistencies caused by external font loading.

Items 5 and 6 are constraint clauses, and they often determine success or failure more than the requirement description itself does. This is also the value of installing the “specification files” during scaffolding: such files are a community-maintained checklist of best practices covering directory organization, animation patterns, asset preloading, and common anti-patterns. Provided to the AI as context, they noticeably reduce code that “runs but renders unstably.”

The recommended human–machine collaboration loop is generate → preview → locate a single frame → fix, rather than one long prompt that generates all the code at once: first have the AI produce the skeleton and static layout and confirm the layout in preview; then add animation layer by layer, changing only one layer at a time; when something breaks, feed back the frame number, the expected effect, and the actual behavior together, so the correction scope stays as small as possible.

8. Common Pitfalls and a Troubleshooting Checklist

Most problems with this approach cluster around “preview looks fine, export is broken,” and the root causes are almost always non-determinism and timing:

• Screenshotting before assets are ready. Images, videos, fonts, and SVG references must finish loading and decoding before rendering, otherwise you get blank frames or font fallback. For external assets, prefer inlining or localizing them.
• Unseeded randomness. Calling a global random function directly makes frames inconsistent when rendering in parallel across processes, which shows up as particles “jumping.”
• Depending on asynchronous state. Issuing a request inside a component and waiting for the result guarantees inconsistent data during frame-level debugging. The correct approach is to prepare all data before rendering and pass it in as props.
• Ignoring device pixel ratio. Sharp preview but blurry export usually means the render resolution and pixel ratio settings are not aligned.
• Concurrency set too high. Headless browser instances are memory-heavy; too much concurrency causes frame render failures or timeouts, so benchmark against available machine memory before choosing a value.
• Missing fonts and glyphs. Special characters, emoji, and CJK glyphs render noticeably differently across environments and must be validated in the target rendering environment.
• Frame boundary errors. Animation that finishes only on the last frame, or an end frame beyond the total frame count, causes the tail frame to flicker; explicitly agree on a closed or half-open interval.

One low-cost self-check: extract frame 0, a middle frame, and the last frame for manual comparison, and sample a few additional frames for pixel-level comparison to verify rendering determinism.

9. Scaling from Single-Machine Rendering to Production

When the rendering demand grows from “a few per day” to “tens of thousands per day,” a local single machine is no longer enough. At that point, turn rendering into a stateless render service and connect it to general-purpose cloud capabilities for smooth scaling:

1. Task-based execution. Turn each render request into a task record containing the template identifier, parameters, priority, and status, which makes retries and idempotency straightforward.
2. Containerized deployment. Package the rendering environment together with the code into an image to keep dependencies, fonts, and browser engine versions identical — this is the fundamental way to eliminate “rendering inconsistency caused by environment differences.”
3. Elastic compute. Host render worker nodes on a container platform, scale automatically based on queue length, and scale in during idle periods to control cost.
4. Artifact storage and distribution. Write output files to object storage and serve them to business systems for download or embedded playback through accelerated distribution.
5. Asynchronous decoupling. Route task dispatch and status callbacks through a message queue so rendering never blocks the main business path.
6. Observability. Connect log services with monitoring and alerting, focusing on render success rate, per-item duration distribution, queue wait time, and clustering of failure reasons, with dedicated alerts for high-frequency failures such as “missing font” and “asset timeout.”

Two additional practical optimizations: render only changed segments (render a long video in parts and concatenate them) and cache unchanged layers (render the static background once and repaint only the animated layers). Both can substantially reduce compute consumption.

10. Scope and Technology Selection

Scenarios that fit code-generated video: data-driven bulk short videos, automated e-commerce and product card output, screen recordings of reports and dashboards, data visualization animations, overlaying subtitles and packaging layers, generating test and demo material, and automated output pipelines that need deep integration with business systems.

Scenarios that do not fit: brand films centered on subjective creativity, live-action edits that need extensive manual refinement, and shot selection that relies heavily on human aesthetic judgment. A common combination in practice is: creative shots are produced manually, while the scalable, template-driven portion is handed to code rendering, with the two connected through shared assets and specifications.

11. Conclusion

The value of writing video with code is not replacing editing software; it lies in bringing video into a software engineering system: frames are reviewable, parameters are configurable, builds are reproducible, artifacts are batch-producible, and cost is predictable. Master the core abstraction of “a pure function from frame to picture,” treat determinism and asset timing as first-class constraints, and use AI coding assistants to generate and iterate component code against a specification, and you can build a stable video production line at very low tooling cost. Once volume grows, simply make the render process stateless and hand it off to elastic compute plus general-purpose cloud capabilities such as object storage, logging, and monitoring, and the architecture evolves smoothly.