Generate a 3D Short Film from a Single Prompt: Code-Driven Modeling, Camera, and Rendering

From Writing Code to Directing Films

Over the past year, “generating video with natural language” has become a hot topic in developer communities. Different from the idea of “output a clip directly from one prompt,” there is a more unconventional route: an agent writes an entire set of graphics-drawing code from a description, the computer computes each frame, the sound is synthesized, and everything is packaged into a short film. This route has gone through three clear stages.

1. The frame-by-frame code-drawing stage. The earliest work that drew attention was 2D animation experiments. Developers had agents write animations using front-end drawing APIs, producing risograph-style shorts: the picture is drawn frame by frame in pure code, resembling manual color registration with an ink roller, with visible grain and slight misregistration. Some developers later reproduced this style. According to public information, one such solution outputs roughly 3,500 lines of 2D canvas drawing code, implementing 26 illustration scenes inside a single web file and generating the soundtrack in real time with the browser’s built-in audio synthesis API. This is a typical case of “code as picture.”

2. The code-narrative stage. A month later, community work appeared in a form much closer to a short film. The creator’s request to the agent was quite open-ended: make a video about an ordinary workday, with an eerie atmosphere, where both picture and sound are done entirely through programming, leaving the choice of specific tools to the model. According to the author, this roughly one-minute finished piece took about 45 minutes end to end, including the final render, once high-intensity reasoning mode was enabled. The technical components involved a numerical computing library, a signal processing library, a vector graphics library, a local text-to-speech engine, and audio/video muxing and transcoding tools. This shows that agents can already autonomously select and orchestrate a cross-domain tool chain.

3. The code-3D stage. The ceiling of 2D drawing is obvious, so agents began to “write renderers” directly. The model generates 3D scene description files from a prompt, or writes its own camera, lighting, and sampling code, rasterizing or ray tracing frame by frame to output image sequences, then mixing them with code-synthesized voice-over and sound effects and packaging them into a video file. By this point, “modeling, camera work, and rendering” are genuinely all done by code.

This route is worth taking seriously because it turns video production from “dragging a timeline” into “writing a program.” The following sections break down how it works.

The Principle: Why Writing Code Can Replace Editing Software

The physical essence of a digital video is not complex: a sequence of static frames ordered in time, plus a time-aligned audio track. Any tool that generates video ultimately cannot avoid two things: computing the pixels of each frame, and packaging those pixels under a unified timecode.

In a programmatic solution, both are translated into code problems:

• The picture is determined jointly by geometric descriptions, camera parameters, materials, and lighting; given the same parameters, it computes exactly the same pixels.
• Time is determined by the frame rate and frame index; frame n corresponds to time t = n / fps, and animation is simply a function of parameters that change with t.
• Sound is a sequence of waveform samples, which can be synthesized (sine waves, noise, envelopes, filters) or generated by a text-to-speech engine, likewise arranged along the time axis.

So for an agent to “film” a short clip is essentially to complete a function: the input is a text description and a shot structure, and the output is a sequence of frames plus an audio track, packaged into a file.

Compared with generating frames directly using neural networks, programmatic rendering has four characteristics:

• Determinism and reproducibility: the same script and random seed produce byte-identical output, which is convenient for version control and regression testing.
• Pixel-level control: camera position, focal length, motion blur, light intensity, and color transforms can all be specified precisely; changing one parameter and re-running is enough, which suits batch production of variants.
• A different cost structure: the main consumption is CPU/GPU compute and agent reasoning time, with no need for massive training data.
• A clear style ceiling: it performs well on geometry, illustration, infographics, data visualization, low-poly, and stylized art, but struggles with “photorealistic humans plus complex physical detail.”

Understanding these four points lets you judge when to use it and when not to.

The End-to-End Pipeline: Five Gates from One Sentence to a Finished Film

3.1 The Storyboard: Turning Natural Language into a Structured Intermediate Representation

A single-sentence description cannot be rendered directly; there must be a “storyboard” layer in between. This is actually the step agents excel at: translating “an ordinary workday, eerie atmosphere” into a shot list — shot number, duration, shot size, camera position, movement, scene elements, sound effects, and subtitles. This intermediate representation is best captured as a machine-readable configuration that humans can also review and modify easily.

{
  "project": "An Ordinary Day",
  "fps": 24,
  "resolution": [1920, 1080],
  "duration_sec": 60,
  "shots": [
    {"id": "S01", "dur": 4.0, "shot_size": "wide",  "move": "dolly_in",   "set": "office_dawn",   "sfx": ["hum_low"]},
    {"id": "S02", "dur": 3.0, "shot_size": "close", "move": "static",     "set": "desk_monitor", "vo": "Day seven. Everything as usual."},
    {"id": "S03", "dur": 5.0, "shot_size": "medium","move": "handheld",   "set": "corridor",     "sfx": ["footstep","flicker"]}
  ]
}

Two engineering principles are worth following: decouple configuration from the renderer, so the config drives a generic renderer and adding shots never changes render code; and converge parameters into discrete tiers, taking duration, camera position, and camera movement from finite enumerations to reduce the agent’s degrees of freedom, which significantly lowers the error rate.

3.2 Modeling and Materials: Geometry Descriptions Must Be “Program-Readable”

3D modeling does not have to call modeling software; geometry can be generated directly in code: primitives (cube, sphere, cylinder), swept volumes (lofting along curves), procedural textures (noise functions for wall stains, wood grain, grain), and implicit surfaces defined by distance fields. For stylized imagery, this layer offers the best value: a single noise-perturbed plane can carry the entire texture of an eerie corridor.

Practical tips:

• Prioritize parameterized component reuse; the same “room,” “desk,” and “corridor” components are shared across multiple shots, avoiding rewriting geometry for each shot.
• Tie polygon count to resolution; use low-poly models for distant shots and high-poly for close-ups to control the memory peak.
• Use physically based rendering parameters for materials (roughness, metalness), which is easier for achieving a consistent look than hand-writing lighting formulas.

3.3 Camera Work: The Camera Is a Curve of Parameters Over Time

The precise meaning of “camera work” in code is: given a shot’s duration and its start and end composition, generate a smooth camera trajectory. Camera state includes at least position, orientation, and focal length (field of view). The recommended approach is to set keyframes, then use spline interpolation together with ease-in/ease-out functions:

def smoothstep(u):                # ease in/out to avoid mechanical constant-speed motion
    return u * u * (3 - 2 * u)
def catmull_rom(p0, p1, p2, p3, u):   # smooth interpolation between keyframes
    return 0.5 * ((2*p1) + (-p0 + p2)*u +
                  (2*p0 - 5*p1 + 4*p2 - p3)*u*u +
                  (-p0 + 3*p1 - 3*p2 + p3)*u*u*u)
def camera_at(t, shot):
    u = smoothstep(min(max(t / shot.dur, 0.0), 1.0))
    pos = catmull_rom(*shot.pos_keys, u)
    target = catmull_rom(*shot.target_keys, u)
    return look_at(pos, target, up=[0, 1, 0]), lerp(shot.fov[0], shot.fov[1], u)

A few details that easily go wrong:

• Orientation interpolation should use quaternions or interpolate-then-normalize; interpolating Euler angles directly produces flips and the “banking effect.”
• Handheld feel is low-frequency noise added to position and orientation, and the noise must use a fixed seed; otherwise each render produces a different jitter trajectory.
• Depth of field and motion blur should have their sample count converted from the shutter angle; for example, 24 fps with a 180° shutter corresponds to a 1/48-second exposure, meaning multiple time samples are taken within the shutter interval per frame and averaged. Skip this step and fast camera moves will show obvious flicker and jumpiness.

3.4 Rendering: Frame-by-Frame Output and Encoding

The shape of the render loop is quite plain:

def render(config):
    scene = build_scene(config)          # assemble geometry, materials, lighting
    audio = build_audio(config)          # synthesize/voice-over/SFX, unified sample rate and total duration
    for shot in config.shots:
        for i in range(int(shot.dur * config.fps)):
            t = i / config.fps
            cam = camera_at(t, shot)
            frame = render_frame(scene, cam, config.samples)  # rasterize or path trace
            write_frame(frame, index=global_index)            # write frame sequence to disk or stream to encoder
    mux(frames_dir, audio, out="final.mp4", fps=config.fps)

Engineering details to watch:

• Color management: do lighting calculations in linear space, then apply tone mapping and sRGB encoding at the end; outputting directly in linear space makes shadows dead black and highlights overexposed, which is the most common source of “dirty-looking” images for beginners.
• Random sampling must carry a seed (per-frame seed = base seed + frame number); otherwise distributed rendering stitching will flicker.
• Fix the audio total length first, then align: voice-over, sound effects, and music are placed by absolute timecode, and everything is resampled to the same sample rate before output (commonly 48 kHz) to avoid audio/video desync after muxing.
• Specify encoding parameters explicitly: frame rate, pixel format, and bitrate or quality tier should be written into the config rather than relying on tool defaults.

3.5 The Agent’s Real Role in This Pipeline

To be clear, the model is not “magically producing a film.” It handles four things: semantic parsing (mapping the description to shot parameters), code generation (writing the renderer and composition scripts), debugging and fixing (reading error messages, viewing frame screenshots, locating issues, and modifying code), and aesthetic selection (choosing a style-consistent combination among candidate shots). Therefore, the higher the reasoning intensity and the more iterations allowed, the better the finished result tends to be — the earlier “45-minute film” case is essentially the product of a great deal of self-correction.

Engineering: From “Running Once” to “Stable Mass Production”

The gap between running once and stable delivery is a full engineering system, with at least four layers:

1. The determinism layer: fixed random seeds, fixed dependency versions, and fixed font and asset versions. Render results must be reproducible, or regression testing is impossible.
2. The task orchestration layer: split “storyboard generation → scene assembly → frame-by-frame rendering → audio synthesis → muxing” into independently retryable stages driven by a message queue, so a single failed frame only reruns that frame; name artifacts by content hash, which naturally supports resume-from-breakpoint and incremental rendering.
3. The elastic compute layer: rendering is a typical parallelizable, stateless workload. After containerization, it can be sliced by shot or by frame and distributed to elastic compute instances, with object storage relaying frame sequences and finished films, a logging service collecting render times and failure reasons, and monitoring and alerting tracking queue backlog, failure rate, and the cost curve. For one-off short-film tasks, elastic scaling is far more economical than an always-on cluster.
4. The quality gate layer: before muxing, automatically validate that frame count = duration × frame rate, that audio track length matches the video, that there are no all-black frames or empty scenes, and that resolution and encoding format meet delivery requirements.

Compared with neural generation, the programmatic route has a clear advantage in batch consistency: the same template with a different dataset or different copy can produce hundreds of style-consistent short clips, which is very hard to achieve with manual editing.

Evaluating Results: Don’t Just Look at “Does It Look Real”

It is recommended to measure output quality with three categories of metrics:

• Technical metrics: render time per frame, failure rate, artifact size, audio/video duration deviation (at the millisecond level), and frame rate consistency.
• Visual metrics: use perceptual similarity or structural similarity to measure the stability of multiple renders of the same shot; use no-reference image quality assessment methods to detect overexposure, noise, and color banding.
• Manual review: whether the storyboard tells the story clearly, whether the camera work serves the emotion, and whether the voice-over and subtitles are in sync — this part must still be gated by people.

The key engineering insight is that the quality bottleneck of programmatic rendering is usually “consistency” rather than “single-frame beauty.” Characters, scenes, and lighting jumping between shots hurt the viewing experience more than one frame not being pretty enough. The solution is to build a shared scene and style configuration (a unified palette, a unified lighting scheme, a unified material library) and have the agent reuse it during generation, rather than creating independently for each shot.

Capability Boundaries and Selection Advice

This route suits: data visualization shorts, science and educational animation, product principle demonstrations, UI interaction demonstrations, stylized illustration animation, and batch-templated short videos.

It does not suit: highly realistic real-person performance, complex fluid and cloth detail, and film-grade visual effects with extremely high physical realism. These scenarios are better served by traditional 3D engines or neural rendering solutions.

If you plan to get hands-on, the recommended minimum viable path is: first use a 2D canvas to validate the “description → storyboard → frame-by-frame picture” closed loop, and stabilize the basic constraints of frame rate, duration, and muxing; then introduce 3D cameras and lighting to produce a ten-second, three-shot sample; and finally complete the audio chain (synthesized sound effects + text-to-speech + background audio) and engineering orchestration (message queue, object storage, logging, and monitoring) to gain mass-production capability. Going further, you can move the rendering step to the cloud for elastic scaling: a container platform handles render tasks, object storage accumulates frame sequences and finished films, and a logging service with monitoring and alerting ensures task observability, using the general compute and storage offered by cloud platforms such as Baidu AI Cloud on a pay-as-you-go basis, avoiding long-term resource occupation for a one-off task.

It is worth emphasizing that AI’s value here is not “making decisions for you,” but lowering the barrier to “writing code to make films”: creators describe their intent in natural language, and the agent translates that intent into executable, verifiable, reproducible engineering code. Once you understand the constraints of each layer of this pipeline, you can turn “generating a short film from one sentence” from a demo effect into a deliverable production capability.