Evaluating LLM-Driven Video Generation: A Practical, Pipeline-Based Benchmarking Approach

    This article lays out a practical way to evaluate LLM-powered agents that generate videos end to end. Instead of judging a single output, it treats each run as a full programmatic video production pipeline and measures it under unified conditions. It explains how to control variables, define acceptance criteria, and break down where time and cost are actually spent, then separates efficiency metrics (completion time, effective interaction time, tool calls, output per turn) from cost-effectiveness metrics built on token structure and prompt caching. It also shows how to fold cost and quality into a single weighted score, summarizes typical behavioral patterns across models, and offers engineering techniques—cache-friendly prompt layering, plan-first workflows, staged rendering, scripted quality gates, and pipeline orchestration—for reducing cost while keeping quality in check.

    1. From Judging a Single Output to Evaluating Whole Pipelines

    LLM-based agents are moving from “answering questions” to “delivering finished work.” Given a requirement description, an agent can autonomously search for material, write scripts, and call tools to produce a video with background music and motion effects—without step-by-step human intervention. A task like this spans understanding, retrieval, planning, code generation, tool calling, and long-running execution, which makes it a good way to assess a model’s real engineering capability.

    But to answer “how good is the actual capability, and what about cost-effectiveness,” the look and feel of a single run is not enough:

    • A single run has variance: the same model may produce noticeably different results across two executions.
    • “Fast” and “cheap” often can’t be achieved at the same time—for example, an execution strategy that repeatedly calls tools and uses more reasoning rounds takes longer but has a lower cost per unit of output.
    • Whether a finished video looks good is subjective, whereas whether the facts are accurate and whether the specs are met can be measured objectively.

    A more reliable approach is to treat each generation as a complete run of a “programmatic video production pipeline,” and measure its performance with quantifiable criteria under the same prompts, tools, and environment.

    2. Unified Evaluation Criteria: Making Different Runs Comparable

    2.1 Controlling Variables

    • Same prompt: includes all constraints—rely only on the specified reference material, no voiceover needed, licensed royalty-free music allowed for the soundtrack, 4K resolution, complete motion effects, a given color scheme, no fixed duration but the information must be fully conveyed.
    • Same initial context: no extra material or examples provided beyond the prompt.
    • Same tool permissions and runtime environment: the same material acquisition, code execution, file read/write, and rendering/encoding tools, plus comparable compute resources.
    • Aligned billing model: whether each service enables prompt caching by default, and how the scope of cache hits is defined, should be recorded clearly; otherwise the token numbers can’t be compared directly.

    2.2 Unified Timing Convention

    Timing starts when the prompt is submitted and ends when the final video file passes inspection. The time the agent spends writing rendering code, rendering frame sequences, and encoding/packaging should all be counted—not just the stage where the model outputs text.

    2.3 Predefined Acceptance Criteria

    • Hard requirements: resolution, frame rate, encoding format and bitrate range, whether an audio track exists, whether the duration is reasonable.
    • Factual accuracy: zero errors in the core numbers and conclusions from the reference material.
    • Motion and pacing: whether element entrances, transitions, and emphasis serve the content, and whether the switches are natural.
    • Information completeness: whether the key content of the material is covered.

    Only after defining what “qualified” means does a subsequent cost-effectiveness comparison become meaningful.

    3. Breaking Down the Generation Pipeline: Where Time and Cost Go

    A typical agent video generation pipeline usually includes the following stages:

    1. Material acquisition: read the specified reference material, extract titles, core data, and conclusions, and strip out navigation, ads, and other noise.
    2. Content planning: divide into chapters, arrange the shot order, and define the on-screen text, visual elements, and duration for each shot.
    3. Visual design: decide the color scheme, fonts, layout, and motion pacing, and keep them consistent across the whole video.
    4. Programmatic implementation: translate the design into component code in a rendering framework and compute animation state frame by frame.
    5. Audio track production: select music, align to beats, adjust volume, and mix it with the visuals.
    6. Rendering and encoding: render frame by frame at the target resolution and frame rate, then encode to the specified format.
    7. Self-check and iteration: check duration, keyframes, text overflow, black frames, and audio-video sync, and re-render locally where problems are found.

    Steps 1–4 are mainly done by the agent on the model side, steps 5–6 are usually done with tools in the runtime environment, and step 7 is purely deterministic programmatic checking. As a result, the total run time is roughly made up of three parts: the interaction time of model inference and tool calls, the local compute time for rendering and encoding, and the extra overhead from failure retries.

    1. # Example: record key metrics for a single generation task; replace the collection method as needed for your evaluation setup
    2. metrics = {
    3. "prompt": prompt,
    4. "model_id": model_id,
    5. "start_at": start_at, # time the prompt was submitted
    6. "finished_at": finished_at, # time the video passed acceptance checks
    7. "tool_calls": tool_calls, # total number of tool calls made by the agent
    8. "render_seconds": render_seconds, # time spent rendering and encoding
    9. "input_tokens": input_tokens, # total input (prompt) tokens
    10. "cached_input_tokens": cached_tokens, # portion that hit the prompt cache
    11. "output_tokens": output_tokens, # model output (generated) tokens
    12. "video_seconds": video_seconds, # duration of the finished video
    13. "video_size_mb": video_size_mb, # file size
    14. "spec_pass": spec_pass, # whether hard specs passed
    15. "fact_errors": fact_errors, # number of factual errors
    16. }

    With this structured record, the time share of each stage, call density, and token structure can all be reviewed, rather than staying at the level of subjective impressions.

    4. Metrics System: Measure Efficiency and Cost-Effectiveness Separately

    4.1 Metrics for Measuring “Efficiency”

    • Completion time: end-to-end wall-clock time, reflecting actual waiting time; also the most common time metric.
    • Effective interaction time: total time minus rendering/encoding time and failure-retry time; this better reflects the agent’s own decision-making efficiency.
    • Number of tool calls: reflects whether the planning was clear and whether there was too much rework. When the call count is far higher than the minimum number of steps needed to complete an equivalent task, it often means circular groping or repeated rework.
    • Average output per turn: output tokens divided by number of calls, an approximation of how much effective content each interaction round produces. A low value may mean many rounds were only used for probing, reading files, or making scattered fixes.

    4.2 Metrics for Measuring “Cost-Effectiveness”

    The key to cost-effectiveness is seeing the token structure clearly. Total input tokens can reach tens of millions, but if the vast majority hit the prompt cache, the actually billed portion may be only a small fraction; model-generated tokens usually don’t enjoy caching discounts and typically have a higher unit price. Therefore, it’s recommended to record at least the following:

    • Total input tokens, the cached portion among them, and the non-cached portion;
    • Output tokens and their share of all tokens;
    • Local compute time such as rendering;
    • If billed by plan or credits, also record the corresponding deduction relationships.
    1. def estimate_cost(input_tokens, cached_tokens, output_tokens,
    2. rate_in, rate_cached, rate_out,
    3. render_seconds, rate_compute):
    4. """Illustrate cost composition with unit-price parameters; rates should follow the actual billing rules of the chosen service."""
    5. billed_input = input_tokens - cached_tokens
    6. token_cost = (billed_input * rate_in
    7. + cached_tokens * rate_in * rate_cached
    8. + output_tokens * rate_out)
    9. return token_cost + render_seconds / 3600 * rate_compute
    10. # 37 million input tokens in total, 98% of which hit the prompt cache
    11. input_tokens = 37_000_000
    12. cached_tokens = int(input_tokens * 0.98)
    13. print("Non-cached input share:",
    14. round((input_tokens - cached_tokens) / input_tokens * 100, 2), "%")
    15. print("Share of output tokens in all tokens:",
    16. round(163_000 / (input_tokens + 163_000) * 100, 2), "%")

    The output is:

    1. Non-cached input share: 2.0 %
    2. Share of output tokens in all tokens: 0.44 %

    This example shows that a seemingly “astronomical” total token count, when the prompt cache is heavily hit, may have its actual billing concentrated in a very small portion. Conversely, the larger the output and the fewer cache hits, the larger the share of the higher-priced part, and the faster the cost grows.

    1. def delivery_cost(costs, scores, weights):
    2. """Estimate the average cost per 'qualified video' by weighted score."""
    3. scores = [(s * w) for s, w in zip(scores, weights)]
    4. costs = [c / s for c, s in zip(costs, scores)]
    5. return sum(costs) / len(costs)
    6. weights = (0.35, 0.20, 0.15, 0.15, 0.15)
    7. # Illustrative weighted scores between 0 and 1, used to demonstrate the calculation
    8. print(delivery_cost([8.0, 5.5, 4.0], [(0.92, 0.95, 0.85, 0.9, 0.9),
    9. (0.9, 0.9, 0.8, 0.85, 0.8),
    10. (0.95, 1.0, 0.7, 0.8, 0.85)],
    11. weights))
    1. 6.44

    4.3 Combining Cost and Quality into One Metric

    Looking at only time or only cost can lead to a one-sided conclusion. It’s recommended to weight each quality dimension by a predefined weight:

    • Factual accuracy: 35%;
    • Meeting hard specs: 20%;
    • Motion and pacing: 15%;
    • Information completeness: 15%;
    • Visual consistency: 15%.

    A run whose weighted score is below a threshold (e.g., 0.8) is treated as unqualified. On this basis, compute the “average cost per qualified video,” putting cost and quality on the same ruler, and then compare cost-effectiveness.

    5. Horizontal Comparison: Reading Behavioral Differences from the Data

    Applying the above criteria to a multi-model comparison under the same prompt, several typical patterns can be observed:

    • Balanced type: completion time near the median, moderate call count, average output of about 2000–3000 tokens per turn, and the lowest delivery cost per run. The planning is relatively mature with less rework.
    • Intensive type: shortest completion time, but a noticeably higher call count, with average output of only around 1000 tokens per turn. Its advantage is fast iteration and tight pacing; the cost is a higher delivery cost. It suits scenarios sensitive to launch speed with a relatively loose budget.
    • Precise-but-slow type: completion time on the longer side of the middle, the fewest calls, and the highest average output per turn, approaching 1000 tokens. It tends to “think it through before acting,” with the least rework, but doesn’t have an advantage in speed.
    • Trial-and-error type: completion time over an hour, over 200 calls, and average output of only a few hundred tokens per turn. Most of the time goes to repeated searching and trial-and-error, but the prompt cache hit rate is close to full, and billing is concentrated on a small amount of non-cached input and output, so the final bill is actually quite low. This pattern doesn’t apply to models billed proportionally by token where output is the dominant part; the cost would then be determined directly by the huge output volume.

    Also, video duration and file size are not quality metrics. A longer version may have calmer visuals and more relaxed information delivery; a high-bitrate 4K file is larger, but its detail and compression quality are usually better too. When comparing, fix the duration and encoding parameters, or convert within the same spec, to avoid “judging quality by size.”

    A noteworthy phenomenon is that, given a unified fact list and reference material, the finished videos from each approach can achieve zero errors on the core data. This shows that the gap in basic skills is narrowing, and the differences show up more in design orientation, pacing control, and cost structure. The visual styles can be roughly grouped into the following categories:

    • Academic magazine style: chart- and layout-driven, emphasizing information density and a sense of rigor, suited to technical explainer content;
    • Modern rhythmic style: cut on the beat, tight pacing, suited to product launches and social distribution;
    • Dark tech style: diffuse glow and particle effects, with strong atmosphere and immersion;
    • Black-and-gold cinematic style: long takes and a cinematic feel, suited to brand storytelling.

    These differences show that comparing only a single-dimension number easily leads to one-sided conclusions; setting weights according to the delivery scenario is what yields results that can truly guide decisions.

    6. Engineering Optimization of Cost and Quality

    6.1 Design the Context in Layers to Improve the Cache Hit Rate

    • Put stable content such as system instructions, tool descriptions, and design specs at the very front of the prompt;
    • Put volatile content such as the video script and component code at the back;
    • Keep the cache-friendly prefix stable and unchanged, avoiding inserting volatile content like timestamps or random IDs in the middle;
    • Set a cache time-to-live for longer tasks to prevent the cache from expiring during execution, which would cause later rounds to be billed again in full.

    6.2 Plan First, Reduce Ineffective Calls

    After material acquisition is complete, first output a structured storyboard and pass it through programmatic validation before entering code implementation. This can surface omissions and deviations early, and also avoid the repeated rework of the “write a bit, render a bit, fix a bit” style.

    6.3 Render in Stages to Lower Trial-and-Error Cost

    First render keyframes and sample clips at a lower resolution to check layout, motion pacing, and audio-video sync; after confirming everything is correct, do the high-resolution final render. Rendering an entire high-resolution video in one go means that once a problem is found, you can only re-render the whole thing, which wastes compute and lengthens total time.

    6.4 Solidify Quality Gates

    Turn the key inspection items from manual acceptance into scripts, as a mandatory step before delivery:

    1. class QualityGate:
    2. def __init__(self, min_seconds, max_seconds, width, height):
    3. self.min_seconds = min_seconds
    4. self.max_seconds = max_seconds
    5. self.width = width
    6. self.height = height
    7. def check(self, probe, duration, width, height, has_audio, black_frames):
    8. errors = []
    9. if not (self.min_seconds <= duration <= self.max_seconds):
    10. errors.append("Duration is outside the expected range")
    11. if (width, height) != (self.width, self.height):
    12. errors.append("Resolution does not match the target spec")
    13. if not has_audio:
    14. errors.append("Audio track is missing")
    15. if black_frames > 0:
    16. errors.append("Black frames are present")
    17. # Can add checks such as keyframe text overflow and audio-video sync
    18. return errors

    6.5 Chain the Stages Together with a Pipeline

    Split material acquisition, storyboard generation, rendering, encoding, QA, and archiving into independent stages, driven by a task queue and run in containers, with each stage kept idempotent:

    • Failed tasks are retried automatically with backoff;
    • Rendering and QA results are written to object storage;
    • Key checkpoints are connected to logging services and monitoring/alerting;
    • Elastic compute is used to handle rendering peaks.

    7. Conclusion

    Back to the original questions: how do you judge actual capability? How do you calculate cost-effectiveness? The answer isn’t in how impressive any single video is, but in reproducible data under unified criteria. Completion time reflects the experience, call counts reveal planning quality, output volume and cache structure determine the direction of cost, and factual accuracy sets the quality floor.

    For long-chain tasks like video generation, rather than chasing first place in a single metric, it’s better to define weights based on your own scenario:

    • For a fast launch where a higher unit price is acceptable, prioritize completion time and rework rate;
    • For budget-sensitive cases that can tolerate longer waits, prioritize cache-friendliness and output share;
    • For content aimed at public distribution, put factual accuracy and visual standards first.

    When evaluation can be reliably reproduced, costs can be clearly broken down, and quality can be automatically validated, model selection will move from “feels good to use” to “trustworthy data,” and generative video production will change from a one-off demo into a sustainable engineering capability.