LLM-Generated Motion Graphics Videos: Real-World Benchmark of Cost, Latency, and Caching
This article analyzes a same-task benchmark across four large language models (anonymized as A, B, C, and D) generating code for a motion graphics video, comparing completion time, API call counts, token usage, and final video specs. It also explains how context caching affects the final bill, and offers practical guidance on model selection and cache optimization. Key caveats: the original test did not disclose model versions or call parameters, so conclusions apply only to this dataset and should not be generalized as any model's universal capability.
1. Test Task and Measurement Definitions
The test prompt asked each model to first read a technical blog post, understand its technical details and benchmark data, then generate code for a motion graphics video introducing that technology. The prompt explicitly required no voice-over but specified upbeat instrumental music. Duration was unrestricted, but the video had to clearly explain the technical principles, feature smooth animations, use a specified color palette, and meet 4K resolution standards.
This task simultaneously tests three capabilities: information extraction, long-code generation with multi-turn debugging, and the aesthetic ability to translate abstract concepts into visual rhythm. There is no single correct answer, so any comparison must weigh efficiency, cost, and final output quality together.
The metrics in the original data are defined as follows:
- Completion time: total time from submitting the first prompt to obtaining a runnable final video.
- Call count: number of model requests issued during the task.
- Total tokens: combined input and output tokens.
- Output tokens: tokens generated by the model.
- Average output per call: output tokens divided by call count.
- Video specs: video duration and file size.
The “4K” criterion also needs a clear acceptance method. When reproducing this test, you should record the resolution (3840×2160), frame rate, and video codec, and actually play back the video to verify quality. The original data lacks these details, so we cannot confirm whether all four videos strictly meet the 4K standard.
2. Benchmark Results for the Four Models
| Model | Completion Time | Calls | Total Tokens | Output Tokens | Avg Output/Call | Video Duration | Video Size |
|---|---|---|---|---|---|---|---|
| A | 17 min 32 s | 28 | 5.07M | 71,104 | ~2,540 | 132 s | 120.7 MB |
| B | 16 min 53 s | 41 | 6.01M | 57,624 | ~1,405 | 156 s | 37.7 MB |
| C | 19 min 51 s | 79 | 11.66M | 41,234 | ~522 | 120 s | 12.5 MB |
| D | 62 min 43 s | 230 | 37.23M | 163,078 | ~709 | 240 s | 309.7 MB |
Output efficiency is defined as output tokens divided by total tokens. By this measure, A, B, C, and D score approximately 1.40%, 0.96%, 0.35%, and 0.44%, respectively. This ratio only reflects the share of generated content in total token usage — it says nothing about code quality or video quality. C’s low ratio is mainly because input context dominates its token count. The original data does not break down the billing between input and output tokens, so this interpretation needs to be verified against the itemized bill.
Video duration and file size should not be treated as direct quality metrics either. D’s video is the largest, but size also depends on bitrate, compression settings, and visual complexity. B’s video is longer than A’s yet much smaller. A proper evaluation should use uniform criteria, such as factual accuracy, narrative pacing, and adherence to the required color palette.
The original test claimed that the technical facts and benchmark figures in all four videos were error-free, but the sampling scope and methodology were not disclosed. To adopt that conclusion, you should cross-check every number in the videos against the source article and record the number of errors.
Visual style descriptions are inherently subjective: B favored large academic-journal-style charts; A had a tight, modern rhythm; C used a dark tech aesthetic with diffuse lighting; D went with a black-and-gold style with long shots. For a formal comparison, it would be better to have multiple reviewers score all videos on the same rubric.
3. Completion Time vs. Call Count
B finished fastest with 41 calls. A made only 28 calls and took just slightly longer. D made 230 calls and took over 60 minutes. A high call count does not necessarily mean slow — what matters is the response time and wait time per call.
D’s average output per call was only about 709 tokens. This could mean it adopted a generate-segment-by-segment, fix-segment-by-segment strategy, or it could relate to the model’s per-call output limit. The original data includes no call logs, so this is only one plausible explanation. A way to verify it is to inspect each request’s input length, output length, and latency.
Additionally, multi-turn calls repeatedly resend prior context, so total token usage grows quickly with the number of calls. This is where caching becomes relevant.
4. How Caching Affects Your Bill
Context caching works like this: when multiple requests share the same prefix, the server can reuse the already-computed prefix, reducing redundant computation. In multi-turn code generation, system instructions, prompts, and previously generated code recur in every turn, so caching delivers noticeable savings.
The original data shows that 98% of D’s tokens hit the cache, with an actual bill of roughly 3.4 RMB. A bill should be calculated against the pricing table using this formula:
Bill ≈ uncached input tokens × standard input price+ cached input tokens × cached input price+ output tokens × output price+ cache storage fee (if any)
The original text does not specify whether the 98% refers to input tokens or total tokens. If calculated against the 37.23M total, roughly 36.49M tokens hit the cache and about 740K did not. However, output tokens typically do not go through input caching, so the actual hit rate should be recalculated based on input tokens only.
Caching reduces input costs for repeated prefixes. It does not reduce output token costs, nor does it shorten generation time. The savings depend on three factors: the cache discount rate, the hit rate, and the input-to-output ratio. Without these data points, no definite savings percentage can be stated.
To improve cache hit rates, consider the following:
- Place fixed instructions, role definitions, style guidelines, and background documents at the very beginning of the prompt.
- Avoid timestamps, random IDs, or per-turn changing fields within the prefix.
- In multi-turn conversations, keep the order and content of historical messages stable; do not rewrite them arbitrarily.
- Log the cached tokens for each request and aggregate daily hit rates.
When choosing a model API or cloud platform, verify the caching billing rules, including minimum cacheable length, expiration time, and whether storage fees apply. Details are available in the official documentation of each provider.
5. Selection Recommendations
Different workloads prioritize different factors. The following directions may help:
- If you need fast responses and want the model to output large blocks of continuous code: prioritize models with fewer calls and higher output per call, such as A or B.
- If you have specific requirements for video duration and file size and can tolerate long wait times: consider a multi-turn, iterative model like D, but be prepared for substantially higher token usage and total cost.
