Evaluating Generative Models: How to Measure the Quality and Cost-Performance of Generative Tasks

    Generative models and AI agents are increasingly used for marketing assets, training materials, and data interpretation, yet comparing a few outputs side by side captures only part of the picture. This article introduces a four-layer evaluation framework—benchmark task design, process metrics, output quality, and cost breakdown—and demonstrates it with a real multimodal content generation test. It explains how to interpret completion time, call counts, token usage, and cache hits, why cost should be measured by billable volume and cost per effective output unit rather than raw token totals, and how prompt design and orchestration engineering improve both quality and cost-performance. Readers will learn to build reproducible benchmarks and quantifiable, observable evaluation pipelines for generative applications.

    1. Introduction: From “Who Generates Better” to “What Is the Overall Input-Output”

    Generative models and generative AI agents are now widely used for marketing assets, training materials, data interpretation, and more. When people discuss the effectiveness and cost-performance of these systems, their first instinct is often to have a few models generate the same content and compare which visuals and copy look better. This kind of intuitive comparison is valuable, but it tends to obscure two key facts:

    1. A single quality dimension. Complex tasks often require the model to retrieve materials, break down steps, call tools, and verify repeatedly. Output quality is only one part of the result; whether the task can be completed within an acceptable time and budget matters just as much.
    2. A complex cost structure. Public product pricing is quoted per million tokens. Estimating cost from output volume alone often severely underestimates the real bill. In agent-type applications, whether repeatedly read context hits the cache may be the main cost variable.

    To address this, this article proposes a four-layer evaluation framework covering “task design → process metrics → output quality → cost breakdown,” illustrates it with a real multimodal content generation test, and closes with actionable prompt and orchestration engineering recommendations.

    2. Benchmark Task Design: Making Evaluation Reproducible

    Evaluation must first answer the question “what exactly are we comparing?” Recommended tasks should have the following characteristics:

    • Composite capabilities: simultaneously test material retrieval, content understanding, multi-step planning, tool calling, and multimodal output;
    • Clear acceptance criteria: requirements such as duration, resolution, audio-visual form, and style can be checked item by item;
    • Reproducibility: fix model versions, prompts, tool environments, and rendering parameters; run each configuration multiple times and take the median rather than a single run.

    Below is an example prompt. It requires the model to first read public materials and then generate multimodal content, with clear constraints that make item-by-item acceptance easy:

    1. First, consult the public materials in the model's official blog that introduce the new version,
    2. then generate a motion-graphics video introducing the new version's features:
    3. - No audio narration is needed; pair it with upbeat instrumental music;
    4. - Duration is unrestricted, but the content must be thorough and the animation refined;
    5. - Use the specified brand color scheme;
    6. - Output a 4K HD video.

    Note that phrases like “no duration limit” and “refined animation” in this prompt are subjective. In a formal evaluation, they should be translated into objective metrics, such as a maximum duration, a target frame rate, whether the resolution meets the standard, and the number of key knowledge points covered.

    3. Process Metrics: Time and Token Consumption

    Process metrics reflect how the model works, mainly including:

    • Completion time: the end-to-end time from task dispatch to the finished product, influenced jointly by model inference speed, tool response, and rendering queue;
    • Number of calls: the number of round trips between the model and the model service, which reflects task-decomposition granularity and retries;
    • Total tokens: the sum of input and output tokens, where the input side usually accounts for the majority;
    • Pure output volume: the number of tokens the model actually generates, which affects output richness;
    • Average output per call: pure output divided by the number of calls, reflecting the work density of each call.

    The following uses a real test result as an example (data has been anonymized and is only used to illustrate how to interpret the metrics). Four models received exactly the same prompt and all generated motion-graphics videos introducing the new version’s features:

    Metric Model A Model B Model C Model D
    Completion time ~17m 30s ~17m ~20m ~63m
    Number of calls 28 41 79 230
    Total tokens ~5.07M ~6.01M ~11.66M ~37.23M (~98% cache reads)
    Pure output volume ~71K tokens ~58K tokens ~41K tokens ~163K tokens
    Average output per call ~2540 tokens ~1405 tokens ~522 tokens ~709 tokens
    Output specs ~132s / 120.7MB ~156s / 37.7MB ~120s / 12.5MB ~240s / 309.7MB

    3.1 Number of Calls: Distinguishing “Well-Planned” from “Idle Loops”

    Model A completed a high-bitrate ~2-minute video with 28 calls and about 71K tokens of output, averaging about 2540 tokens per call—a typical case of “thorough planning, batch output.” Model C made 79 calls with an average of only about 522 tokens per call, indicating many small trial-and-error steps or corrections.

    The number of calls is neither absolutely good nor bad on its own; it must be judged alongside quality:

    • If increased calls bring a clear quality improvement, it is effective investment;
    • If output does not improve accordingly, be wary that the model has fallen into an idle “read → try → read again” loop during long tasks.

    3.2 Completion Time Is Determined by Multiple Factors

    Completion time combines inference speed, tool-calling overhead, and rework. Model D took about 63 minutes with 230 calls, a large part of which came from read overhead caused by repeatedly loading context, plus verification round trips in a long workflow.

    In user-facing products, end-to-end latency is itself an experience metric. When evaluating, record “model inference time,” “tool execution time,” and “retry time” separately so the bottleneck can be located.

    3.3 Total Tokens: Repeated Reads Are Not New Computation

    A high total token count does not equal a large workload. The ratios of total tokens to pure output for the four models are about 71x, 104x, 283x, and 228x, with the differences coming mainly from how many times context was repeatedly read. Read tokens and output tokens usually carry different unit prices, so the two cannot simply be added together as cost.

    3.4 Output Specifications: Beware the “Bigger Number” Trap

    Model D’s video duration is about 1.8x that of Model A, its file size about 2.5x, and its pure output the highest. Looking at the numbers alone, its absolute input-output seems the highest, but whether it is “worth it” depends on cost per unit output and content density. If the longer duration merely stretches the same knowledge points and slows the pacing, user value may not actually be higher.

    4. Output Quality: Factual Accuracy Is the Baseline

    Evaluation should first look at the factual level. In this test, all four models’ videos showed no obvious errors on key facts such as version features and performance data. This shows that mainstream models already have a fairly solid foundation in “fact-based paraphrasing from public materials,” and the differentiation in model selection comes more from the following aspects:

    • Factual accuracy: version numbers, parameters, release dates, and feature boundaries consistent with the original material, without fabricating undisclosed information;
    • Instruction following: constraints such as audio form, duration, resolution, and color scheme met item by item;
    • Visual expression: whether layout, animation rhythm, and information hierarchy are clear;
    • Style adaptation: the four models’ videos presented distinct styles such as modern rhythmic beat-syncing, academic magazine-style charts, dark-tech diffuse lighting, and black-gold cinematic long takes—there is no absolute better or worse, and the choice should match the use case.

    It is recommended to use a scoring sheet to convert subjective impressions into quantifiable, trackable metrics:

    Evaluation item Weight Evaluation method
    Factual accuracy 30% Cross-check item by item against the source material
    Constraint satisfaction 25% Judge against each constraint
    Visual quality and pacing 25% Multiple reviewers score on a unified scale
    Generation efficiency (time / number of calls) 10% Take the median across multiple runs
    Cost per effective output unit 10% Computed with the cost model in Section 5

    5. Cost Breakdown: Understanding the Real Composition of the Bill

    5.1 Billing Model and Estimation Code

    Mainstream model services usually bill input tokens and output tokens separately, and the input side may further distinguish “ordinary input” from “cache-hit input”; always refer to the pricing published by the provider. Without relying on any vendor SDK, a generic function can estimate task cost:

    1. def estimate_cost(token_usage, unit_price):
    2. """
    3. Estimate the token cost of a single task without depending on any specific vendor API.
    4. token_usage: {
    5. "input": number of ordinary input tokens,
    6. "cache_read": number of input tokens served from cache (cache hits),
    7. "output": number of output tokens,
    8. }
    9. unit_price: unit price for each type (per 1,000 tokens)
    10. """
    11. cost = 0.0
    12. for key in ("input", "cache_read", "output"):
    13. cost += token_usage.get(key, 0) / 1000 * unit_price[key]
    14. return round(cost, 2)

    5.2 A Counterintuitive Conclusion

    Take Model D as an example: it consumed about 37.23M tokens, of which about 98% were cache hits. Under the caching pricing approach common in the industry, the unit price for cache hits is significantly lower than ordinary input, so its actual bill is in the same order of magnitude as Models A, B, and C. This brings two insights:

    • Measure cost by “billable volume,” not “total tokens.” The higher the cache-hit ratio, the more controllable the per-task cost.
    • High cache hit rates may also come with efficiency loss. 230 calls indicate the workflow was cut too finely. Optimization should attend to both cost efficiency and completion efficiency.

    5.3 From Task Cost to Cost per Effective Output Unit

    Evaluating cost-performance cannot rest on comparing the cost of a single task; the following metrics are recommended:

    • Pass rate: qualified outputs divided by total runs. In complex generation tasks, rework and failures often affect the total budget more than a single call’s cost;
    • Cost per effective output unit: total task cost divided by the effective value of qualified outputs, such as video minutes or the number of key knowledge points covered.
    1. def effective_cost(total_cost, qualified_runs, total_runs, output_minutes):
    2. """Compute the cost per effective output unit based on pass rate and output duration."""
    3. if qualified_runs == 0 or output_minutes <= 0:
    4. return None
    5. actual_cost = total_cost * total_runs / qualified_runs
    6. return round(actual_cost / output_minutes, 4)

    If a long generated piece contains factual errors or missing constraints, rerunning it once can cost several times the call fees saved. Pass rate should therefore be included as a core selection metric.

    6. Prompt and Orchestration Engineering: Achieving Both Quality and Cost-Performance

    Based on the evaluation framework above, generative applications can be optimized in the following ways:

    6.1 Prompt Design

    • Stable prefix, dynamic content last. Most inference services reuse the same context prefix across requests. Placing unchanged content—such as system instructions, role settings, and output specifications—at the beginning of the prompt, and variable content—such as retrieval results and temporary variables—at the end, helps improve cache reuse;
    • Quantify constraints. Convert subjective requirements like “thorough” and “perfect animation” into an acceptance checklist, for example “cover N key features,” “each feature includes at least one comparison chart,” and “key transitions use the specified animation type.” Clear standards both reduce rework and give reviewers a basis;
    • Stepwise output. Long tasks can first generate storyboard scripts and knowledge-point outlines, verify them manually or programmatically, then generate and stitch content section by section into the final video, reducing the loss from a single failed generation.

    6.2 Orchestration Engineering

    • Set a call budget per task. Set a maximum number of calls and maximum token usage for each task; when the threshold is exceeded, output interim results and raise an alert rather than letting the process loop indefinitely;
    • Full-process observability. Record each call’s input/output tokens, cache hits, elapsed time, and tool results. On Baidu AI Cloud, outputs can be archived uniformly to object storage, call details written to a logging service, and thresholds set for timeouts, failure rates, and budget overruns through monitoring and alerting;
    • Failure isolation and resumable runs. Split the overall flow into stages such as retrieval, script, storyboard, rendering, and composition, and persist each stage’s results. If a later stage fails, there is no need to rerun stages that already completed correctly;
    • Build an evaluation pipeline. Put prompt versions, model versions, and tool environments under version control, and automatically run regression tests on the benchmark tasks after each change. Also output quality scores and cost reports to avoid misjudging overall effectiveness based on “one good run.”

    7. Conclusion

    Returning to the question of “actual capability and cost-performance,” the core conclusions of this article can be summarized in three points:

    1. Capability assessment must be multidimensional. Factual accuracy is the baseline, instruction following and visual expression set the ceiling, and time and number of calls reflect execution efficiency. All three are indispensable and should be repeatedly verified within a unified task;
    2. Cost-performance depends on cost per effective output unit. Total tokens, time, and number of calls are each easy to misjudge in isolation; only by combining them with cache-hit rate and pass rate can you see the real bill;
    3. Engineering optimization sets the floor. The impact of prompt structure and orchestration strategy on cost, latency, and stability is often no less than the choice of model itself.

    For developers and architects, rather than asking “which model is the best,” it is better to establish reproducible benchmark tasks, quantifiable scoring standards, and a sustainable observability view of costs. Only by making both quality and cost measurable can generative applications thrive steadily in real business.