Evaluating AI Agents: Balancing Performance and Cost in Report Generation

I. From Spec Sheets to Real Tasks: Why Make Models Write Research Reports?

The release frequency of new Large Language Models (LLMs) has accelerated to a weekly cadence. For developers and enterprise users, the real challenge is no longer a lack of information, but rather quickly answering one critical question amidst the dense release schedule: Is this model effective and affordable for my specific business tasks?

Instead of endlessly comparing marketing claims, it is far more effective to design a complex task that closely mirrors real production requirements and simply let the model “get to work.” Generating an industry research report serves as an excellent testing ground. It simultaneously tests web search integration, multi-source data synthesis, long-context comprehension, structured writing, and factual consistency—covering nearly all key dimensions of an AI Agent’s capabilities.

Following the recent release of a new generation of agent models, the evaluation community adopted this exact approach: having the model independently complete a research report, and then assessing the balance between performance and cost using public benchmark data. This article abstracts this practice into a reusable methodology.

II. Why Agent Evaluation is Hard: From Chatting to Doing

Traditional static benchmarks primarily consist of multiple-choice questions and Q&A pairs, measuring single-turn knowledge capabilities. Agent tasks, however, present three significant differences:

• Multi-step Execution: Writing a single research report might involve over twenty model invocation steps—including searching, filtering, extracting, analyzing, and drafting. A failure in any single step will propagate downstream and be amplified.
• Environmental Interaction: The model must utilize tools such as search engines, databases, and code execution. Its capability depends not just on “what it knows,” but on “how well it uses tools.”
• Open-ended Results: The output is a long-form document rather than a standard answer, making it impossible to grade using simple string matching.

Therefore, the standard industry practice is to establish dynamic agent evaluation leaderboards. These platforms place numerous real-world tasks (report generation, data processing, automated operations, etc.) into controlled environments. The primary metrics are the final task success rate and the quality of intermediate processes, while simultaneously tracking resource consumption at each step.

III. Designing the “Report Writing” Evaluation Task: Reproducibility is Key

A reproducible evaluation task definition typically contains four core elements.

1. Task Specification: The input is an industry theme or company name. The output is a fully structured research report containing sections like industry overview, competitive landscape, key data, and risk warnings. Constraints include traceable citations, explicit formatting, and defined length limits.

2. Scoring Criteria:

• Factual Accuracy: Extract key claims, verify their consistency with sources, and calculate the hallucination rate.
• Structural Integrity: Section coverage and logical coherence.
• Data Compliance: Citation markup, unit consistency, and unified time frames.
• A three-tier verification approach can be employed: “automated metrics + LLM-based scoring against rubrics + manual spot-checks.”

3. Environment Control: Freeze search snapshots or restrict data sources to prevent score drifting caused by external information changes. Fix random seeds and tool versions. Run the same task multiple times (3 to 5 times) and take the average to eliminate sampling fluctuations.

4. Cost and Latency Logging: Record the input/output token count, tool invocation count, and wall-clock time for each call to form a “cost per task” metric—the protagonist of the next section.

IV. Cost Per Task: The Underestimated Selection Dimension

Many teams only look at the “price per million tokens” when selecting a model. In agent scenarios, this is a trap. The real cost formula is:

Cost Per Task = Σ (Step i Input Tokens × Input Price + Step i Output Tokens × Output Price) + Tool Invocation Overhead

Where Σ iterates through all model calls throughout the task. Because agents roll historical context into subsequent steps, token consumption grows like a “snowball.” The cumulative input for a complex report generation task can easily reach hundreds of thousands of tokens. Two models with identical unit prices might incur total costs that differ by several times for the same task, simply due to different context management strategies.

This leads to two conclusions:

• “Low unit price” does not equal “low total cost.” Selection must be based on “cost per task.”
• Context engineering (pruning, summarization, state externalization) is a cost-reduction lever in itself, holding value equal to model selection.

In public evaluations, a common phenomenon is observed: two models with a performance difference of less than 1% might have a cost-per-task variance of over 60%. In this scenario, choosing the cheaper option is virtually a lossless decision—perfectly illustrating the principle of “looking at cost-effectiveness when performance is similar.”

V. Pareto Frontier: A Joint Perspective on Performance and Cost

By placing candidate models on a two-dimensional “Performance vs. Cost” plane, we can derive a Pareto frontier. If no other model simultaneously achieves “higher performance and lower cost,” then a model is on the frontier. Models off the frontier are considered “dominated”—no matter how strong they are in a single dimension, a better alternative exists.

Performance
 ↑
 │ ● E
 │    ● A      ● D (High price, high performance)
 │      ● B
 │        ● C (Frontier inflection point: Cost-effectiveness sweet spot)
 └──────────────────→ Cost Per Task

The selection rules thus become very clear: only allow models on the frontier into the candidate pool. Slide along the frontier based on the business value function (revenue from performance gains vs. cost budget). Remain cautious of dominated models.

A representative industry view puts the priority bluntly: for models, performance always ranks first—only when performance is sufficient does it make sense to discuss cost-effectiveness. The primary factor of cost-effectiveness is performance; cost is only the secondary factor. This reflects a fundamental engineering reality: in production agents, insufficient performance means task failure, rework, and human fallbacks. These implicit costs often far exceed the token bill. The correct sequence is: first use evaluations to confirm performance meets the business threshold, then select the model with the lowest cost per task within that performance tier.

VI. Engineering Practices: Five Levers to Push Down Costs

1. Task Decomposition and Context Pruning: Split “retrieval-analysis-writing” into independent phases. Each phase should carry only the necessary state, preventing the entire historical context from being repeatedly billed at every step.

2. Prefix Caching: Place fixed system prompts, domain knowledge bases, and tool descriptions in the prompt prefix. Utilize the caching mechanisms of mainstream inference services to bill based on cache hits, drastically reducing the marginal cost of long prefixes.

3. Model Cascading: Route simple steps (format conversion, field extraction) to small models, while complex reasoning and critical writing are handled by large models:

def route(step):
    if step.kind in ("extract", "format", "summarize"):
        return "small-model"
    return "large-model"

4. Result Reuse and State Externalization: Structurally persist tool return results into databases (e.g., object storage, standard databases). Subsequent steps read the summary from storage rather than repeatedly invoking the tool.

5. Budget Guardrails and Early Stopping on Failure: Set a maximum step count and token budget for each task. Degrade or terminate if limits are exceeded, preventing individual failed trajectories from infinitely consuming resources:

class CostTracker:
    def __init__(self, in_price, out_price, budget):
        self.in_price, self.out_price = in_price, out_price
        self.budget = budget
        self.total = 0.0
        self.ledger = []
    def record(self, step_id, usage):
        cost = (usage["input"] * self.in_price
                + usage["output"] * self.out_price) / 1e6
        self.ledger.append((step_id, cost))
        self.total += cost
        if self.total > self.budget:
            raise BudgetExceeded(f"step={step_id}, cost={self.total:.2f}")

By embedding a cost tracker into the agent execution loop, combined with general cloud capabilities like logging services and monitoring alerts (for instance, building an evaluation pipeline on cloud platforms like Baidu AI Cloud), you can form a closed loop: “run evaluations → generate cost bills → locate hotspot steps.”

VII. Making Evaluation a Routine Practice

One-off evaluation conclusions will quickly become obsolete as versions iterate. It is recommended to solidify them into continuous benchmarks: automatically re-run tests when models or prompts change; conduct differential comparisons between old and new versions across the two axes of task success rate and cost per task; regularly recalculate the Pareto frontier of the candidate pool to eliminate dominated options.

VIII. Conclusion

“Having a model write a research report” may seem like a mere experiment, but it actually distills the complete logic of model selection in the agent era: measure performance with real-world tasks close to your business, measure expenses by cost per task rather than unit price, jointly examine both using the Pareto frontier, and actively push down the cost curve through context engineering and model cascading. Performance is the entry ticket; cost is the long-term bill. Only when both are calculated clearly can technical decisions truly land.