Agent-Oriented LLM Evaluation: 6 Real-World Task Scenarios and a Practical Methodology

    As new-generation large models arrive and top scores on third-party leaderboards keep converging, agent-task leaderboards reveal a shift from measuring what a model knows to what it can actually accomplish. This article explains how to design a rigorous, reproducible evaluation for judging whether a model can reliably get tasks done in real business scenarios. It covers the paradigm shift from answer accuracy to task completion, an evaluation setup spanning four agent carriers, a five-dimension scorecard, two hands-on test cases (cross-application information delivery and multi-skill stylized video generation), four additional high-frequency scenarios, and how to turn evaluation into an automated regression pipeline—along with practical guidance on interpreting the results.

    Agent-Oriented LLM Evaluation: 6 Real-World Task Scenarios and a Practical Methodology

    Leading large-model vendors have recently rolled out new-generation models one after another, and discussions around model capabilities have heated up again.

    On continuously updated third-party leaderboards, the scores of top models are already very close—some even tied. A more noteworthy phenomenon: on leaderboards that measure agent task execution, some models outperform their results on traditional benchmarks focused on general knowledge. This is no coincidence. Model capability is shifting from “what it knows” to “what it can accomplish,” so rankings across different dimensions naturally differ.

    The agent-task leaderboards mentioned here evaluate something clearly different from traditional benchmarks. Traditional benchmarks are mostly built from single-turn Q&A items, measuring how accurately a model answers within a specific knowledge domain. Agent-task leaderboards place models in environments closer to real work, where the model must call tools such as retrieval, execution, and file processing, move through multiple rounds of reasoning and operation, and finally deliver a complete “work product.” The measure is whether the task is genuinely completed—not merely whether a single step is correct.

    This article therefore does not rank specific models. Instead, it answers a question with more universal value for enterprises: how to design a rigorous, reproducible agent evaluation to judge whether a model can reliably “get the job done” in current business scenarios.

    1. The shift in evaluation paradigm: from “answer accuracy” to “task completion”

    To understand this shift, we first need to clarify the basic assumptions of the two evaluation types.

    Dimension Traditional capability benchmark Agent task evaluation
    Evaluation target Accuracy of single-turn Q&A End-to-end results of multi-step tasks
    Core assumption Broader knowledge coverage and more accurate reasoning mean stronger ability Each step being correct does not equal overall task success
    Focus Knowledge, reasoning, math, code comprehension, etc. Instruction understanding, task decomposition, tool calling, exception recovery, delivery quality
    Failure mode Wrong or incomplete answers Getting stuck midway, calling the wrong tool, missing steps, corrupted deliverables
    Scoring method Comparing answers against reference answers Artifact acceptance + process auditing

    The essence of agent execution is a “decide—act—observe” loop: based on the current state, the model decides which tool to call next and what parameters to pass; after the tool returns a result, the model revises its plan accordingly. In long tasks this loop typically repeats a dozen to several dozen times, and a tiny deviation in a single decision can be amplified along the execution chain. For example, missing one filter condition during retrieval produces an incomplete table; an error in the table structure then distorts the visualization page generated from that table. A final deliverable that “looks done but isn’t actually done right” is almost never exposed in traditional evaluation, yet it is costly in real business.

    It is also important to recognize that agent capability does not fully belong to the model itself. It is jointly composed of:

    • Model capability: instruction following, long-context understanding, reasoning and planning;
    • Tool ecosystem: number of available tools, tool stability, and documentation quality;
    • Orchestration framework: context management, memory mechanisms, failure-retry strategies;
    • Permissions and security: authorization scope, sandbox isolation, confirmation mechanisms for sensitive operations.

    The same model placed under different tools and frameworks can have vastly different task success rates. A rigorous evaluation should therefore “fix the environment and swap the model,” rather than mistaking framework differences for model differences. For the same reason, any single leaderboard result should be treated only as a reference signal: leaderboard question sets have limited coverage, and difficulty distribution, tool configuration, and scoring standards vary, so they cannot be directly equated with a model’s real level in an enterprise’s own business scenarios.

    This article’s evaluation follows three principles:

    • Artifacts as the standard: task success or failure is determined by automatically verifiable final artifacts, not by whether the answer “seems reasonable”;
    • Both process and result matter: not only whether the task is completed, but whether the path is efficient and free of redundant steps and rework;
    • Cross-validation across multiple entries: the same task is retested on as many carriers as possible to identify issues tied to a specific carrier.

    2. Evaluation environment

    To ensure reproducibility, the evaluation was conducted on the following carriers. To avoid interference, the text below refers to them by capability category.

    Carrier category Form description Main focus
    Coding agent tools Agents running in a developer terminal or IDE that can directly read and write local files and execute commands Code generation and verification, local file operations
    Browser-driven agent execution platform Cloud-hosted agent runtime that can load skill sets and connect to external systems Multi-step orchestration, tool ecosystem, long-task stability
    CLI agent tools Terminal-oriented workflows with text commands and the file system as the main interaction interface Batch file processing, process scripting
    Web-based agent entry Conversational interaction supporting web search, document parsing, and attachment generation Instruction understanding, artifact compliance

    The underlying model used in the evaluation was kept at the same version. Where model versions differed across carriers, the results can only serve as a cross-validation reference and cannot support conclusions about which carrier is better or worse.

    3. Scoring card design: making “completion” measurable

    “Whether a task is completed” cannot rely on the evaluator’s subjective feeling; it should be broken down into verifiable checklist items. This article uses a five-dimension scorecard, with acceptance criteria defined before each task type begins and scored accordingly.

    Dimension Focus Typical verification method
    Task completion (40%) Whether each sub-step is carried out and artifacts are complete Whether files exist and can be opened, whether each key step has a corresponding execution record
    Artifact quality (20%) Whether content is accurate, structure is clear, format is compliant Spot-checking data against sources, checking whether fields and units are complete
    Process reasonableness (20%) Whether the path is efficient and rework is reduced Counting tool calls, retries, and manual interventions
    Instruction following (10%) Whether all constraints are met Comparing against hard requirements one by one: count, topic, format, delivery target, etc.
    Exception handling (10%) Whether it can self-correct when facing obstacles Observing whether the retry strategy and alternatives after a failure are reasonable

    The total is 100 points, weighted by the dimensions each case actually involves. The judgment rules are defined as follows:

    • Pass on first try: all hard checks are met on the first execution;
    • Pass after correction: met after prompting or retry;
    • Fail: deliverables are missing, wrong, or unusable.

    When recording each case, save the input prompt, tool-call trace, final artifact, and scoring details. This makes it easy to compare models side by side and to locate capability regressions during later regression testing.

    4. Test case 1: a cross-application multi-step information delivery task

    4.1 Task design

    This case simulates the typical office workflow of “from public information to reporting materials,” split into three subtasks of increasing difficulty:

    1. Log in to a film community website, search for recent popular films, filter 10 by combining popularity, rating, and release date, and generate a spreadsheet;
    2. Design a visualization web page on top of the spreadsheet, containing charts and film info cards;
    3. Deliver both the spreadsheet and the web page to a specified location in a team collaboration platform.

    This type of task was chosen because it covers nearly every key stage of the agent office scenario: reading and parsing dynamic web pages, understanding fuzzy conditions, generating structured data, writing front-end code, and permission verification during cross-application delivery. The task has 5 steps—web search → content organization → spreadsheet generation → code writing → automated delivery—placing high demands on both multi-step execution and tool-calling ability.

    To stay closer to real needs, the acceptance criteria added several implicit constraints beyond the original prompt:

    • The number of films must be exactly 10, no more and no less;
    • The spreadsheet must contain core fields such as title, rating, popularity, and release date;
    • The visualization web page must open independently, with charts consistent with the source data;
    • Delivery must occur after the user completes authorization, and a receipt must be reported back after successful delivery.

    Implicit constraints are where problems surface most easily. A model that only “does what it is literally told” often slips on field completeness, data consistency, and authorization confirmation.

    4.2 Execution process and results

    On the browser-driven agent execution platform connected to the model, the task was completed on the first run, delivering the spreadsheet and visualization page at once. A few details in handling the final artifact are worth noting:

    • Multi-sheet organization: The spreadsheet did not mix “the summary for humans to read” with “the data for programs to read” in one sheet; instead, it split them into a cover sheet and a raw data sheet. The cover sheet carries the title, description, and reading guidance, while the raw data sheet keeps a clean two-dimensional structure that is easy to sort, filter, or reprocess later. This distinction reflects an understanding of “who the deliverable is for.”
    • Lightweight visual design: Header coloring and background styling were added to the spreadsheet, improving its professional look without hurting readability.
    • Visualization based on real data: The charts in the web page are not static placeholders but are generated by directly reading the spreadsheet data, ensuring data consistency.

    From the execution trace, the model first visited the target page during retrieval and extracted the film list, then checked each candidate’s rating and release date one by one to filter out the 10 qualifying films; it next generated the spreadsheet according to the predefined fields; then it wrote the page code to render the charts and cards; finally it called the delivery tool to send both files to the specified location. The whole chain showed no obvious invalid calls or repeated back-and-forth.

    It should be noted that the quality ceiling of such tasks is closely tied to how detailed the prompt is. If a color scheme, chart types, card layout, or page count were further specified, the model’s output would be more refined and closer to expectations. But this is exactly the point: agent evaluation does not measure “whether the model’s limits can be reached,” but “whether reasonable, usable results can be produced under common, vague, information-incomplete real instructions.” Real business needs are often exactly like this—no one writes out every detail before starting.

    4.3 Key risks and engineering recommendations

    • Permissions and authorization: Operations involving login, writing, and external delivery should fall under least-privilege management. For actions involving external sending, the user should explicitly confirm before execution, to avoid accidental sends.
    • Credential management: Account credentials should be injected through a unified key management mechanism and must not appear in prompts, logs, or artifacts.
    • Execution sandbox: Browser operations and file writes should run in an isolated environment, with restrictions on accessible domains and directory scope.
    • Artifact traceability: Every delivery should retain version records and operation logs, making it easy to trace “which step went wrong.”

    5. Test case 2: multi-skill collaborative stylized video generation

    5.1 Task design

    The second case examines the model’s ability to execute creative tasks under “skill extension” support. The task required invoking a video generation skill set to produce a stylized short film of about 40 seconds, with the following theme and requirements:

    • Theme: Pixel Wasteland. Live-action footage and low-poly 3D rendering are mixed in real time as the main visual language, with the world switching between “high-definition reality” and “corrupted mosaic” states.
    • Plot: The protagonist walks through a bustling city; everywhere he passes, buildings instantly degrade into rough geometric color blocks and pedestrians turn into static textures; he tries to run to escape the “rendering lag” but finds his own hands beginning to pixelate and dissipate; the final shot pulls back, and the entire city is just an image on an old monitor about to lose power—after a burst of electrical noise the screen goes completely black, reflecting only the viewer’s own face.
    • Style and metaphor: Glitch art is used to create unease and to evoke a digital existentialist crisis; the ending prompts viewers to reflect on their relationship with the digital world.

    This type of task is fundamentally different from technical office tasks: it requires the model to perform multiple translations of “abstract concept → visual language → storyboard script → generation parameters → clip composition.” It must understand the visual meaning of metaphors such as “rendering lag” and “texturing,” and convert them into a sequence of operations the skill set can execute.

    5.2 Execution chain breakdown

    A mature creative agent typically executes along the following chain. This chain is itself the observation framework for evaluation:

    1. Requirement parsing: extract hard constraints from the prompt such as duration, style, plot points, and emotional tone;
    2. Structured storyboarding: break the plot into shots, specifying each shot’s visual content, camera movement, duration, and transitions;
    3. Prompt compilation: translate each shot into a visual description the generation tool can understand, including subject, scene, lighting, camera parameters, and style keywords;
    4. Clip generation: call the video generation skill shot by shot to produce material;
    5. Post-production compositing: complete clip splicing, rhythm adjustment, sound effect and music overlay, and reinforcement of transitions and glitch effects;
    6. Export verification: check total duration, resolution, and encoding format, and spot-check whether the visuals meet expectations.

    Along this chain, the model’s main value lies in “translation” and “orchestration” rather than “drawing the picture itself.” Being able to accurately translate a textual artistic vision into operations the skill set can execute is the core competitiveness of a creative agent.

    5.3 Evaluation observation points

    • Style consistency: whether the visual style is unified across shots and whether the mixed-rendering texture runs through the whole film;
    • Narrative completeness: whether the setup, development, turn, and conclusion are clear, and whether the ending’s “screen reflecting a face” is actually visualized;
    • Rhythm and duration: whether shot changes serve the narrative and whether the total duration matches the requirement;
    • Failure recovery: when some shots are unsatisfactorily generated, whether the model can locate the problem shots, adjust the prompt in a targeted way, and retry, rather than starting over from scratch;
    • Process controllability: whether intermediate products are traceable and whether individual shots can be replaced locally.

    Compared with office tasks, creative tasks are more subjective. It is recommended to add a “human review + structured scale” approach beyond the scorecard, in which multiple reviewers independently score across dimensions such as visuals, narrative, and audio-visual coordination, to reduce bias from individual preferences.

    6. Supplementary recommendations: four more high-frequency scenarios

    Beyond the two task types above, enterprises should also pay attention to the following scenarios during actual model selection:

    1. Software engineering tasks: fixing defects or implementing new features in a given code repository, with the acceptance criterion being whether all unit tests pass;
    2. Long-document processing and research report generation: extracting information from multiple documents of varying formats and producing a summary report with cited sources, with emphasis on factual accuracy and source traceability;
    3. Browser operations and form workflows: completing actions such as querying, filling out forms, and approving in a business system, with emphasis on operational correctness and tolerance for abnormal pages;
    4. Long-cycle tasks and interruption recovery: when task execution spans a long time or requires waiting for external conditions, and the session is interrupted or the context is compressed midway, whether the model can continue from existing progress rather than starting over.

    These four scenarios together point to one conclusion: agent evaluation should not be a one-time “model selection” action but should become a continuous regression testing system. Only by re-running the same task set after a model version update can changes in capability be detected.

    7. Evaluation automation: from manual scoring to regression evaluation

    When the task set grows large, manual item-by-item scoring becomes unsustainable. It is recommended to write acceptance criteria as executable check scripts to achieve “one-click regression.” The example below shows the core structure of the evaluation framework: a task consists of a prompt and a list of check functions; each model version is executed several times on the same task set, and success rate and defect distribution are finally output.

    1. from dataclasses import dataclass, field
    2. from typing import Callable, List
    3. # Record of a single execution
    4. @dataclass
    5. class Trace:
    6. steps: int # number of tool calls / interactions
    7. retries: int # number of retries
    8. artifacts: dict # artifact paths or content
    9. human_interventions: int = 0 # number of manual interventions
    10. error_log: List[str] = field(default_factory=list)
    11. # Evaluation task definition: prompt + auto-verifiable check items
    12. @dataclass
    13. class Case:
    14. case_id: str
    15. prompt: str
    16. checkers: List[Callable[[Trace], bool]]
    17. weight: float = 1.0 # task weight
    18. def evaluate(runner, cases: List[Case], repeat: int = 3) -> dict:
    19. report = {}
    20. for case in cases:
    21. results = [runner.run(case.prompt) for _ in range(repeat)]
    22. pass_cnt = sum(all(c(t) for c in case.checkers) for t in results)
    23. rate = pass_cnt / repeat
    24. avg_steps = sum(t.steps for t in results) / repeat
    25. report[case.case_id] = {
    26. "success_rate": round(rate, 3),
    27. "avg_steps": round(avg_steps, 1),
    28. "failures": [t.error_log for t in results if not all(c(t) for c in case.checkers)]
    29. }
    30. return report

    With check functions, “whether completed” can be turned into a deterministic judgment. For example, for information retrieval and spreadsheet tasks:

    1. def check_artifact_exists(t: Trace) -> bool:
    2. return "report.xlsx" in t.artifacts and "index.html" in t.artifacts
    3. def check_table_schema(t: Trace) -> bool:
    4. # verify fields are complete, row count is 10, and there are no empty values
    5. rows = t.artifacts.get("rows", [])
    6. return len(rows) == 10 and all(set(r) >= {"title", "rating", "popularity", "release_date"} for r in rows)
    7. def check_html_consistency(t: Trace) -> bool:
    8. # verify the chart data in the page matches the spreadsheet data
    9. return t.artifacts.get("chart_matches_table", False)
    10. def check_delivery_receipt(t: Trace) -> bool:
    11. # verify the delivery action is completed and a success receipt is obtained
    12. return t.artifacts.get("delivery_status") == "success"
    13. def check_no_excessive_steps(t: Trace) -> bool:
    14. # process reasonableness: steps and retries within thresholds
    15. return t.steps <= 30 and t.retries <= 2

    In practical deployment, three points deserve attention:

    • Repeated execution: model output is stochastic, so the same task should be executed 3–5 times, with conclusions drawn from the success rate rather than a single run;
    • Task layering: divide tasks into a core set and an extended set. The core set must be run after every model version update, while the extended set is periodically supplemented to uncover long-tail issues;
    • Environment isolation: each case runs in an independent sandbox environment, avoiding the residual state of a previous task affecting the next.

    On platform selection, enterprises can focus on whether multi-model unified access, flexible orchestration of tools and skills, and complete execution logs and monitoring/alerting are supported. For example, agent development platforms, log services, and monitoring/alerting capabilities offered by cloud vendors such as Baidu AI Cloud can serve as infrastructure options for running this kind of evaluation workflow. Whatever toolchain is chosen, the core principles stay the same: reproducible environments, measurable metrics, and traceable results.

    8. How to correctly interpret evaluation results

    The value of evaluation is not to produce a ranking, but to provide a basis for decisions. Drawing on the cases above, four lessons are worth emphasizing:

    • A single leaderboard does not equal the real level: leaderboards cover only limited task types and should be cross-checked with a task set built from the enterprise’s own scenarios.
    • One success does not equal reliable stability: for critical workloads, focus on the success rate and failure modes across multiple runs, not just “the best single run.”
    • Carrier and toolchain significantly affect performance: evaluation conclusions should state the runtime environment and must be re-verified after switching frameworks.
    • Cost and efficiency are also metrics: the inference time, number of tool calls, and manual interventions consumed to complete the same task directly affect the feasibility of large-scale deployment.

    For enterprises, a more pragmatic approach is to split evaluation into two steps: first use public leaderboards for initial screening to narrow the candidate set; then run a final evaluation with a task set close to their own business processes, focusing on end-to-end success rate, per-task cost, and exception recovery ability, ultimately choosing a model and toolchain combination that is “good enough and stable enough.”

    9. Conclusion

    Back to the opening question: what level are these new models actually at? A more accurate answer is that, on multi-step tasks such as web search, information organization, spreadsheet and visualization generation, and cross-application delivery, they already possess the basic ability to “finish the job in one go”; on creative tasks such as stylized video generation, they can translate a textual vision into an executable generation workflow, but the final quality still depends on the maturity of the skill ecosystem and the human oversight involved.

    For developers and enterprises, rather than arguing over which model is “the strongest,” it is better to build their own agent evaluation system as soon as possible: turn high-frequency business processes into verifiable task sets, quantify results with a scorecard, and continuously regress with automated scripts. Only by turning “can it get the job done” into a metric that is measurable, comparable, and continuously improvable can large models truly move from “a surprise in demos” to “reliability in production.”