How to Run a Fair LLM Benchmark for Agentic Applications
Public LLM leaderboards mostly score single-turn Q&A and isolated code snippets, but agents run long, multi-turn workflows with real side effects, so leaderboard rankings rarely predict real performance. This article lays out a reproducible way to compare large models in agent scenarios: locking down every variable except the model, evaluating through five progressively harder task levels, scoring with a mix of automated assertions, blind review, and manual checks, and normalizing cost into per-successful-task terms. It includes a reusable runtime configuration, a scoring rubric, and practical recommendations on tiered model routing, fallback strategies, and enforcing safety at the runtime layer.
Why general leaderboards fail to pick an agent workhorse
Most public evaluations still center on single-turn Q&A, short-context knowledge questions, or isolated code snippets, and they score one thing: whether the answer is correct. Agents run a very different workload—long chains that span multiple turns and produce side effects: read the requirements, retrieve context, write files, run commands, inspect the results, and course-correct. Three properties of this shape cause leaderboard scores to diverge sharply from the real experience.
- Errors compound. A model with 95% per-step accuracy will have an expected completion rate well below 95% over a 10-step chain, because any single wrong step can poison the context that follows.
- Runs are stateful. Each turn’s intermediate artifacts, error messages, and already-executed actions become the next turn’s inputs. Whether the model can remember and correctly interpret its own history often decides the outcome.
- Actions can’t be rolled back. Writing files, sending requests, and mutating data are real side effects. An action that looks correct may also do something it was never authorized to do.
The failure modes that show up again and again in practice cluster around these same points. (The following describes a general class of phenomena to show what an evaluation must cover; it does not point at any specific product.)
- Skill/tool declaration failures: the tool or skill pack is mounted correctly, but the model doesn’t load it, doesn’t call it, or calls it with arguments that don’t match the declaration.
- High-risk actions taken without a second confirmation: delete, overwrite, and outbound-send operations execute without explicit permission.
- Context overhead blow-up: repeated self-checks and re-reads on the same task push token consumption several times above a normal configuration.
- Instruction decay over long context: early constraints (format, naming, no-edit zones) get forgotten after dozens of turns.
- Structured-output drift: JSON or table fields go missing or extra, breaking downstream parsing.
So the goal of the evaluation isn’t to pick the highest-scoring model. It’s to find, under one shared agent runtime, which model can complete tasks stably and predictably.
Experimental design: making “change only the model” real
For a comparison to mean anything, the only variable allowed to change must be the model itself. The way to enforce that is to clone several structurally identical agent instances and then swap in each model under controlled conditions.
Controlled variables
| Item to hold fixed | What it means | Bias introduced if not fixed |
|---|---|---|
| System prompt and constraints | Role, output format, no-edit zones, confirmation policy | Differences can’t be attributed to any single cause |
| Memory store | Same history summary and same retrieval implementation | Context quality differs |
| Skill/tool set | Same skill packs plus the same tool allowlist | Capability boundaries differ |
| Permission policy | Confirmation gates for high-risk actions, writable-path allowlist | Security isn’t comparable |
| Reasoning tier | All candidates that support a higher tier run at the higher tier | Comparing a low tier against a high-tier rival distorts the result |
| Context budget | Same truncation/summarization policy and the same window limit | Long-task performance gets masked by context strategy |
| Runtime environment | Separate session directory, clean virtual workspace snapshot | One candidate’s artifacts contaminate the next |
| Repeat count | At least 3 runs per level, report the median and failure rate | A lucky single pass gets mistaken for capability |
Screening candidates and justified exclusions
Excluding a candidate shouldn’t be arbitrary; it needs a reason that can be re-derived. Common grounds for exclusion include: the cost per successful task is uncontrollable; free or low-tier quotas can’t sustain back-to-back runs; poor long-connection stability keeps interrupting sessions; and structured output can’t reliably meet downstream parsing requirements. The key point: the exclusion reason must go into the report. Otherwise readers can’t tell whether the model “can’t do it” or “the budget couldn’t support it.”
Review approach: automated assertions + blind scoring + manual review
Subjective impressions are the least reliable metric. A three-layer review works best:
- Automated assertions: does it run, does it error, is the JSON parseable, does the output file exist and is it non-empty.
- Blind scoring: with model identifiers removed, two or more reviewers score against the same rubric, and you check their agreement.
- Failure forensics: every failure keeps the full execution log, a diff of key intermediate artifacts, and screenshots.
A reusable runtime configuration looks roughly like this (illustrative):
# bench.yaml — change only the model field; hold everything else fixedagent:system_prompt: ./prompts/agent_system.md # one shared file, incl. confirmation policy and format constraintsmemory: ./memory/shared_store # one shared memory and retrieval implementationskills_dir: ./skills # one shared set of skill packstools_allowlist:- file_read- file_write # constrained by write_allowlist- shell_exec # guarded by high-risk-command confirmation- web_fetch- browser_opencontext_budget: 128kcompaction: summarize # same truncation/summarization policy for all candidatespolicy:require_confirmation_for: ["delete", "overwrite", "network_write", "prod_db"]write_allowlist: ["./workspace"]suites: [skill_pack, web_ui, doc_gen, kb_rag, browser_flow]repeats: 3candidates:- name: candidate_amodel: <model-id>effort: high # unified high reasoning tier- name: candidate_bmodel: <model-id>effort: high
Five-level task suite: from mechanical cases to progressive capability checks
Don’t use fragmented cases named “case1 / case2 / case3”—they have no explanatory power and won’t surface real defects. A better design is a set of progressive levels, each verifying one irreplaceable capability, where failing an earlier level makes the later ones meaningless.
Level 1: Capability encapsulation (packaging a skill pack)
Task: given a long prompt or spec document, turn it into a distributable capability pack that “works the moment someone installs it.”
Expected artifacts: a clear directory layout, usage instructions, trigger conditions, parameter definitions, exception handling, plus one self-check and a risk note.
capability-pack/├── manifest.yaml # name, version, trigger conditions, input params, permission declarations├── README.md # install and usage examples├── references/ # reference docs (abstracted, not copied verbatim)└── checks/ # self-checks and expected outputs
Scoring points: is the directory complete and portable; can someone reproduce it from the docs; does it cover edge cases such as missing params, nonexistent files, and insufficient permissions; and—critically—did it merely restate the source, or did it abstract it into an interface. This level is the most effective filter: a model that can’t even package existing experience reliably isn’t worth testing on the later levels.
Level 2: Page and frontend engineering
Task: generate a runnable page or small app from a requirements description.
Scoring points: does it run in one shot; any console errors; responsive behavior; basic accessibility (image alt text, form labels); whether it pulls in external dependencies that must load online and therefore break offline; and whether the artifact can be hosted directly (dropped into object storage as static assets). The focus here is engineering discipline, not visual wow-factor.
Level 3: Structured content generation (slide decks and copy)
Task: from given materials and data, produce a presentation outline, per-slide talking points, and accompanying copy, and save it to files.
Scoring points: does the narrative structure hold; is the information density per slide appropriate; do the numbers in the copy match the source material (this is where hallucination shows up most easily); do the exported files open correctly; and does it strictly honor the “don’t fabricate data” constraint.
Level 4: Knowledge base governance and retrieval augmentation
Task: dedupe, chunk, and tag documents from mixed sources, build a searchable index, and answer questions with source citations; then run an incremental update.
Scoring points: chunking and metadata quality; whether citations are verifiable (you can locate the original snippet); whether it gives a rationale when facing conflicting information; whether it explicitly refuses when it doesn’t know rather than fabricating; and whether the old index is correctly invalidated after the incremental update. In practice, indexes usually sit on object storage alongside vector/keyword retrieval services, and async writes go through a message queue.
Level 5: Long-chain browser automation
This is the most resource-hungry level and the one that separates models most sharply. The task has to be long enough: log in, page through, filter, bulk-scrape, and aggregate to disk.
Scoring points:
- Selector robustness: does it degrade gracefully when page structure changes (prefer semantic locators over absolute paths);
- Failure recovery: on timeouts, missing elements, or network errors, can it retry and change strategy rather than looping forever;
- Forensics: do key steps leave screenshots and logs for auditing;
- Compliance: rate limiting, and how it handles captchas / step-up login verification—it must not bypass identity verification;
- Permission boundaries: does it only touch authorized paths and domains, and does it avoid taking unrequested high-risk actions.
One point deserves emphasis: security gates belong to the policy layer, not to the model’s own discretion. High-risk-action confirmation, writable-path allowlists, and command blacklists should be enforced by the runtime; the model only decides within the authorized boundary. Betting safety on “the model won’t do anything bad” is the most common wrong assumption in evaluation.
Turn subjective impressions into a re-derivable scorecard
| Dimension | Measurement | Type |
|---|---|---|
| Task completion rate | Levels passing all assertions / total levels | Automated |
| First-pass rate | Tasks succeeding without a retry / total tasks | Automated |
| Average interaction turns | Median turns to reach success | Automated |
| Tool-call correctness | Calls with fully compliant arguments / total calls | Automated |
| Recovery success rate | Self-recovered after fault injection / injection count | Automated |
| Manual intervention count | Times a reviewer had to step in and fix | Manual |
| Total consumption | Token count, wall-clock time, converted cost | Automated |
| Cost per success | Total spend / successful tasks (incl. failed-retry consumption) | Computed |
Weights can be tuned to need; an illustrative set is correctness 40%, stability 25%, cost and efficiency 20%, engineering fit 15%. What comes out isn’t just a ranking—it’s the shape of each radar chart. A model that scores low overall but is extremely strong on one specific level might be exactly the right answer for that one part of the pipeline.
Unify the cost basis: subscription quotas aren’t the same as API billing
Billing models vary widely across products, so putting raw price numbers side by side is meaningless. Normalize everything into cost per successful task:
unit_cost = (input tokens × input price + output tokens × output price + duration × time price) / successful tasks# consumption from failures and retries must go into the numerator, or you'll systematically undercount a weak model's true cost
Factors to fold into the analysis:
- Reasoning-tier cost amplification: turning on a high reasoning tier sharply increases output-side consumption, and the same model’s cost-effectiveness can invert between tiers.
- Linear context amplification: long-chain tasks resend history every turn, so input tokens grow roughly linearly; weigh the payoff of prompt caching and summary compression.
- Subscription-quota burn rate: for subscription plans, watch “how long the quota lasts”—heavy continuous use can eat a meaningful share of the monthly quota within one or two tasks, so don’t conclude a plan is cheap just because its unit price looks low.
- Hidden costs: manual rework, failed retries, migration effort, and patch code written to work around defects should all be converted in.
The engineering recommendation is to do unified accounting on the model-service side: put artifacts in object storage, decouple async tasks through a message queue, and make the call chain observable with a logging service and monitoring/alerting, so token consumption, latency, and failure rate become metrics you can slice by candidate model.
Conclusions and practical recommendations
- There’s no single best model—only tiered configurations. Group candidates into a drafting tier (cost-sensitive, high volume), a production tier (correctness and stability first), and a batch tier (throughput first), and route by task type instead of using one model everywhere.
- Main model + fallback model + degradation strategy. When the main model keeps failing key assertions or its structured output fails to parse, switch to the fallback automatically and resume from a checkpoint of the completed steps.
- Maintain the evaluation set as an asset. The five levels plus the assertion scripts are themselves one of a team’s most valuable assets. Models iterate quickly, so the eval set must grow and be re-run alongside them—otherwise your conclusions have a shelf life you aren’t tracking.
- Enforce safety boundaries at the runtime. High-risk-action confirmation, writable-path allowlists, sensitive-data masking, and full operation auditing should all be platform-layer capabilities, not a reliance on the model’s judgment.
- Make selection conclusions reproducible. Publishing your controlled-variable list, scorecard, repeat count, and cost basis matters far more than publishing the ranking.
For most teams, a sensible path is: first build a controlled evaluation rig where only the model field changes, get the five levels running and turn them into a durable scorecard, and then connect the evaluation to production canary releases using general capabilities from cloud vendors such as Baidu AI Cloud (model services, object storage, message queues, logging services, and monitoring/alerting). That way every model swap is backed by data rather than decided on a hunch.
