How to Choose an Agent Foundation Model: A Reproducible Head-to-Head Benchmarking Methodology
Choosing a foundation model is one of the first decisions that determines whether an agent succeeds in production. This article presents a reproducible way to compare candidate models instead of relying on public leaderboards or per-token pricing. It explains how to hold every variable constant except the model adapter, how to score behavior through five graded task stages, and how to account for real cost using task-level token consumption. It closes with engineering practices for putting the benchmark to work: a pluggable model adapter, tiered routing across task types, safety gates for high-risk actions, and turning the evaluation into a regression suite. Readers will get a method they can rerun whenever models change.
1. Model Selection Is the First Hurdle in Agent Deployment
An agent differs from an ordinary conversational application in that it is a “stateful executor.” The system prompt defines its role and constraints, memory carries context across turns, skills package domain expertise into reusable workflows, and the tool layer lets it actually touch the file system, the browser, and business APIs. These components form a closed loop, and the underlying large language model is the “engine” that drives the loop. Swap the engine and the performance of the entire drivetrain shifts with it: with the same prompt and the same tools, a different model can turn a job that “ran through on the first try” into one that “keeps spinning in circles.”
When developers pick a model, they often fall into one of two extremes. The first is judging only by public leaderboard rankings, but leaderboards mostly measure standardized problem types, while the real difficulty in agent scenarios is not drifting during long-horizon tasks, keeping tool-call formatting stable, and self-repairing after errors. The second is looking only at the per-token price, ignoring “the total consumption required to complete a task”: a model that is cheap per token but prone to detours can, because of repeated retries and repeated file reads, end up with a higher real bill than a pricier model — and burn the developer’s waiting time along the way.
So the conclusion up front: choosing a model for an agent is not about picking the “strongest” one, but the one that “best matches the current framework constraints and task distribution.” For that sentence to become an executable decision, you first have to control the variables.
2. Controlling Variables: The Prerequisite for Credible Results
My approach this time: I copied a complete Agent prototype — one that had already been running for a few months — into four identical copies. The memory snapshots were the same, the skill library was the same, the system prompts and constraints were identical word for word, the callable tools and permission allowlists were the same, and the context-compression strategy and timeout/retry parameters were the same. The only variable was the lowest-level model adapter layer. All four candidate models were run with the high-reasoning tier enabled when needed, since agent tasks mostly require multi-step planning rather than single-turn Q&A.
The key to this design is “identical word for word” rather than “roughly the same.” Agent frameworks are extremely sensitive to subtle differences in prompts; the presence or absence of a single line like “you must self-check after finishing” can change whether the model proactively performs verification.
| Fixed item | Variable allowed? |
|---|---|
| System prompts, role constraints, output-format requirements | No |
| Memory snapshot and retrieval strategy | No |
| Skill package contents and directory structure | No |
| Tool allowlist, permission levels, parameter validation rules | No |
| Reasoning tier, context-compression threshold, timeout and retry | No |
| Underlying model and its adapter layer | Yes (the only variable) |
Two easily overlooked disciplines also matter:
- A single run is not a result. Repeat each task at least 3 times and take the median and success rate; the randomness of model output is enough to completely reverse a single run’s outcome.
- Fix the test environment. Lock dependency versions, fix the network egress, and turn demo pages into offline fixed samples, so you don’t mistake network jitter for a model-capability problem.
3. Five-Stage Task Design: Turning “Feels Good” into Something You Can Score
I designed five stages rather than case1/case2/case3 because the order of the stages carries information itself: the earlier a stage is, the closer it is to a “passing line,” and if a model can’t get past it, there’s no point testing further.
Stage 1: Turning experience into an asset. The model is asked to turn a fairly long prompt document into a skill package that can be published online and used by anyone who installs it, rather than just copying the original text. After producing the output, it must self-check once and point out where this skill might go wrong.
This stage tests four things at once: long-instruction following, file read/write and directory conventions, structured metadata generation, and the authenticity of self-review. The pass criteria include whether the directory and metadata are compliant, whether it fabricated non-existent fields or interfaces, whether it can be installed in one step, and whether the self-review points to real risks (rather than empty phrases like “more testing may be needed”). If the model can’t even package experience stably, there’s no need to run the later stages.
Stage 2: Page generation. Given a short requirement, the model must produce a page that opens and runs directly. The evaluation looks at three things: whether it runs without errors, whether the layout is usable on narrow screens, and whether interactive elements have the necessary accessibility labels. This stage focuses on whether the model hands in code that “looks right” — many failures are not logic errors but missing references or wrong paths.
Stage 3: Content production. The model is asked to produce a presentation outline and accompanying copy from a structured piece of material. The focus is on structure rather than flair: whether the hierarchy is clear, whether information is missing, and whether facts have been “filled in” with content that isn’t in the material. This stage is especially sensitive to fabrication, because made-up content is hardest to spot immediately in text tasks.
Stage 4: Knowledge base governance. Given a batch of mixed documents, the model must deduplicate, classify, tag, summarize, and build a searchable structure. This tests information fidelity and consistency when handling long documents, especially details that need global consistency such as metric definitions and naming conventions. The evaluation looks at the number of duplicate entries, tag consistency, and retrieval hit rate.
Stage 5: Browser automation. This is the most token-hungry stage and the one that best differentiates models. It requires visual input plus multi-turn tool loops, where each round sends the page structure or a screenshot back into the context, and consumption grows roughly with the square of the number of steps.
This stage looks at three things:
- Whether it spins in place. Ineffective loops are the most typical failure mode — clicking the same element over and over, refreshing the same page repeatedly, exhausting the step limit while the task never advances.
- Whether it has a termination judgment. Being able to state clearly that “the current page doesn’t meet the preconditions, so the task should stop and be reported” is far more valuable than forcing through until the budget runs out.
- Whether it does a second confirmation. When facing irreversible actions like submitting an order, deleting data, or sending a message externally, does it stop and request human confirmation? This is a safety baseline, not a bonus capability.
4. Cost Accounting: Don’t Make Decisions from a Price Table
The real cost of a single task can be roughly written as:
real cost = Σ(prompt tokens + output tokens per step) × retry multiplier+ extra calls from context compression
From this, you can define a more explanatory metric — task-level token consumption: the median of the total tokens consumed to complete the same task. It naturally folds in the cost of “detours.” If a model’s reasoning-chain output has no cap, output tokens balloon significantly; but if it gets things right the first time, the retry multiplier is 1, and the total budget is actually lower.
There’s another layer: billing model. Pay-as-you-go suits the experimental phase with high variability; quota-based plans burn through faster under long sessions and high-reasoning tiers, where a medium-complexity coding task can eat up a substantial portion of the plan’s quota. When evaluating, you should convert to your own task frequency rather than looking only at the sticker price.
A practical measurement hook can be written like this:
class Usage:def __init__(self):self.prompt_tokens = 0self.output_tokens = 0self.calls = 0def add(self, resp):u = resp.usageself.prompt_tokens += u.prompt_tokensself.output_tokens += u.completion_tokens + getattr(u, "reasoning_tokens", 0)self.calls += 1@propertydef task_tokens(self):return self.prompt_tokens + self.output_tokens
5. Engineering Recommendations: Pluggable, Tiered Routing, Safety Gates
The output of a head-to-head benchmark shouldn’t be just a report; it should be distilled into architectural constraints.
First, abstract the model into an adapter. As long as the interface is unified, switching models is a configuration change, and the benchmark can keep running over time:
from typing import Protocolclass ModelAdapter(Protocol):name: strsupports_vision: boolsupports_high_reasoning: booldef chat(self, messages: list[dict],tools: list[dict] | None = None,*,effort: str = "medium",max_tokens: int = 8192) -> "LLMResponse": ...
Second, use tiered routing rather than binding one model globally. Different stages have completely different cost-benefit structures: lightweight summarization tasks use the fastest tier, code generation turns on high reasoning and loosens the output limit, and browser automation caps the step count and prioritizes stability:
ROUTING = {"summarize": {"effort": "low", "max_tokens": 2048, "max_steps": 1},"codegen": {"effort": "high", "max_tokens": 16384, "max_steps": 1},"browser": {"effort": "medium", "max_tokens": 4096, "max_steps": 30},}
Third, give each task type a structured scoring rubric so conclusions can be reviewed rather than based on impressions:
{"rubric": {"skill_packaging": {"weight": 0.25, "checks": ["manifest_valid", "no_hallucinated_field", "self_review_present"]},"frontend": {"weight": 0.20, "checks": ["runs_clean", "responsive", "a11y_labels"]},"content": {"weight": 0.15, "checks": ["structure_ok", "no_fabrication"]},"knowledge_base": {"weight": 0.20, "checks": ["dedup", "tag_consistency", "recall"]},"browser_automation":{"weight": 0.20, "checks": ["no_idle_loop", "terminate_on_precondition_fail", "confirm_on_irreversible"]}}}
Fourth, add safety gates. Regardless of which model is used, high-risk tools must go through human confirmation; tool parameters must pass allowlist validation before execution; retries are capped when structured-output parsing fails; and loop detection is applied to repeated calls of the same tool. These are the framework’s responsibility; you can’t count on the model to self-regulate.
Fifth, turn evaluation into a regression. Models iterate fast, and today’s conclusions may be invalid a few months later. Lock down the task set, the fixed initial states, and the scorer, wire them into continuous integration, and rerun the whole suite automatically with a saved report every time a model is swapped or upgraded. Multi-model integration can be handled uniformly through a cloud vendor’s model service — for example, Baidu AI Cloud’s model service layer supports accessing multiple models in a unified way, which makes switching and canary rollouts easier.
6. Conclusion
“The strongest model” is a false premise. The real answer is “the model that best matches the current framework,” and that answer needs to be measured with the controlled-variable method, costed with task-level token consumption, and put into practice with tiered routing. What’s most worth keeping from this benchmark isn’t the ranking but the method itself: five stages solidified into a regression set, models abstracted into pluggable adapters, cost and success rate considered together, and high-risk actions always backed by human confirmation. Once the foundation model becomes a configuration item you can swap at any time, you no longer need to keep worrying about whether “the next model will be stronger,” because you can switch whenever you want.
