Why Top Models Score Only Around 50% in Agent Evaluations—and How to Build Reliable Agent Test Systems
Even top-ranked models often score only slightly above 50% on realistic agent evaluation tasks. This article explains why such results are not simply a sign of weak model capability, but a signal of insufficient reliability across the entire agent execution chain. It covers why small task sets still have diagnostic value, the four common root causes of agent failure, how to design a trustworthy evaluation system, a lightweight evaluation framework, and engineering practices for turning a 50% score into usable production capability.
1. What a Score Around 50% Really Means
In recent public agent capability evaluations, a widely discussed phenomenon is that even the highest-ranked models achieve only slightly more than a 50% overall score on dozens of tasks designed to resemble real work. Such results are easy to interpret as “the model is not capable enough,” but for practitioners who have delivered agents in real environments, the result looks more like a system-level health check. The conclusion usually points elsewhere.
To understand this, first consider the fundamental difference between agents and single-turn conversational models. Failure in a single-turn Q&A setting is local: if the answer is wrong, the user can ask again. An agent, however, operates in a multi-step, stateful execution process that changes the external world. It must understand the goal, break the goal into steps, choose the right tools, construct valid parameters, read execution results, adjust its strategy based on feedback, and finally leave the environment in a correct, verifiable state. Every link in this chain has its own probability of failure, and failures propagate downstream.
A simplified model makes this clear. Suppose the single-step accuracy is 95%. For a task that requires 20 steps, the end-to-end success rate is approximately 0.95^20, or about 36%. In other words, in long-horizon tasks, small single-step errors are amplified by the number of steps into an overall unusable result. This explains a common confusion: why does a model that looks “very smart” in conversation still fail at the last step in a real workflow? A score around 50% is therefore not the upper limit of model capability. It is a signal of insufficient reliability budget along the execution chain.
2. Why Small Task Sets Still Have Diagnostic Value
Some may question whether a task set with only dozens of cases has enough statistical power to support conclusions. This criticism is valid when the goal is ranking, but not when the goal is diagnosis. These are two different use cases.
When used for ranking, sample size does matter. Confidence intervals are affected by sample size, task difficulty distribution, prompt wording, and the strictness or leniency of the evaluator. In that context, conclusions can be fragile.
When used for diagnosis, value comes from information density rather than statistical significance. A good task set deliberately includes multi-step dependencies, environmental side effects, ambiguous instructions, distractors, and mid-task states that require correction. Each failure trajectory then carries substantial structural information: Did the agent plan incorrectly? Did it choose the wrong tool? Did it construct bad parameters? Did it fail to recover after making a mistake? Dozens of tasks are often enough to reveal high-frequency failure patterns. These patterns are usually systemic rather than random.
A more practical positioning is therefore to treat such evaluations as a regression test suite, not a leaderboard. It should be run after every change to prompts, tool descriptions, or retry policies, in order to answer a specific question: Which tasks became better after this change, and which became worse?
3. Four Root Causes of Agent Failure
Without tying the discussion to any specific product, failure trajectories from real tasks generally fall into four categories.
1. Planning and task decomposition failures.
The model jumps directly to execution without confirming preconditions. Or it breaks a large goal into disconnected fragments without maintaining an overall route. A typical symptom is discovering halfway through the task that a required ID or prerequisite was never obtained.
2. Tool-calling and parameter-construction errors.
These include choosing the wrong tool, missing required parameters, type or format mismatches, and treating natural-language fields from context as enum values. A more subtle failure is this: the tool returns an error message, but the model either ignores it or misreads it, then continues retrying with the same parameters or in the wrong direction.
3. Loss of control over state and context.
In long tasks, history accumulates continuously. Early critical information may be diluted or truncated, leading to drift where the agent “misremembers” what it has already done. The opposite problem is also common: excessive context growth causes costs to spike and responses to slow down. When the model compresses context into summaries, it may lose details that are needed later.
4. Lack of self-verification and rollback.
The agent treats “the tool call succeeded” as equivalent to “the goal was achieved,” without confirming the result. After a partial write, there is no compensating action, leaving dirty data. Many tasks also have vague acceptance criteria, making failure difficult to detect and problems even harder to expose.
4. Designing an Evaluation System That Does Not Fool Itself
The credibility of the evaluation system matters more than the score itself. An evaluator with false positives continuously provides a false sense of safety. The following steps are recommended in order.
Build the task set in layers.
Do not start with long-horizon tasks only. Separate tasks into single-step tool calls, two- or three-step combined tasks, long-horizon multi-turn tasks, and tasks with ambiguous conditions that require clarification. Measure success rates by layer. The value of this approach is that when the overall score drops, you can immediately identify which layer has degraded.
Ensure determinism in judging results.
Each task must have a deterministic verifier. The verifier should assert the final state of the environment, not the model’s natural-language claim that it succeeded. Common forms include database record field matching, file content and directory structure comparison, external system request log verification, and snapshot diffs.
Enforce isolation and idempotence.
Each task should run in a disposable sandbox. The initial state should be rebuilt by a script, and network access and credentials should be granted with least privilege. This makes tasks parallelizable, repeatable, and replayable, and prevents one execution from contaminating the next.
Separate outcome metrics from trajectory metrics.
Looking only at success rate can hide “lucky but wrong” paths. At minimum, record task success rate, first-pass success rate without retries, average number of steps, token consumption, dangerous or unauthorized calls, and the ratio of no-progress steps. A no-progress step is defined as repeated calls to the same tool with unchanged parameters over multiple steps.
Run perturbation tests before discussing overfitting.
Take the same task and rephrase it, add or remove an irrelevant piece of context, or move a critical field into earlier history. Then observe the variance in success rate. High variance means the solution depends on the literal wording of the prompt and is not robust.
Validate the evaluator first.
Prepare a set of known-correct and known-incorrect trajectories and feed them to the verifier. Confirm that it neither passes incorrect trajectories nor misjudges correct ones. This step is often skipped, yet it is the foundation of evaluation credibility.
5. A Lightweight Evaluation Framework
Below is a minimal orchestration skeleton. The focus is on three things: budget control, trajectory recording, and state-based verification.
# Task definition: prompt + initial environment + deterministic checker@dataclassclass Task:id: strprompt: strsetup: dict # Declarative description of the initial environment statecheck: Callable # (final_state) -> (passed: bool, reason: str)def check_order_paid(state):row = state.db.query_one("select status, amount from orders where id = 1024")if row is None:return False, "order missing"if row["status"] != "paid":return False, f"unexpected status: {row['status']}"if abs(row["amount"] - 299.00) > 1e-6:return False, "amount mismatch"return True, "ok"
# Execution loop: budget constraints + replayable environment side effectsdef run_task(task, agent, max_steps=25, max_tokens=200_000):env = Sandbox.from_snapshot(task.setup) # Create a new environment each time to ensure isolation and idempotencebudget = Budget(max_steps=max_steps, tokens=max_tokens)for _ in range(max_steps):action = agent.next_action(env.observation())result = env.execute(action) # Log tool calls so the trajectory can be replayedbudget.record(action, result)if result.is_high_risk and not result.approved: # Dangerous actions require human confirmationreturn verdict("needs_human_review", budget)if budget.exhausted():return verdict("budget_exceeded", budget) # Explicitly distinguish failure reasonsif env.finished():breakpassed, reason = task.check(env.final_state()) # Check the final statereturn verdict("passed" if passed else "failed", budget, reason)
# Aggregate outcome metrics and trajectory metrics side by sidedef summarize(results):total = len(results)return {"success_rate": sum(r.status == "passed" for r in results) / total,"first_pass_rate": sum(r.retries == 0 and r.status == "passed"for r in results) / total,"avg_steps": mean(r.steps for r in results),"avg_tokens": mean(r.tokens for r in results),"no_progress_ratio": sum(r.repeat_calls for r in results)/ max(sum(r.steps for r in results), 1),"unsafe_calls": sum(r.unsafe_calls for r in results),}
Tracking failure reasons separately is critical. budget_exceeded, tool_error, and verify_failed point to completely different fixes. The first requires changes to task decomposition and termination conditions. The second requires changes to tool descriptions and error feedback. The third requires checking whether execution failed or whether the verifier itself was written incorrectly.
6. Engineering Checklist: Moving from 50% to Usable
Evaluation tells you the temperature. Engineering is how you treat the cause. The following items usually produce the most direct gains.
Use structured constraints to narrow the output space.
Have the model produce tool-call intents that conform to a schema, rather than free-form text that must be parsed afterward. Enum values should be validated against a whitelist. When validation fails, feed back a repairable error message instead of terminating immediately.
Treat tool error messages as product design.
Error responses should explain why the call failed, which field was invalid, and what format was expected. For the same tool, better error messages can make a noticeable difference in retry success rate.
Explicitly manage retry budgets and termination conditions.
Unbounded self-correction loops burn cost and fail to converge. Set a maximum number of steps, a maximum number of repeated calls, and a stop condition for continuous no-progress behavior. Record the termination reason as a metric.
Add checkpoints for critical state transitions.
For irreversible operations such as writes, deletions, payments, and outbound sends, assert preconditions before execution. If necessary, require a second human confirmation. This is not only a reliability mechanism but also a security baseline: grant credentials with least privilege, write all external side effects to an audit log, and do not rely on the model to “behave well.”
Close the observability loop.
For every execution, retain the full trajectory, including inputs, tool calls, parameters, returns, latency, and consumption. Trends in success rate and cost across versions should be visible on dashboards. On cloud platforms, evaluation pipelines can typically be deployed on container platforms, with traces and state snapshots archived in object storage, and metrics continuously tracked through logging, monitoring, and alerting. This helps avoid a common failure mode: offline evaluation passes, but online performance silently degrades.
Use production traffic to improve the evaluation set.
Real failure cases are the best source of new test tasks. Cluster them by failure mode, anonymize them, add verifiers, and turn them into regression cases. This allows the evaluation set to evolve with the business rather than remain a static leaderboard.
7. Conclusion
A score around 50% is not a final verdict. It is a clear to-do list. It reminds us that the competitiveness of an agent has shifted from “can it answer a single question correctly?” to “can the entire execution chain be verified?” Whoever does the following well can turn demo effects into sustainable online capability: layered tasks, deterministic verification, state-based checking, budget constraints, and observability. Conversely, if teams only repeatedly tune prompts without closing the evaluation loop, scores will drift randomly between versions, and you will never know whether a change is progress or regression. Build a trustworthy evaluation first, then optimize. That is the path least likely to lead you astray.
