Long-Horizon AI Agent Evaluation: Capability Layering and Budget Governance

    A model that looks concise and reliable in short conversations can loop indefinitely in long-horizon tasks that run for tens of minutes or hours. This article explains why conversational quality and task-completion quality must be evaluated separately, catalogs three typical failure modes of long-running agents — over-verification, style drift, and review-perspective collapse — and proposes a reusable evaluation framework built on cognitive task layering (L0–L3) and metrics that pair quality with cost. It also covers quota governance through observability, checkpointing, tiered model routing, and interruptible design, plus a stage-and-budget prompt contract for review tasks. Readers will find practical guidance for turning subjective impressions of agent performance into a measurable, reproducible, budget-aware engineering practice.

    1. The Evaluation Mismatch: From “Conversation Quality” to “Task Completion Quality”

    Most model evaluations come from short exchanges: ask a question, get an answer, reach a clear conclusion in seconds — which naturally feels good. But once a model is dropped into a long-horizon task that runs for tens of minutes to hours, the evaluation dimensions shift entirely. Users no longer care whether a single-turn reply is polished; they care whether the task can converge, whether cost stays under control, whether the output can be reviewed by a human, and whether failures are recoverable.

    Mixing these two sets of standards is the most common source of misjudgment. The same model may look concise and dependable in short conversations yet circle back repeatedly in long tasks. Conversely, a model that prefers to lay out evidence and keeps a flat, even tone across long conversations is often better suited to review and proof-style work. Evaluation must therefore declare the scenario first, and only then discuss which model is better.

    2. Three Typical Failure Modes in Long-Horizon Tasks

    2.1 Over-Verification: Doing Checks Nobody Asked For

    The symptom is that the model expands its own scope of retrieval and tool calls, burning a large share of the budget on details only weakly related to the conclusion. This produces three direct costs: fast quota depletion, unpredictable task duration, and the truly critical conclusion getting buried in a lengthy chain of evidence.

    The cause is not mysterious. Training objectives often reward behavior that “looks diligent,” while the reasoning process has no explicit stopping condition. On top of that, the model cannot accurately estimate the marginal value of “checking one more time,” so it simply keeps going out of behavioral inertia.

    2.2 Style Drift: Concise in Short Chats, Unfocused in Long Ones

    In long contexts, the attention weight on early constraints gets diluted. Output tends to pile up metaphors, coin new terms, and under-front-load the conclusion. The opposite extreme is over-elaborating evidence and dodging clear judgments, leaving the reader to summarize the conclusion on their own.

    This is not a matter of “style preference” but of usability. If the conclusion cannot be located at a glance, the output of a long task requires extra manual re-organization, which silently inflates the real cost.

    2.3 Review-Perspective Collapse: Stuck on Detail Verification, Unable to Give an Overall Verdict

    This is the most expensive failure in long-horizon tasks. When you hand paper review or project-credibility review to a model, it may spend most of its time checking citations one by one to see whether they resolve and whether every number is exact — yet never deliver an overall judgment such as “does the core argument of this work hold, are the contribution boundaries clear, and what blocking problems exist.”

    The result: a question that could have been answered in a few minutes runs for tens of minutes and returns nothing more than an errata table of details.

    3. Building a Reusable Evaluation Framework

    3.1 Task Layering

    Do not judge a model with a single average score. First, stratify by cognitive span:

    • L0 — Single-point fact extraction: has a definite answer, used to verify basic correctness;
    • L1 — Local reasoning: multi-step derivation, automatically verifiable;
    • L2 — Long-chain proof and overall review: requires structural judgment, so it can only be human-reviewed or semi-automatically scored;
    • L3 — Open-ended problem advancement: no known answer, so only process quality can be assessed.

    Conclusions from different layers cannot be extrapolated to one another. High accuracy at L0 does not prove usability at L2, and vice versa.

    3.2 Metric Dimensions

    Dimension Meaning Collection Method
    Conclusion accuracy Whether the final answer is correct Automatic verification or human review
    First-pass rate Share of tasks needing no human rework Marked during human review
    Evidence verifiability Whether citations and facts pass spot checks Sampled verification
    Unit cost Tokens and tool calls consumed per completed task Instrumentation metrics
    Convergence time Time elapsed before a usable conclusion is reached Log timestamps
    Truncation rate Share of runs cut off by budget exhaustion Determined at runtime
    Style usability Whether the conclusion can be located at a glance Human grading scale

    The metrics must cover both quality and cost at the same time. Looking at quality alone systematically overestimates the real value of expensive models.

    3.3 Evaluation Scaffold Skeleton

    1. # Simplified skeleton for layered evaluation + multi-dimensional metric collection
    2. def run_eval(client, case, budget):
    3. trace, usage = [], {"in_tokens": 0, "out_tokens": 0,
    4. "tool_calls": 0, "wall_clock": 0.0}
    5. while not case.done(trace):
    6. if usage["out_tokens"] > budget["tokens"] or usage["wall_clock"] > budget["minutes"]:
    7. trace.append({"event": "budget_exhausted", "usage": usage})
    8. return {"case": case.id, "level": case.level,
    9. "status": "truncated", "trace": trace}
    10. msg = client.step(case.state(trace), trace)
    11. usage["in_tokens"] += msg.prompt_tokens
    12. usage["out_tokens"] += msg.completion_tokens
    13. usage["tool_calls"] += len(msg.tool_calls)
    14. usage["wall_clock"] += msg.elapsed
    15. trace.append(msg)
    16. return {"case": case.id, "level": case.level,
    17. "status": "done", "trace": trace, "usage": usage}

    The key point: the budget is a first-class citizen of evaluation. Only when “truncated” is recorded as a legitimate outcome can you see how a model actually performs under a limited quota.

    4. Quota Governance: Orchestrating the Budget as a Finite Resource

    What hurts most in long-horizon tasks is often not model capability but quota structure. A common setup stacks instantaneous rate limits, hourly rolling windows, and weekly plus monthly total quotas. The catch is that these limits do not simply multiply: a plan where “the weekly quota is roughly three hourly windows” offers a real usable duration far shorter than users intuitively expect, and unattended overnight tasks hit the wall again and again.

    There are four classes of engineering countermeasures:

    1. Quota observability: expose remaining quota and window-reset times as metrics, wire them into monitoring and alerts, and run a pre-flight estimate of “how long this will take and whether the quota is enough” before launching a task.
    2. Checkpoints and idempotency: persist intermediate state at every step so a restart resumes from the checkpoint rather than beginning again.
    3. Tiered model routing: assign retrieval, formatting, and initial screening to low-cost models, and invoke high-capability models only at the decisive steps — this usually cuts total cost significantly.
    4. Interruptible design: split a long task into mutually independent subtasks with concatenable results, and use a message queue for peak shaving and retries.
    1. class QuotaAwareRunner:
    2. def __init__(self, limiter):
    3. self.limiter = limiter # wraps instantaneous rate / rolling window / periodic total quota
    4. def run(self, task):
    5. state = task.load_checkpoint() # idempotent recovery
    6. while not task.finished(state):
    7. cost = task.estimated_cost(state) # estimate this step's consumption
    8. if not self.limiter.acquire(cost):
    9. task.save_checkpoint(state) # persist first, then wait
    10. self.limiter.wait_until_available()
    11. continue
    12. state = task.step(state)
    13. task.save_checkpoint(state)
    14. return task.result(state)

    The value of this pattern is that exhausting the quota no longer means the task failed — it simply means “continue a bit later.”

    5. A Prompt Contract for Review-Type Skills

    Against the perspective collapse in 2.3, the most effective fix is not switching models but using a hard contract to constrain the serial stages and budget of the review process:

    1. Role: paper/project credibility reviewer
    2. Stage constraints (strictly serial, no skipping levels):
    3. S1 Structural: goals, contribution claims, boundaries vs. prior work
    4. S2 Argumentative: dependencies between propositions, circular reasoning, implicit assumptions
    5. S3 Evidentiary: only after S1/S2 show no blocking issues, spot-check no more than N key citations
    6. Stopping condition: terminate immediately if S1 or S2 has a blocking issue; do not enter S3
    7. Budget statement: at most K retrievals this round; on timeout, submit the current conclusion and mark unfinished items
    8. Output contract: conclusion first; {blocking issues, non-blocking issues, verdict}

    Pair this with two output-style constraints — the conclusion must be visible above the fold, each paragraph stays within a fixed sentence count, and uncertain points are explicitly flagged — and you can ease both over-verification and style drift at once.

    6. A Realistic Positioning for Hard Reasoning Tasks

    For open math problems and similar work, capability is extremely unevenly distributed: only a few models can genuinely advance the problem, while the rest are better suited to supporting roles such as literature review, conjecture organization, proof-draft writing, counterexample search, and formal expression. A sound human–machine collaboration chain looks like this: the human proposes a conjecture → the model organizes relevant background and drafts a proof sketch → formal proof assistants and symbolic computation tools perform mechanical verification → the human reviews the key steps. Putting an auxiliary model in the decisive position is a common root cause of runaway long-task costs.

    7. Implementation Checklist

    • Declare the scenario and layer first, then evaluate which model is better; cross-layer extrapolation is forbidden.
    • Keep the evaluation set fixed, use blind review with two-person cross-checking, and make metrics cover both quality and cost.
    • Write stage constraints, stopping conditions, budget caps, and output contracts for each skill type.
    • Give every long task checkpoints, idempotency, and interruptible splitting; quota exhaustion does not equal failure.
    • Manage quota headroom with monitoring and alerts, and shrink total cost with tiered model routing.
    • For high-difficulty tasks, make explicit “which steps go to the model and which steps a human must take over.”

    The essence of this approach is turning a subjective impression of a model into a measurable, reproducible engineering practice that keeps running sustainably under a budget constraint.