How to Choose an LLM Agent: Long-Horizon Capability and Cost per Successful Task

1. From “Answering Correctly” to “Completing the Task”

The traditional way to evaluate a general-purpose LLM focused on knowledge question answering and single-turn code completion. Recent model generations have raised the bar significantly. For enterprises, the key question is no longer whether a model can answer an isolated question, but whether it can complete a task with little or no human intervention—one that may span dozens of steps, multiple tool calls, several files, and multiple subgoals.

This shift reflects how applications are changing. When a model enters a real engineering workflow, it must handle requirement understanding, task decomposition, tool selection, execution, verification, rollback, and retry as a complete loop. A model may be strong at isolated reasoning but fail at step 15 because of context drift, lost state, or poor error recovery, causing the whole task to collapse. For this reason, the ability to complete long-horizon tasks has become the primary selection criterion.

Model rankings change quickly with each release, so any specific conclusion has a short shelf life. A more reliable approach is to build your own evaluation and costing methods and ground decisions in your own business data rather than vendor claims.

2. How to Break Down and Verify Four Capability Areas

2.1 Complex Programming: From Generating Code to Modifying an Engineering Project

Real programming tasks are rarely “write a sorting function.” They usually involve locating modules inside a repository with tens of thousands of lines, understanding existing conventions, making cross-file changes, adding tests, preventing regressions, and submitting reviewable changes. It is useful to separate at least four difficulty levels:

• Unit level: Generating or completing a single function or class, including whether edge cases and exception handling are correct.
• Module level: Adding a feature within existing style and dependency constraints, including whether existing project abstractions are reused correctly.
• System level: Cross-package refactoring, interface migration, build and dependency repair, and whether the change passes the full test suite.
• Long-horizon level: The requirement itself is ambiguous and requires the model to clarify, implement in phases, and verify step by step.

For model selection, the last two levels should be the focus because they show the largest differences between models and are the easiest to hide behind demos.

2.2 Long-Horizon Agents: Planning, Tools, and Recovery

At its core, agent capability means three things: breaking a large goal into verifiable steps; choosing the right tool at the right time; and recovering from failure at a controllable cost. Recommended metrics include:

• Task completion rate: The proportion of tasks that can be delivered without human correction.
• Average interaction rounds: The number of model calls and tool calls needed to reach the goal.
• Human intervention rate: The proportion of tasks that require human correction, and whether intervention occurs during planning or execution.
• Failure-mode distribution: Loops, tool misuse, unauthorized actions, context overflow, and premature declaration of completion.

Among these failure modes, “prematurely declaring completion” and “looping without progress” are especially costly and often overlooked.

2.3 Research Reasoning: Long Vision and Verifiability

Research workflows often have long chains and intermediate steps that cannot be verified directly. Examples include predicting the capsid structure of a biological macromolecule, evaluating delivery vehicles, or deriving testable hypotheses from literature. In these cases, the model’s role is usually not to provide the final answer, but to shorten the exploration path by proposing candidate approaches, designing controlled experiments, and identifying weak assumptions.

Evaluation should not rely on exact answer matching. Expert scoring is more appropriate: the plausibility and novelty of the hypothesis, whether key constraints are missed, whether the experimental design is executable, and whether uncertainty is honestly labeled. A model that can say “I cannot verify this step” often has more engineering value than one that confidently produces a wrong conclusion.

2.4 Visual Understanding: Structuring Graphical Information

When tasks move from pure text to interfaces, drawings, charts, and scanned documents, visual understanding directly determines whether an agent can “see” its environment. The evaluation should focus on the stability of structured output rather than subjective “how good the description is.” Suggested quantitative acceptance metrics include:

• Structured field accuracy: The proportion of extracted key-value fields that match ground truth.
• Coordinate/region recall: The proportion of target regions correctly localized.
• Chart value extraction error: The relative error between read values and ground truth, such as MAPE.

3. Internal Evaluation Sets Are More Reliable Than Public Leaderboards

Public benchmarks solve comparability problems but do not solve your business problems. It is practical to build a small but high-fidelity internal evaluation suite within two weeks.

3.1 How to Construct the Task Set

Sample tasks from historical tickets, real code repositories, and real customer-support or operations scenarios. Divide them into difficulty levels, with 30 to 50 tasks per level as a starting point. Each task must include four parts:

• Input, including necessary context.
• Acceptance criteria, preferably an automatically executable script or unit test.
• Budget limits, such as steps, tokens, and cost.
• Allowed toolset.

Compliance cannot be skipped. Data sources must be authorized and desensitized, with access controls and compliance review. Acceptance scripts should be read-only or run in a sandbox and must not connect to production credentials or personal information.

3.2 How to Design Metrics

Do not rely only on pass/fail. Also record:

• Pass@1 and pass@k to distinguish capability problems from stability problems.
• Total tokens and total cost for completing the same task.
• End-to-end latency at P50/P90, not just average generation speed.
• Human repair time, which is a key conversion from technical metrics to business value.

3.3 Example Evaluation Configuration

suite: agent-eval-v1
tasks:
  - id: repo-bugfix-001
    type: code_system_level
    context: repo_snapshot://2026-11-a
    acceptance: pytest tests/regression -q   # read-only or sandbox execution
    limits: { steps: 60, input_tokens: 2000000, usd: 8.0 }
  - id: ticket-triage-014
    type: workflow
    context: dataset://tickets/014            # desensitized
    acceptance: scripts/grade_triage.py
    limits: { steps: 25, input_tokens: 400000, usd: 2.0 }
scoring:
  primary: pass_at_1
  secondary: [cost_per_pass, human_fix_minutes, p90_latency_s]

4. Inference Speed: Token Generation Speed vs. Task Completion Time

Vendor claims about faster generation, such as “up to about 30% improvement,” are useful signals but cannot be mapped directly to user experience. Time to first token is often determined by queueing and inference depth; no matter how fast generation is, it cannot recover lost queue time. Concurrency limits determine batch throughput. In long-horizon tasks, many tokens are spent on tool calls and rereading context, so generation-speed gains may not improve the task experience. The metrics can be compared as follows:

Metric	Meaning	Impact on experience/cost
Time to first token	Queueing + inference depth	Determines whether the system feels slow; generation speed cannot compensate
Generation speed	Tokens output per second	Noticeable mainly in long-text scenarios
Queueing latency	Concurrency congestion	Dominates P90 at peak times
Concurrent throughput	Number of tasks handled per unit time	Determines cost in batch scenarios

For measurement, use P90 end-to-end latency and specify sample size, concurrency, and whether queueing is included. Industry acceleration tiers often double price to obtain higher throughput. They are suitable when tasks are latency-sensitive and have a high success rate, such as interactive code completion or online customer-support assistance. Offline batch processing, such as bulk code auditing, data cleaning, or offline document understanding, often gains little business value from acceleration tiers while paying twice.

Capacity estimation can use Little’s Law: steady-state concurrency occupancy is approximately λ × T, where λ is the task arrival rate and T is the average time each task occupies resources. For P90 waiting time, you must combine server concurrency limits, queueing models, and real load testing. If only a rough estimate is needed, calculate based on peak arrival rate and leave redundancy. This is enough to decide how much quota is needed and whether peak shaving is necessary.

5. Long Context and Maximum Output: Limits and Pitfalls

Context windows of a million tokens and maximum outputs around 128K change architectural possibilities, but they do not automatically create capability. Three practical observations:

• Effective utilization of long context tends to degrade with length. A widely observed issue is that “middle information is ignored.” A robust approach is to layer context: store raw materials, retrieve only relevant snippets, generate structured summaries and state files during long processes, and read back as needed.
• Maximum output length determines the generation strategy. When a module exceeds a single output limit, the right approach is to split the work into “interface and skeleton + block-by-block implementation + unified verification,” rather than repeatedly retrying the full generation.
• Long-context cost is cumulative. An agent may replay the entire history on every round, causing token usage to grow roughly quadratically with rounds.

def build_prompt(task, state, retriever, max_ctx=200_000):
    stable = render_system_policy(tools=state.tools)   # stable instructions at the front
    notes = render_state_digest(state)                 # structured summary with controlled length
    hits = retriever.search(task.query, top_k=8)       # retrieve only relevant source snippets
    parts = [stable, notes, *hits, task.latest_observation]
    return truncate_by_budget(parts, max_ctx, keep_tail=True)

6. Cost Accounting: From Unit Price to Cost per Successful Task

This is the core of value-for-money judgment and one of the easiest areas to misjudge.

6.1 Billing Is Segmented

Mainstream inference services usually split a request into four billing categories: uncached input, cache write, cache read, and output. Their unit prices differ significantly. Cache read is usually much cheaper than normal input, while cache write is more expensive than normal input. As a result, the same “input price” means different things under different call patterns.

6.2 The Correct Comparison Method

Cost per request, with prices per million tokens:

C = (T_in × P_in + T_cw × P_cw + T_cr × P_cr + T_out × P_out) / 1e6

The task-level metric should be cost per successful task:

CostPerSuccess = Total session cost / Number of successful tasks = C × Average attempts per success
Here, average attempts per success ≈ 1 / pass@1 when attempts are independent and per-attempt cost is similar.

Note: use average attempts per success. Do not separately multiply retry count and pass rate, which would double-count the same effect.

6.3 Example Calculation

The following hypothetical unit prices are used only to explain the method and do not represent any specific product price: input 8, output 40, cache read 0.8, cache write 10 (USD per million tokens).

An agent task has 2.0M total input tokens, split into 0.2M uncached input, 0.2M cache write, and 1.6M cache read (sum = 2.0M); output is 0.15M.

• With cache enabled: 0.2×8 + 0.2×10 + 1.6×0.8 + 0.15×40 = 1.6 + 2.0 + 1.28 + 6.0 = USD 10.88
• With cache disabled: 2.0×8 + 0.15×40 = 16 + 6 = USD 22.00
• If pass@1 is 60%, effective task cost = 10.88 / 0.6 ≈ USD 18.1; if pass@1 rises to 90%, it drops to about USD 12.1

The conclusion is clear: in long-horizon scenarios, cache hit rate and pass@1 often affect total cost more than unit price itself. A model with a 20% higher unit price but a 30% higher success rate is usually more cost-effective. This is the precise meaning of “a more expensive model may be cheaper”—it holds when the success-rate or round-saving gain is large enough, not unconditionally.

6.4 Tiered Routing and Budget Governance

• Tiered routing: Use small models for deterministic tasks such as intent recognition, format conversion, and summarization; use stronger models for planning and complex repair. With reasonable design, most call volume can be absorbed by low-cost models.
• Budget circuit breakers: Limit steps, tokens, and cost per session. When a limit is exceeded, stop and escalate to a human or fall back to a degraded retry.
• Cache-friendly design: Put system instructions, tool definitions, and reference documents at the front of the prompt and keep them stable; put volatile content at the end. This can significantly improve cache hit rates.

MAX_STEPS, TOKEN_CAP, USD_CAP = 60, 3_000_000, 15.0
def run_agent(task, tools, llm, store):
    state = {"task": task, "notes": [], "used_tokens": 0, "usd": 0.0}
    for step in range(MAX_STEPS):
        if state["used_tokens"] > TOKEN_CAP or state["usd"] > USD_CAP:
            store.save_checkpoint(state, reason="budget_exceeded")
            return escalate_to_human(task, state)
        prompt = build_prompt(task, state, tools.retriever)
        reply = llm.chat(prompt, tools=tools.spec(), cache=True)
        state["used_tokens"] += reply.usage.total
        state["usd"] += estimate_cost(reply.usage)
        if reply.finish_reason == "tool_use":
            result = tools.execute(reply.tool_call, sandbox=True, timeout=60)
            state["notes"].append(digest(result))   # compress observations to control context
            continue
        if verify(task.acceptance, reply.output):
            return reply.output
        state["notes"].append(record_failure(reply.output))
    return escalate_to_human(task, state)

7. Multi-Agent Orchestration: Benefits, Costs, and Boundaries

There are already demonstrations in which a single strong model coordinates thousands of sub-agents working in parallel for long periods to generate a bootable prototype operating-system kernel and drivers from scratch. This illustrates the potential of parallel decomposition: a system-level engineering problem can be divided into many independently verifiable subsystems, with the main agent managing interface conventions and integration checks.

However, production adoption must acknowledge three constraints:

1. Coordination overhead is real. As the number of sub-agents grows, the cost of interface alignment, conflict detection, and integration failure increases, while marginal benefit declines.
2. Verification cost may exceed generation cost. Every output must be automatically checked; otherwise, errors in parallel work are amplified rather than diluted.
3. State consistency is the hardest problem. Parallel writes require clear ownership of files/resources to avoid overwriting each other.

In practice, a better pattern is “limited parallelism + strict interfaces + integration gates”: assign ownership by directory or module, fix interfaces with contract tests first, then let the main agent handle integration and regression. Start with 5 to 10 parallel units, and use the integration failure rate to decide whether to scale up.

8. Engineering Implementation: Observability, Rollback, and Control

Long-horizon agents are essentially distributed systems and require corresponding infrastructure. A common approach combines object storage (snapshots, checkpoints, raw materials), message queues (asynchronous processing and peak shaving), container platforms (sandbox execution), logging services (retaining full call and tool traces), and monitoring/alerting (success rate, cost, latency anomalies). Platforms such as Baidu AI Cloud also provide these general infrastructure categories; specific quotas and compliance requirements should follow official documentation and your own review.

Security boundaries cannot be skipped: minimize tool permissions; write and delete operations require human confirmation by default; sandbox network egress should use an allowlist; and all tool calls must retain audit records. Any code-execution capability must run in an isolated environment, and production credentials must not be mounted directly.

9. Selection Checklist and Common Mistakes

Correct approach:

• Run your internal evaluation set instead of relying on generic leaderboard conclusions.
• Focus on system-level code changes and long-horizon task completion, not single-point Q&A quality.
• Record pass@1, average rounds, P90 latency, and human repair time together.
• Compare by cost per successful task, including cache hit rate and retry rate.
• Confirm whether the acceleration tier’s benefit covers its price multiple, and enable it by task type.
• Design a layered context strategy instead of stuffing the entire window.
• Implement budget circuit breakers, least-privilege permissions, audit trails, and a human fallback path.

Common mistakes:

• Looking only at leaderboards instead of your own business data.
• Treating “price per million tokens” as total cost and ignoring cache and retries.
• Not setting budget limits, allowing long-horizon loops to burn money.
• Having no human fallback or rollback, so failures are exposed directly to users.

10. Conclusion

Next-generation flagship models have made real progress in complex programming, long-horizon agents, and research reasoning. Their optimization goal has shifted from “knowing more” to “getting tasks done.” As an engineering decision-maker, however, it is more important to ask: under what conditions is a model stronger, how does cost change with scale, and how do you provide a fallback when failure occurs? Build capability judgments on internal evaluations, value judgments on task-level cost, and reliability on budget circuit breakers and observability. When these three things are done well, each model version change becomes quantifiable benefit rather than a new gamble.