LLM Selection for AI Agents: Cost-Performance Evaluation and Multi-Agent Practices

    This article explains why model selection for AI agents should weigh capability, cost, stability, and controllability together. It covers how to evaluate long-context understanding, iterative code generation, and multi-agent collaboration, how to estimate task cost, and how to build an in-house evaluation process based on cost per successful task.

    1. Background: Selection Criteria Change in the Agent Era

    In single-turn question answering, a model’s general capability is the main metric. In agent applications, tasks usually involve multi-step planning, tool calls, file reading and writing, test execution, and error fixing. A single task can run for tens of minutes and consume dozens or even hundreds of inference calls. At that scale, the price per call and the quota limit directly determine whether a team can use a given model over the long term.

    A common view in the industry is that benchmark scores only reflect performance on a specific set of questions. If a model scores noticeably higher on a benchmark but doubles the cost per task, the productivity gain may not hold. Selection should therefore consider four dimensions: capability, cost, stability, and controllability.

    2. Long-Context Input and File Understanding

    Long-context capability determines whether a model can read a full document, a code repository excerpt, or several financial reports in one pass. When evaluating it, consider the following points:

    • Effective context length: The advertised context window often differs from the length a model uses reliably. Verify this with a “needle in a haystack” test, which places known information in a long text and measures the model’s recall rate.
    • Structured understanding: How accurately a model interprets tables, heading hierarchies, footnotes, and cross-page references directly affects data extraction quality.
    • Source attribution: Whether an answer points to its location in the source text, which makes manual review possible. Output without attribution carries higher risk in finance, legal, and similar fields.

    Example: Build a test set of 50 financial report pages, embed five known values in each file, and measure the model’s extraction accuracy and miss rate. This approach is more quantifiable than judging answers by subjective reading.

    3. Code Generation and Iteration on Long Tasks

    Evaluating code capability should go beyond the quality of a single generated function. For long tasks, the key questions are:

    1. Whether the requirements are broken down sensibly, splitting a large goal into verifiable subtasks;
    2. When modifying existing code, whether the model follows the original structure and avoids unrelated changes;
    3. After running tests, whether the model can locate the problem from failure logs and iterate on fixes;
    4. After an error occurs mid-task, whether the model can recover its state instead of starting over.

    A simple evaluation loop can measure iteration ability:

    1. def evaluate_agent_task(agent, task, max_rounds=10):
    2. history = []
    3. for round_idx in range(max_rounds):
    4. patch = agent.propose_change(task, history)
    5. apply_patch(patch) # assumed to be implemented
    6. result = run_test_suite() # assumed to be implemented
    7. history.append({
    8. "round": round_idx,
    9. "passed": result.passed,
    10. "log": result.summary,
    11. })
    12. if result.passed:
    13. return {"success": True, "rounds": round_idx + 1, "history": history}
    14. return {"success": False, "rounds": max_rounds, "history": history}

    Common metrics include first-round pass rate, average number of iteration rounds, and final pass rate. If only the final success is recorded, you cannot distinguish a task completed in one attempt from one that succeeded by chance after repeated trial and error. The cost gap between these two cases is large.

    4. Engineering Essentials for Multi-Agent Collaboration

    Multi-agent collaboration means that several sub-agents process different subtasks in parallel, and a coordinator then aggregates the results. It suits tasks that can be split and are independent of each other, such as collecting public financial reports from several listed companies at the same time and producing a comparison table. In practice, pay attention to the following points:

    • Task boundaries: Define the input and output format of each subtask in advance. Otherwise, field inconsistencies will appear during aggregation.
    • Concurrency and rate limiting: Parallel calls amplify quota consumption, so set concurrency caps and timeouts.
    • Result verification: The coordinator should cross-check key figures. For example, when two subtasks obtain the same metric separately, compare the two values and flag the item for manual review if the difference exceeds a threshold.
    • Failure isolation: When one subtask fails, the overall task should not be interrupted. The system should record missing items and state them clearly in the final report.

    5. Quota and Cost Model

    Quotas are usually calculated per time window, such as daily or weekly, or by call volume. To estimate cost, build a simple model:

    Cost per task is approximately (average input volume per round × input price + average output volume per round × output price) × average number of iteration rounds.

    1. def estimate_task_cost(rounds, in_tokens, out_tokens, in_price, out_price):
    2. per_round = in_tokens * in_price + out_tokens * out_price
    3. return rounds * per_round

    The function above only illustrates the structure of the calculation. For actual figures, use the current billing rules. In multi-round tasks, context accumulates with each round, so cost grows faster than linearly. The more iteration rounds a task requires, the stronger this amplification effect becomes.

    6. Building Your Team’s Own Selection Process

    1. Map out typical tasks and group them by category, such as document understanding, code modification, data analysis, and multi-step execution;
    2. Prepare 10 to 20 representative samples per category, and define clear pass criteria;
    3. Run the candidate models under identical conditions, and record success rate, number of iteration rounds, time taken, and cost;
    4. Convert the results into cost per successful task, and use this as the main decision metric;
    5. Run a limited pilot for one to two weeks to observe stability and how exceptions are handled.

    Cost per successful task equals total cost divided by the number of successful tasks. This metric reflects both capability and price, so the team does not focus only on unit price or leaderboard scores.

    7. Summary

    In the agent era, model selection has shifted from asking “which model is stronger” to asking “can it reliably complete tasks at an affordable cost.” Readers can use long-text processing, code iteration, multi-agent collaboration, and quota cost as four evaluation dimensions, and compare models with a unified task set and consistent metrics. Building an in-house evaluation process is more reliable than relying on a single leaderboard or personal experience.