Tiered LLM Selection and Configuration for Multi-Agent Systems

    This article explains how to assign large language models to agents in a multi-agent system by role and task difficulty instead of using the largest model for every agent and task. It covers role-based model tiers, evaluation metrics, runtime routing, fallback and cost controls, and an illustrative configuration approach for balancing quality, latency, and API cost.

    In multi-agent system development, the large language model is the reasoning core of each agent. Many teams in early design assign the same largest-parameter model to all agents and all tasks. This increases API call costs and compute overhead, and it also adds response latency. A more reasonable approach is to allocate models according to each agent’s responsibilities and task difficulty, combined with dynamic routing, failure fallback, and cost caps, so that the cost of each call matches its value.

    How Should Model Tiers Be Selected by Agent Role?

    Different agents have very different responsibilities. The main planning agent is responsible for decomposing tasks and needs to read long requirement documents, historical conversations, and feedback from sub-agents, so it is most sensitive to context length and instruction following. The deep execution agent needs to turn plans into code, requiring both reasoning and code generation capabilities. The review and analysis agent is responsible for finding logical gaps and has the lowest tolerance for hallucinations. Code retrieval and simple format conversion have lower model capability requirements, and small models can handle them.

    When selecting models, do not look only at parameter count. Parameter scale is only a rough indicator. A more reliable basis is the team’s own evaluation set. Teams can use real task samples to measure pass rate, format correctness rate, and average latency, then combine these with the per-call price to calculate the cost per passing sample. The table below summarizes the capability requirements and selection approach for each role. Specific tiers should be based on evaluation results.

    Agent Role Core Capability Requirements Metrics to Watch Recommended Model Tier Fallback Plan
    Main Planning Agent Long context, strong instruction following Context window (128K+ recommended), multi-turn instruction retention Large-parameter tier Downgrade to medium-parameter tier and shorten context
    Solution Design Agent Global view, architecture understanding Long-document QA accuracy Large-parameter tier Downgrade to medium-parameter tier and split input documents
    Deep Execution Agent Balanced reasoning and code generation Unit test pass rate, first-generation runnable rate Medium-parameter tier with code-specific optimization Escalate to large-parameter tier for retry after failure
    Review Analysis Agent High-precision analysis, low hallucination Defect recall rate, false positive rate Medium-to-large parameter tier with high alignment quality Route to human review when results are uncertain
    Lightweight Reasoning Agent Formatting and simple inference Response latency, format pass rate Small-parameter tier Hand off to medium-parameter tier after failure
    Code Retrieval Agent Code understanding, keyword extraction Retrieval hit rate, number of relevant files recalled Small-parameter tier or vector retrieval Expand search scope when retrieval is empty
    Subtask Execution Agent Deterministic template-based generation Proportion of outputs matching the template Small-parameter tier, quantized deployment possible Retry once after validation failure, then escalate
    Multimodal Processing Agent Visual understanding, image-text conversion Chart data extraction accuracy, UI element recognition rate Multimodal tier, triggered on demand Degrade to OCR plus text model, or route to manual handling

    The multimodal agent should be invoked only at workflow nodes that need to process interface screenshots, charts, or document images, and it does not need to be resident. Its compute overhead is relatively high. It is recommended to set a switch outside the node and define a degradation path after failure.

    How Should Models Be Dynamically Scheduled by Task Difficulty?

    In addition to static model assignment by role, runtime dynamic routing is also needed based on request difficulty. Routing rules can be determined jointly by task type, input length, and historical failure count.

    Task Category Typical Scenario Routing Target Notes
    Very Hard Reasoning Complex algorithm design, deep logic localization Strong reasoning model with built-in reasoning chain High per-call cost; set a call cap
    Deep Analysis System architecture review, refactoring solution analysis Large general-purpose model Chunk and summarize first when input is too long
    Creative Writing Technical documentation, product copy Model with strong writing capability Factual content requires human review
    Fast Response Simple Q&A, field extraction, format conversion Small-parameter model or quantized version Latency-sensitive; set a short timeout

    Routing rules need accompanying risk controls; otherwise, dynamic routing itself may introduce new failures. The following practices are recommended:

    • Timeout control: Set an independent timeout for each model call. Trigger fallback after timeout instead of waiting indefinitely.
    • Rate limiting and cost caps: Set daily call quotas and maximum tokens per task by project or user. Stop calls and send alerts when the cap is reached.
    • Failure fallback: After the primary model fails, try fallback models in sequence. If all fail, move the request to a human handling queue.
    • Caching and batching: Cache results for repeated retrieval and format conversion requests, and submit small mergeable requests in batches.
    • Evaluation regression: Whenever a model is changed or routing rules are modified, rerun the evaluation set and compare pass rate and cost.

    Routing rules should remain simple. The more rules there are, the harder troubleshooting becomes. In the initial stage, routing can be limited to three or four levels by task type. Refine it after sufficient monitoring data is available.

    Configuration Example (Illustrative Code; Replace per Platform)

    An illustrative configuration should decouple roles from models. Model names are placeholders and should be replaced with the model identifiers provided by the platform in use.