LLM Selection for Agentic Applications: A Practical Guide
This article explains how to choose large language models for agentic applications in a practical, engineering-oriented way. It covers LLM fundamentals and evolution, cloud API, local open-source, and hybrid deployment options, application-level model categories, why no single “best model” exists, agent-specific evaluation dimensions, how to build a custom evaluation set, layered routing and cost control, implementation steps, and common pitfalls. The goal is to turn model selection from a ranking-based guess into a verifiable and iterable engineering decision.
1. Basic Concepts of Large Language Models
A large language model (LLM) is an AI model based on deep learning. It is pretrained on massive text corpora to learn statistical language patterns and knowledge structures, enabling it to understand and generate natural language. In engineering practice, LLMs typically provide the following capabilities:
- Text generation: drafting, rewriting, summarization, translation, and other content-processing tasks.
- Knowledge Q&A: answering questions based on context or external materials.
- Code writing: generating, explaining, and debugging code snippets.
- Logical reasoning: performing multi-step analysis, planning, and judgment.
Research suggests that when parameter scale and training data reach certain levels, models can exhibit abilities that are not obvious at smaller scales, such as multi-step reasoning and more complex code generation. This phenomenon is often called emergent ability. It is important to note that the conditions and stability of emergent abilities are still under research. During model selection, they should not be treated as fixed features that can be relied on long term; they should be verified through actual testing.
2. Technical Evolution
Understanding the current capability boundaries of models requires an understanding of their development path. It can be roughly divided into four stages:
- Emergence of pretrained language models (around 2018): Pretraining methods represented by bidirectional encoding and autoregressive generation appeared, demonstrating the feasibility of learning general language representations through large-scale unsupervised pretraining.
- Rapid growth in parameter scale (2019 to 2020): Model scale and training data volume continued to expand, and generation quality and task transfer capability improved significantly.
- Popularization of conversational applications (2022 to 2023): Conversational models aligned through instruction tuning and human feedback entered public view, and LLMs shifted from research tools to general services.
- Multimodal and agent stage (2024 to 2025): Models began to process images and other inputs, and tool-calling capabilities improved substantially, allowing them to call external functions through structured protocols to complete operations such as search, calculation, and data queries.
This evolution has a direct impact on model selection. Early selection mainly compared generation quality, while current selection also needs to consider the stability of tool calling, the level of multimodal input support, and the ability to maintain state consistency across multi-turn tasks.
3. Classification by Deployment Mode
Deployment mode determines the cost structure, data control, and operational burden. It is the first dividing line in model selection.
3.1 Cloud API Models
Cloud API models are hosted by service providers and called by users through network interfaces. Their main characteristics are as follows:
- Cost structure: Usually billed by usage volume. Upfront investment is low, making them suitable for businesses with uncertain usage.
- Operational cost: Model upgrades, scaling, and availability assurance are handled by the service provider, so the user has less maintenance work.
- Data boundary: Request content passes through external networks. When sensitive data is involved, it is necessary to confirm the provider’s data retention policy, compliance certifications, and cross-border transfer requirements.
- Dependency risk: Interface changes, rate limiting, and service interruptions directly affect the business. Retry, fallback, and timeout controls should be designed at the architecture level.
3.2 Open-Source Local Models
Open-source models can be deployed on self-owned servers or private cloud environments. The user has full control over model weights, the inference process, and data flows.
- Data controllability: The inference process does not leave the domain, making this suitable for data-sensitive scenarios such as finance, healthcare, and government affairs.
- Customization space: Domain fine-tuning, quantization compression, and inference acceleration are possible.
- Resource threshold: Sufficient GPU memory and RAM are required. Large-parameter models often require multiple cards or high-performance accelerators.
- Operational burden: Model loading, version management, monitoring and alerting, and capacity planning must be handled in-house.
The hidden costs of local deployment are often underestimated. A medium-scale model that has been quantized and can run stably on a single card may have a lower overall cost than equivalent cloud calls, but only if hardware utilization is sufficiently high.
3.3 Hybrid Deployment
Hybrid deployment combines the two approaches, usually routing by data sensitivity and task complexity:
- Requests containing sensitive information are handled by local models.
- Requests with strong general-purpose requirements and high reasoning demands are handled by cloud models.
- A unified gateway layer performs routing, authentication, and logging between the two.
The key to hybrid deployment lies in routing rule design. Rules that are too coarse can cause sensitive data to enter the cloud by mistake, while rules that are too conservative can put pressure on local resources.
4. Classification by Application Scenario
From an application perspective, models can be roughly divided into three categories:
- General-purpose large models: Broad coverage, suitable for Q&A, writing, summarization, and various mixed tasks. They have strong general capabilities but may not be optimal for specific industry terminology and specialized processes.
- Vertical domain models: Specially optimized on industry corpora such as healthcare, legal, and finance. They offer better terminology accuracy and compliant expression, but general capabilities may be somewhat reduced.
- Tool-calling models: Optimized for agent scenarios, with a focus on adherence to function-calling formats, multi-step planning capability, and error recovery capability.
During model selection, first determine the primary contradiction in the business. For knowledge-intensive Q&A, vertical domain models may be more suitable. For automation tasks that require calling multiple systems to complete a process, tool-calling capability should be the top priority.
5. Why There Is No Universal “Best Model”
Many teams directly look up composite rankings during model selection and then choose the first-place model. This approach ignores three issues:
- The ranking test set may not match the business. Public benchmarks measure performance on specific tasks and formats, and they may not correlate with your prompt style, data format, and output constraints.
- There are trade-offs among capability dimensions. Models with stronger reasoning ability usually respond more slowly and have higher per-call costs. Models with longer context windows may suffer from attention dilution in some scenarios.
- Version iteration is fast. Models from the same vendor may differ greatly across versions, and ranking data has limited timeliness.
Therefore, a reasonable approach is to reframe the question as: Under my tasks, my constraints, and my budget, which model combination is most suitable?
6. Selection Dimensions for Agent Scenarios
Agent scenarios impose different requirements on models than ordinary conversational scenarios. The following table lists dimensions that require focused evaluation.
| Dimension | Focus | Typical Risk |
|---|---|---|
| Tool-calling adherence | Whether the model can output call requests according to the agreed parameter structure | Missing parameters, incorrect field types |
| Multi-step reasoning consistency | Whether the model can maintain goals and intermediate state in long-chain tasks | Drifting from the goal midway, repeated execution |
| Context handling capability | Utilization of long documents, history, and tool return values | Key information being ignored |
| Error recovery capability | Whether the model can correctly interpret tool errors and retry | Falling into an infinite loop or abandoning the task |
| Instruction following and safety boundaries | Whether the model can refuse unauthorized operations and ignore embedded malicious instructions | Being induced by external content to perform dangerous actions |
| Latency and cost | Time per call and cost per thousand requests | Excessive user waiting, cost exceeding budget |
| Data compliance | Whether data storage, transmission, and retention meet requirements | Sensitive information crossing boundaries |
Among these, safety boundaries are often overlooked. Agents read web pages, documents, and database content, and this external content may contain text that attempts to change model behavior. System design should treat external content as data rather than instructions, and permission checks should be performed at the tool execution layer.
7. Building a Custom Evaluation Set
Rather than relying on public rankings, it is better to build an evaluation set that is close to the business. A feasible process is as follows:
- Collect samples: Extract typical requests from real business logs, covering common scenarios, boundary scenarios, and failure scenarios, with at least dozens of samples per category.
- Define standard answers or scoring rules: For tasks with deterministic answers, compare results directly. For open-ended tasks, create a scoring rubric, such as scoring accuracy, completeness, and format compliance separately.
- Fix test conditions: Unify prompt templates, temperature parameters, output length limits, and tool definitions to avoid variable interference in comparisons.
- Automate execution and recording: Call candidate models in batches and record outputs, latency, token usage, and failure reasons.
- Analyze and iterate: Aggregate results by task type, identify the strengths and weaknesses of each model, and then optimize prompts or routing rules for those weaknesses.
The following pseudocode illustrates the structure of an evaluation framework. The actual implementation should be adjusted according to the interface specification:
for case in evaluation_set:for model in candidate_models:start = now()response = call_model(model, case.prompt, config=fixed_config)latency = now() - startscore = {"accuracy": check_answer(response, case.expected),"format_ok": validate_schema(response, case.schema),"tool_call_ok": check_tool_calls(response, case.allowed_tools),}record(model, case.id, score, latency, response.usage)report = aggregate_by_task_type(records)
During evaluation, retain a sufficient sample size and run multiple times to observe output fluctuations. For generative tasks with randomness, a single result is not enough to draw conclusions.
8. Layered Routing and Cost Control
In real systems, it is rare to use only one model. A more common approach is to layer by task difficulty:
- Lightweight layer: Handles classification, extraction, simple rewriting, and other relatively deterministic tasks using models with low inference cost and fast response.
- Standard layer: Handles most routine conversations and tool calls, balancing quality and cost.
- High-capability layer: Handles complex planning, multi-step reasoning, and high-risk decisions, allowing higher latency and cost.
Routing can be implemented based on rules, such as assigning models by request type, input length, or user tier. It can also use a lightweight model to judge task difficulty first and then decide whether to escalate. Whichever method is used, the hit rate and failure rate of each layer should be recorded so that routing can be continuously calibrated.
The following measures can further reduce costs:
- Cache results for identical or highly similar requests and set a reasonable expiration time.
- Streamline system prompts by removing instructions unrelated to the current task.
- Control the length of tool return values and keep only the fields the model needs.
- Set a maximum output length to avoid meaningless long-form generation.
9. Implementation Steps
To translate the above methods into engineering practice, proceed in the following order:
- Define task boundaries: List all task types the system needs to complete, and mark the accuracy requirement, latency limit, and data sensitivity level for each type.
- Determine deployment constraints: Based on data compliance requirements and budget, determine the available combination of deployment modes.
- Build a candidate set: Under the constraints, select three to five candidate models. Too many candidates increase evaluation cost, while too few may miss more suitable options.
- Run custom evaluations: Complete the comparison according to the process in Section 7 and produce a comprehensive table covering quality, latency, and cost.
- Design routing and fallback strategies: Specify a primary model and a backup model for each layer, and define fallback paths for timeouts, rate limiting, and failures.
- Roll out gradually and monitor continuously: First validate on a small proportion of traffic, monitoring error rates, user feedback, and cost changes. Re-run the evaluation set when model versions are updated.
10. Common Pitfalls
- Looking at only a single metric. Focusing only on accuracy while ignoring latency and cost can make the system unusable under real load.
- Treating the model as the entire solution. Prompt design, retrieval augmentation, tool definition quality, and output validation often affect final results no less than differences between models themselves.
- Ignoring version pinning. Without fixing the model version, online behavior may change without awareness.
- Ignoring migration cost. Deep dependence on the special format of a particular interface will greatly increase the cost of replacing the model later. It is recommended to encapsulate a unified calling interface at the business layer and isolate model differences in an adaptation layer.
Concluding Judgment
Choosing an LLM is essentially a trade-off among quality, latency, cost, data control, and replaceability. There is no single answer that applies to all scenarios, but by clarifying task requirements, building an evaluation set close to the business, adopting layered routing, and retaining an adaptation layer, model selection can be transformed from an experience-based judgment into a verifiable and iterable engineering decision.
