LLM Model Selection in Practice: Balancing Falling Prices and Rising Agent Capabilities
A practical framework for choosing large language models as mid- and low-tier prices keep falling and agent capabilities such as tool calling, multi-step planning, and long-horizon execution become the focus of upgrades. It explains how to read public benchmarks beyond raw scores, build scenario-based evaluation sets with measurable success criteria, tier models and route requests by capability, cost, and latency, and compare options by cost per successful task rather than unit price. It also covers engineering reliable multi-step agents through tool design, budgeted execution control, state and memory management, and observability, and turns model selection into a sustainable process with version pinning, canary releases, content safety, data masking, and audit logging.
1. The Trend: Price Tiering and Capability Tiering Are Happening at the Same Time
Two parallel storylines are playing out in the model market. First, prices for mid- and low-tier models keep dropping. Second, agent capabilities — tool calling, multi-step planning, long-horizon task execution — have become the focus of a new round of upgrades, with noticeably faster release cycles. Viewed together, these two shifts mean two things for developers:
- The trade-off between cost structure and capability structure is being redefined. Work that used to require a flagship model can now shift down to more economical tiers. At the same time, agent workflows that were previously too expensive to scale are starting to meet the conditions for real deployment.
- Selection is turning from a one-time decision into an ongoing engineering problem. With frequent version updates and blurred capability boundaries between tiers, any static decision can go stale quickly.
Selection should therefore return to the basic framework of matching “scenario — capability — cost,” rather than simply chasing leaderboard rankings.
2. Reading Leaderboards Correctly: Four Things Beyond the Score
Public benchmarks are the starting point for selection, not the finish line. Using evaluations of complex tasks such as terminal tasks and parametric design (CAD) as examples, here is what to pay attention to when interpreting scores.
Distinguish “model capability” from “capability after tool augmentation.” Take terminal-style benchmarks as an example: tasks usually require locating a problem, executing commands, and verifying the result in a command-line environment. Public data shows that some mid- and low-tier models, thanks to strong tool-calling and multi-step execution, now score close to top-tier flagship models. Two points matter here:
- These scores reflect the combined result of “model + toolchain.” The sandboxes, command sets, and feedback loops preconfigured in an evaluation environment differ greatly from a real production environment. The same model can produce noticeably different results with a different set of tool interfaces.
- Scores on complex-domain benchmarks (such as parametric modeling tasks) often correspond to two different execution modes: “answering directly” versus “using tools.” The former tests knowledge and reasoning; the latter tests task planning and tool orchestration. Comparing scores across these modes is meaningless — in engineering practice, you should measure the end-to-end behavior of your own system directly.
Pay attention to the error margin and evaluation confidence. Benchmark scores are typically statistical results averaged over multiple runs. When a gap falls within the benchmark’s own variance, you should not draw a strong conclusion from it. When building your own evaluation set, run each task multiple times and track the variance in scores.
Watch out for the mismatch between leaderboards and real scenarios. Public benchmarks are approaching saturation, and some problems may have entered training data, so high scores on general leaderboards do not necessarily transfer to your business data distribution. Scores on terminal and tool-use tasks depend especially on tool definitions, prompts, and runtime environments, so their transferability is limited.
Focus on the intersection of capability and scenario. If your core business is structured extraction and classification, a model’s lead on complex coding tasks has no direct value. Build your evaluation set around the real task distribution, and use reproducible experiments to answer “which model, under what configuration, at what cost, completes the task.”
3. Building a Scenario-Based Evaluation Set: Turning Subjective Impressions into Measurable Metrics
The evaluation set is the infrastructure for selection work. Build it with the following steps:
- Sample in layers from live traffic. Cover high-frequency, long-tail, hard, and edge cases so the distribution closely matches real business.
- Define a decidable success criterion for each task. Output type, key fields, whether tools may be used, and how large a budget is allowed should all be specified.
- Choose the right judging method for each task type. Use exact matching or rule assertions for deterministic tasks; use a combination of “model judging + manual spot checks” for open-ended tasks, and calibrate regularly.
- Fix the run parameters. Record results of the same evaluation set across different models, different temperature settings, and different system prompts separately, to make comparison easier.
The following sample code shows a schematic for evaluation items and the record structure:
from dataclasses import dataclass, fieldfrom enum import Enumfrom typing import Callableclass TaskType(str, Enum):CLASSIFY = "classify" # Classification / extraction: rule assertionsCODE = "code" # Code generation: run test casesAGENT = "agent" # Agent: check the final state@dataclassclass EvalCase:id: strtask_type: TaskTypepayload: dict # Input data or initial tool statejudge: Callable[[dict], float] # Returns a score from 0 to 1weight: float = 1.0 # Weighted by business frequencyrepeats: int = 3 # Repeat runs to observe stability@dataclassclass RunRecord:case_id: strscore: floatsteps: int # Number of reasoning / tool-call stepsin_tokens: intout_tokens: intlatency_ms: intfallback: bool # Whether a fallback or human takeover was triggerederror: str = ""def summarize(records: list[RunRecord], weights: dict[str, float]):groups: dict[str, list[RunRecord]] = {}for r in records:groups.setdefault(r.case_id, []).append(r)total_w = sum(weights.values())weighted = 0.0for cid, rs in groups.items():mean = sum(r.score for r in rs) / len(rs)std = (sum((r.score - mean) ** 2 for r in rs) / len(rs)) ** 0.5print(f"{cid}: mean={mean:.2f} std={std:.2f} repeats={len(rs)}")weighted += weights[cid] * meanreturn {"weighted_score": weighted / total_w,"records": records, # Keep the raw records to compute cost}
Running each case multiple times lets you both observe stability and use the results as a regression baseline after launch. When scores are comparable, compare the efficiency metrics below:
- Steps: the number of reasoning rounds and tool calls needed to achieve the same result — this directly affects latency and cost.
- Token consumption: counted separately for input and output. The longer the task context and the more iterations, the faster the cost grows.
- First-token latency and end-to-end latency: interactive scenarios care more about first-token latency; batch scenarios care more about throughput and unit cost.
- Error recovery rate: whether the model can self-correct when a tool call fails or returns something unexpected. This often reflects real-world usability better than answering correctly on the first try.
4. Model Tiering and Intelligent Routing: Put Each Task in the Right Tier
- Capability tiers. Divide models into three tiers — high-capability, main, and economy — and maintain a “capability–cost reference table” against a fixed evaluation set. Tiers are based on measured results, not marketing claims.
- Task tiers. Score tasks along dimensions such as input complexity, accuracy requirements, whether multi-step tool calls are needed, and the latency budget, then map them to the corresponding tiers.
- Routing layer. Insert a routing layer between the application and the models, responsible for model selection, fallback, caching, and auditing. Rule-based routing is simple and controllable, suitable for getting started. Classifier-based routing has a small model score each request and then dispatch it. When the scores of two request classes overlap heavily, you can first answer with an economy-tier model and then use a confidence threshold to decide whether an upgrade is needed.
- Periodic re-evaluation. Re-run the evaluation set after a new version release or a price change, and update the routing rules. Configurations should be versioned and included in code review and canary release processes.
Sample routing logic:
@dataclassclass Request:messages: list[dict]max_latency_ms: int = 800need_tools: bool = Falserisk: str = "low" # low / medium / high@dataclassclass ModelTier:name: strability: int # 1 to 5, from scenario evaluation resultscost_per_1k: float # Blended unit price, derived from offline evaluation statsp95_latency_ms: intdef required_ability(req: Request) -> int:level = 3 if req.need_tools else 1if req.risk == "high":level += 1 # High-impact decisions (finance, medical, compliance) need stronger reasoningreturn min(level, 5)def route(req: Request, tiers: list[ModelTier]) -> ModelTier:need = required_ability(req)candidates = [t for t in tiersif t.ability >= need and t.p95_latency_ms <= req.max_latency_ms]if not candidates:return max(tiers, key=lambda t: t.ability) # Fallback: guarantee availabilityreturn min(tiers, key=lambda t: t.cost_per_1k) # Cheapest tier that meets the constraints
This mechanism is consistent with the “one-stop model selection” approach of Baidu AI Cloud’s Qianfan model platform. The platform’s model service page typically provides capability parameters, context window sizes, and pricing details, making it easier for teams to maintain their own tier table against a consistent standard. On top of the platform, the routing logic stays under the business team’s control. This keeps ownership of the model mix and fallback strategy in your hands, and makes it easier to migrate smoothly between model providers.
5. Cost Estimation: Don’t Just Look at Unit Price — Calculate “Cost per Success”
Pricing is usually based on separate input and output rates with pay-as-you-go billing, or in tiered/package form; refer to the specific model service you integrate with. What you should really compare during selection is the cost to successfully complete a task once:
Cost per task = (input tokens × input price + output tokens × output price) × attemptsCost per success = cost per task ÷ task success rate
Using the record structure from the sample code, you can directly compute the cost per success for each tier:
PRICES = { # Example prices: cost per 1K tokens"high": 0.030, # High-capability tier"main": 0.008, # Main tier"econ": 0.001, # Economy tier}def cost_per_success(records: list[RunRecord], scores: dict[str, float],tier_of: dict[str, str], price: dict[str, float]) -> dict:stats: dict[str, dict[str, float]] = {}for r in records:tier = tier_of[r.case_id]s = stats.setdefault(tier, {"cost": 0.0, "n": 0, "success": 0.0})s["cost"] += (r.in_tokens * price[tier] + r.out_tokens * price[tier]) / 1000s["n"] += 1s["success"] += scores[r.case_id]return {tier: {"cost_per_task": s["cost"] / s["n"],"cost_per_success": s["cost"] / s["success"]}for tier, s in stats.items()}
This leads to a few practical conclusions:
- The success rate is the denominator of cost. A model with a cheap unit price but a low success rate may have a higher cost per success, and you also pay extra for manual review and customer complaints.
- Context is a hidden cost amplifier. Input and output are usually priced separately. Repeatedly feeding long documents as input, returning too much tool output, and continuously accumulating history across multiple conversation turns all push cost up sharply as context length grows.
- Combining with caching yields considerable gains. A layered cache can cover three kinds of requests: fully identical requests; the shared or similar prefix of system prompts (depending on whether the chosen service supports it); and results from retrieval-type tools that change relatively little. After a cache hit, the actual billed amount is noticeably lower than the original context size.
- You must leave room for variance. The gap between new and older models on public leaderboards often falls within statistical variance, and an accidental single-run result cannot serve as a pricing basis. Reports should give the mean and variance over repeated runs.
6. Agent Capabilities: From “Can Call Tools” to “Can Reliably Complete Multi-Step Tasks”
The reliability of an agent system depends on “model capability × engineering constraints.” On the model side, focus on planning, tool calling, long-horizon memory, and self-correction. On the engineering side, use constraints to keep uncertainty within a controllable range.
6.1 Tool Design: The Only Interface Between the Agent and External Systems
- Clear descriptions: use strongly typed schemas for parameters, and specify units, valid ranges, and what a failure means.
- Idempotent operations: repeated calls should not produce duplicate side effects, which makes retries after failure easy.
- Least privilege: read-only tools can be executed directly; write operations (placing orders, deleting, paying) must involve human confirmation or a second validation.
- Concise returns: return only the fields needed for the decision. Store raw tool output in persistent storage such as object storage, and give the model only a summary or reference to avoid context bloat.
Sample tool registration:
from typing import CallableTOOLS: dict[str, dict] = {"search_docs": {"description": "Search internal docs by keyword; return the most relevant summaries and their reference IDs","parameters": {"query": "string", "top_k": "integer, 1~10, default 5"},"fn": search_docs,"side_effect": "read", # read / write},"create_ticket": {"description": "Create a ticket; with the same idempotency key, repeated calls return the same ticket and do not create duplicate records","parameters": {"title": "string", "idempotency_key": "string"},"fn": create_ticket,"side_effect": "write",},}
6.2 Execution Control: Set a Budget and an Exit for Every Step
- Set a maximum number of steps, a maximum number of tool calls, a per-call tool timeout, and an overall timeout.
- At every step, check whether the budget is exhausted; once it is, switch to a fallback path (return partial results, offer an actionable suggestion, or hand off to a human).
- If the model repeatedly calls the same tool, returns largely duplicated content, or errors out repeatedly, interrupt in time and switch to a different strategy.
Sample execution loop:
import time@dataclassclass Budget:max_steps: int = 12max_tool_calls: int = 8deadline_s: float = 30.0start: float = field(default_factory=time.monotonic)tool_calls: int = 0def exhausted(self) -> bool:return self.tool_calls >= self.max_tool_calls or \time.monotonic() - self.start > self.deadline_sdef run_agent(model_call, messages: list[dict], budget: Budget):for step in range(budget.max_steps):if budget.exhausted():return {"status": "budget_exhausted", "partial": messages}# stream=True: return while generating, reducing first-token latency in interactive scenariosresp = model_call(messages, tools=list(TOOLS), stream=True)if not resp.tool_calls: # The model returns the final answerreturn {"status": "done", "answer": resp.content}for call in resp.tool_calls:budget.tool_calls += 1spec = TOOLS.get(call.name)if spec is None: # Unknown tool: tell the model to avoid infinite loopsresult = {"error": "unknown_tool", "available": list(TOOLS)}elif spec["side_effect"] == "write":# Write operations are confirmed or re-validated at the business layer before executionresult = request_confirmation(spec, call.arguments)else:result = safe_call(spec, call.arguments, timeout=5)messages.append(tool_result(call.id, result))return {"status": "max_steps_reached", "partial": messages}
6.3 State and Memory Management
- Maintain “task goal, completed steps, remaining items, known constraints” as a structured state, rather than relying on the natural accumulation of conversation history.
- When history exceeds a threshold, first summarize and compress the earlier content, then keep key decisions and tool results.
- Bring in external data on demand through retrieval, and annotate the source for traceability.
6.4 Evaluation and Observability
Before and after an agent goes live, focus on observing:
- Task completion rate and first-pass rate
- Average number of steps and tool calls per task
- Tool error rate and error recovery rate
- Human takeover rate
- Cost per successful task
You can collect this data through logging services, set threshold alerts through a monitoring and alerting platform, and feed anomalous samples back into the evaluation set, forming a closed loop of “evaluate — launch — monitor — feed back.”
7. Deployment and Governance: Turn Selection into a Sustainable Engineering Process
- Version pinning and canary release. Record the model version and call parameters used explicitly; run a new version on a small traffic share in a canary, compare its evaluation results against the baseline version, and then gradually expand.
- Prompt management. Put system prompts under version control and treat them, together with the model version, as a rollback-able combination.
- Content safety up front. Connect content safety capabilities at both the entry and exit of the call chain to detect and filter user inputs and model outputs.
- Data compliance. Mask sensitive data before sending it; keep audit logs of the full call chain for model calls, including input summaries, tool calls, and returned results.
- Organizational mechanisms. Periodically re-evaluate the capabilities of each tier and update the routing rules; treat the evaluation set and cost dashboard as shared team assets.
8. Summary and Selection Checklist
The simultaneous drop in mid- and low-tier model prices and the upgrade in agent capabilities make “multi-tier combination + agent automation” a relatively cost-effective engineering path. To check whether your selection work is complete, use the checklist below:
- Are decisions based on a real-scenario evaluation set rather than public leaderboards alone?
- Have you distinguished bare model capability from tool-augmented capability, and evaluated each by execution mode?
- Do you compare tiers by cost per success rather than unit price?
- Do you have a routing layer, caching, and fallback mechanisms to control cost and availability?
- Does the agent have tool idempotency, least privilege, budget control, and human confirmation?
- Do you have version pinning, canary release, and evaluation regression processes?
- Have you integrated content safety, data masking, and audit logging?
When a plan can give a clear answer to each of the questions above, selection is no longer “picking the top-ranked model on a leaderboard,” but an engineering system that can evolve continuously, keep costs under control, and manage risk.
