LLM Agent Model Selection: Calculate Total Cost Before Comparing Specs
This guide explains how to choose an LLM for an AI agent by estimating total task cost, not just the list price. It covers how success rate, retries, and manual debugging affect cost, how to set acceptance criteria before testing, and how to check context windows, video input, thinking-mode output budgets, and model version pinning before production.
How do you know whether a model is actually cheap?
A model’s list price is only part of the bill. The real cost of a task is the price per call multiplied by the average number of calls, plus the cost of manual debugging and the business loss caused by latency. You can estimate it with this formula:
Total task cost = price per call × average number of calls + manual debugging cost per task + latency loss per task
The average number of calls depends on the task success rate. If every failed call is retried until it succeeds, the average number of calls is roughly 1 divided by the success rate. This is an idealized estimate. Your actual retry strategy will change the result, but the estimate is accurate enough to compare two models at the order-of-magnitude level.
The following example uses hypothetical numbers to show the method. For every 1,000 tasks, Model A costs 0.02 yuan (CNY) per call and has a 60% success rate. It needs about 1,667 calls, for a direct cost of about 33 yuan. Model B costs 0.15 yuan per call and has a 95% success rate. It needs about 1,053 calls, for a direct cost of about 158 yuan. B’s direct cost is 125 yuan higher than A’s. However, about 40 of A’s tasks need manual intervention. At 10 yuan per intervention, that totals 400 yuan. B has only 5 such tasks, totaling 50 yuan. Overall, Model A costs about 433 yuan and Model B costs about 208 yuan. Under these assumptions, the model with the higher per-call price is cheaper overall.
Whether this conclusion holds depends on your real success rate and debugging cost. Put your own business data into the formula instead of relying on the price list alone.
What criteria should you set before testing models?
Define acceptance criteria before you test any model. Each task type needs at least one measurable metric, along with limits on cost, retries, and latency. The table below is a template. Business stakeholders should set the actual values based on their needs.
| Task type | Acceptance metric | Cost baseline | Retry rate limit | Latency threshold |
|---|---|---|---|---|
| High-frequency image-text and structured extraction | Field accuracy meets business requirements | Average cost per task | Set based on business needs | Set based on business needs |
| Very long document summarization or code generation | Content completeness and code runnability | Average cost per task | Set based on business needs | Set based on business needs |
| Video content understanding | Key event recognition rate | Processing cost per video | Set based on business needs | Set based on business needs |
Test models in stages. In the first stage, use a lightweight model for high-frequency image-text and structured tasks to establish a cost baseline. These tasks need less deep reasoning, so response speed and throughput matter most. In the second stage, test models with large output budgets on tasks that involve very long documents and generate large volumes of content. In the third stage, for video tasks, select models that explicitly support video input.
Regardless of the results at each stage, evaluate all models against the same real-world business test set.
When does a 1M-token context window fail?
Some models advertise a 1M-token context window, but many advertise less. Check the official documentation for each model and version, and do not rely on memory. A advertised window also differs from what the model can reliably use. Verify the following three points.
First, check whether the context limit includes the output. Some models share one total window between input and output. The more input you send, the less room remains for output. Do not assume that a full input still leaves space for output of the same length.
Second, check whether the effective input length is shorter than the advertised value. The effective length can depend on the model implementation and the generation budget. Test it with your own long documents.
Third, check whether the model reliably recalls information in the middle of a long text. Key information in the middle of a very long input may be missed or misread. Place key facts at different positions in your test set, then check whether the model retrieves them accurately.
In practice, first use Retrieval-Augmented Generation (RAG) to select relevant passages, and keep the input within the range the model handles reliably. Send a document to a long-context model only when the whole document must be understood at once.
How should you handle video input?
Most mainstream models support image and text input, but video support varies widely. Before integrating, confirm these details in the documentation: supported video formats, frame sampling rate, maximum length per video, and the number of tokens each video consumes.
When a model without native video support receives video data, it may return an error, or it may silently discard the content. Silent discarding is harder to debug. Add modality validation at the system entry point. Identify the input type, then route it to a matching model. If no model matches, return a clear error message instead of falling back to a degraded result.
How do thinking-mode settings affect the output budget?
Some models count their thinking tokens against the output limit, so longer reasoning leaves less room for the visible answer. Other models keep the thinking budget separate from the answer budget. Check the official documentation for each model. The key details are the maximum output length, whether thinking tokens count toward that limit, and whether thinking mode is enabled by default.
Test before launch. Log thinking tokens and answer tokens separately, and track your truncation rate. Handle truncation explicitly. The truncation marker (for example, finish_reason set to length) can trigger a retry with a larger output budget, or the task can be split into smaller steps.
For scheduling, enable deep thinking for complex tasks such as multi-step planning and code repair. Disable it for simple tasks such as classification, field extraction, and format conversion to save tokens and reduce response time.
How should infrastructure support model selection?
Testing and production must use the same model version. Model vendors often add version identifiers, such as dates, to their API IDs when they release updates. If production switches to a new version automatically, the output format may change and break your parser. We recommend the following practices:
- Maintain a model ID mapping table that records the test ID, the production ID, the version date, and the owner.
- Pin the version in production. Before upgrading, run regression tests on the same test set.
- Use canary releases. Route a small share of traffic to the new version first, and compare format parsing failure rates and acceptance metrics.
- Keep a rollback switch so you can revert to the previous version through configuration if problems appear.
When a system calls multiple models, teams often add a unified model gateway to manage several provider APIs. A gateway can use a model service platform from a cloud provider, or it can be built on an open-source project. When choosing one, compare these dimensions: the number of models supported, authentication and rate-limiting capabilities, logging and cost tracking, failover capabilities, data compliance requirements, and fees.
Images and videos are large, so store them in object storage. When calling a model, pass only a reference or extracted key frames. During traffic peaks, use a message queue to smooth the load. Record the latency and success rate of every tool call, and set alerts for both.
Summary
Choose an agent model by setting acceptance criteria first, then comparing models with the real cost formula. Next, verify context, multimodal, and output budget behavior on the same test set. Pinning versions and using canary releases with rollback keeps production results consistent with what you saw during testing.
