Choosing a Multimodal LLM: Pricing, Capabilities, and Agent Testing
Handing the same task to different models can cost anywhere from a few cents to several dollars, and the gap comes down to far more than the headline price. This guide walks through a four-step framework for selecting multimodal large language models: define your inputs and acceptance criteria before comparing capabilities, measure real value with cost per successful task rather than per-token price, decide using a reproducible test against your own workloads, and amplify the payoff with engineering. It covers output-budget misunderstandings, why context length is not the same as effective reasoning, how multi-turn agent loops compound input cost, and the common mistakes that lead to overpaying or underperforming.
1. Why Can the Same Task Cost Tens of Times More?
“Read a PDF, patch a function, call an API” — hand the same description to different models and a single run can cost anywhere from a few cents to several dollars. The difference comes mainly from four things:
- Output tokens usually cost much more than input tokens. Generating tokens is computationally more expensive, and chains of thought and code tend to be long, so spending on the output side gets amplified.
- An agent is inherently multi-turn. Tool results and message history are fed back to the model on the next turn, and the context keeps growing with each round.
- Failures trigger retries. A single bad generation has to be redone, and the number of retries multiplies directly into the total cost.
- Thinking mode counts the reasoning process as generated tokens. With thinking enabled by default, even a short task can produce a substantial output bill.
So the real question when choosing a model is not “which one has the lowest unit price” but “can the cheap one pass on the first try, and can the expensive one reduce rework?” This article lays out a four-step framework: filter by hard constraints, calculate the true cost, re-test on your own workloads, and optimize at the engineering level.
2. Step 1: Define Your Inputs and Acceptance Criteria Before Comparing Capabilities
2.1 Input checklist: what does the business actually need the model to “understand”?
Before choosing, list out the input types, the size of a single item, and peak concurrency:
- Plain text: tickets, logs, long documents, code repositories;
- Images: scans, receipts, screenshots, charts, handwritten content;
- Video: training recordings, surveillance clips, recorded meetings;
- Others: audio, structured table files, and so on.
If the business needs the model to understand video directly, first shortlist the candidates that clearly support video input. For cases with only images or scans, you can also evaluate an alternative — “extract frames or slice and process as images” — before deciding whether to pay a premium for native video capability.
At the same time, verify these input constraints:
- The size of a single file and the number of files that can be attached in one request;
- Whether parsing stays stable under long context;
- Whether the vendor specifies different measures of context length, such as the difference between the total window and the maximum input.
2.2 An output budget is not the answer length
“A 1M-token context” is often misread as “1M for input and 1M for output.” In reality, the output side comes down to two things:
- Generation budget: the maximum number of tokens that can be generated in a single call. Tasks that write long-form content, refactor whole files, or generate test cases in bulk should watch this especially closely.
- Budget sharing: models define budgets differently. For some, the chain of thought and the final answer share the generation budget; for others, the input and the current turn’s generation share the total window. The latter means you cannot assume “fill up a long input and still generate another large block.”
Estimate the longest expected output (including the reasoning process) in your business first, then require the generation budget to leave at least 20% headroom. Very long tasks should be designed as segmented generation followed by merging.
2.3 Name suffixes and context length are not agent capability
Suffixes like Max, Flash, and Pro in model names usually reflect the vendor’s positioning around capability or speed, but specific versions within the same naming family may come from different teams and target different goals — you cannot infer agent performance from them. Context length measures “how much you can fit,” while tool-call success rate measures “whether the task gets done right”; the two cannot substitute for each other.
Public benchmarks also need careful reading:
- Most benchmarks are single-turn Q&A, which differs greatly from multi-turn, tool-using agent scenarios;
- Business data has a different distribution from public data, so a model that tops the rankings is not necessarily right for your task;
- Model versions keep updating, and old evaluation conclusions expire quickly.
Conclusion: every spec is only useful for narrowing the candidate list. The final ranking must be decided by your own task set.
3. Step 2: Measure Real Value with “Cost per Successful Task”
3.1 Align the pricing conventions first
Before comparing prices, unify the conventions:
- Whether input and output are priced separately;
- Whether thinking tokens are billed together with the final answer;
- Whether prompt caching is offered, and how cached portions are billed;
- Whether long context is tiered by price;
- Whether batch or asynchronous pricing is available.
Also watch the units: some platforms describe context with K or M, others use exact numbers, so you cannot assume “1M” is exactly equal across all platforms.
3.2 Cost per task and cost per successful task
The cost of a single-turn task can be written as:
Single-turn cost = input tokens × input unit price + output tokens × output unit price (including thinking)Cost per task = single-turn cost × average attempts + failed attempts × human-intervention costCost per successful task = total cost ÷ number of successful tasks
For an agent, input tokens cannot be just the user’s question. Every turn re-sends the tool returns and message history to the model, so the cumulative input is roughly:
Cumulative input ≈ new content per turn × number of turns × (number of turns + 1) ÷ 2
Context grows roughly quadratically with the number of turns. If the system prompt and tool definitions are long and stable, you can consider prompt caching to cut this part of the cost — provided the business data is compliant and the chosen platform supports it.
Also factor in the true cost of a failure, including retries, human review, repair hours, and loss of user experience. The higher these costs, the more “expensive but right the first time” wins.
3.3 Illustrative calculation: how unit price and success rate jointly determine cost
The unit prices below are illustrative values used only to explain the math. Task A is “extract information from images and text into structured JSON,” and Task B is “code repair involving 8 rounds of tool calls.”
| Item | Task A: low-cost model | Task A: high-capability model | Task B: low-cost model | Task B: high-capability model |
|---|---|---|---|---|
| Input unit price ($/million tokens) | 0.5 | 2.5 | 0.5 | 2.5 |
| Output unit price ($/million tokens) | 2 | 12 | 2 | 12 |
| Input volume per call | 4,000 | 4,000 | 8×4,000=32,000 | 8×4,000=32,000 |
| Output volume per call (incl. thinking) | 600 | 1,500 | 8×2,000=16,000 | 8×1,200=9,600 |
| Cost per call | $0.0032 | $0.028 | $0.048 | $0.1952 |
| Measured success rate | 80% | 98% | 40% | 90% |
| Average attempts | 1.25 | 1.02 | 2.50 | 1.11 |
| Human-intervention cost | 0 | 0 | 1 | 0.2 |
| Cost per successful task | $0.0040 | $0.0286 | $2.20 | $0.24 |
Two conclusions:
- Task A: the low-cost model’s unit price is only 1/5 to 1/6 of the high-capability model, and an 80% success rate is already enough, so the cost per successful task is about 1/7 of the latter’s. This kind of high-frequency, clearly structured task is well suited to the low-cost model.
- Task B: although the low-cost model is cheaper per call, its low success rate and many retries mean failed tasks also need human intervention, so its cost per successful task is actually about 9 times that of the high-capability model. Here the high-capability model with shorter output per call is the more economical choice.
If you change “new content per turn” to be increasing (for example, 8k on the first turn and 4k of tool returns per turn after that), estimating with the cumulative-input formula puts the 8-turn cumulative input at around 120k, and the gap between the two model types widens further.
3.4 Lock in the comparison with a script
Put the prices and measured success rates of the candidate models into one configuration table and you can compute the cost per successful task automatically:
from dataclasses import dataclass@dataclass(frozen=True)class Price:input_per_m: float # $ per million input tokensoutput_per_m: float # $ per million output tokens (including thinking)@dataclassclass Candidate:name: strprice: Pricein_tokens: int # input volume per call, including tool returns and message historyout_tokens: int # output volume per call, including thinkingsuccess_rate: float # 0~1, from business re-testinghuman_cost: float = 0.0 # human-intervention cost per failure ($)def cost_per_success(c: Candidate) -> float:single = (c.in_tokens * c.price.input_per_m+ c.out_tokens * c.price.output_per_m) / 1_000_000expected_failures = (1 - c.success_rate) / c.success_ratereturn single * (1 + expected_failures) + expected_failures * c.human_costcandidates = [# The unit prices, token volumes, and success rates below are illustrative; replace them with the chosen platform's pricing and re-test resultsCandidate("Low-cost model", Price(0.5, 2), 4_000, 600, 0.80),Candidate("High-capability model", Price(2.5, 12), 4_000, 1_500, 0.98),Candidate("Agent-A", Price(0.5, 2), 32_000, 16_000, 0.40, 1.0),Candidate("Agent-B", Price(2.5, 12), 32_000, 9_600, 0.90, 0.2),]for c in candidates:print(f"{c.name}: cost per successful task {cost_per_success(c):.4f}")# Low-cost model: cost per successful task 0.0040# High-capability model: cost per successful task 0.0286# Agent-A: cost per successful task 2.2000# Agent-B: cost per successful task 0.2388
4. Trial Order: Build a Baseline First, Add Candidates, Then Decide Together
- First, establish a cost baseline. Pick high-frequency tasks like image/text recognition, information extraction, and format conversion, and run a batch on a low-cost model as a reference for later comparisons.
- Second, add candidates for budget-sensitive scenarios. For cost-sensitive tasks that also need longer generation, such as long-form writing and whole-file code generation, bring in candidates with larger generation budgets for comparison.
- Third, validate multimodal input. For businesses that involve direct video input, prioritize validating candidates with native video support, while also evaluating alternative paths like image frame extraction.
- Fourth, decide together. Run all candidates for a complex agent through the same task set and the same judging script, and rank them primarily by cost per successful task.
5. Step 3: Make Decisions with a Reproducible Business Test
- Draw the task set from real workflows. Sample from production logs and stratify by scenario and difficulty: simple extraction, multi-step reasoning, tool calls, long context, and adversarial samples, each in some proportion, with a total of no fewer than 100 items; keep key scenarios as separate groups.
- Make acceptance criteria machine-judgeable. For example, whether JSON parses, whether required fields are present, whether citations are traceable, whether code passes unit tests, and whether tool parameters match the agreed format. Subjective evaluation is only a supplement.
- Control variables. Candidate models use the same prompt, the same temperature, the same tool descriptions, and the same timeout. Run each task 3–5 times, and take the average success rate and average token consumption.
- Metrics should cover three dimensions: volume, quality, and cost.
| Metric | Meaning | Focus |
|---|---|---|
| Task success rate | The proportion of all tasks that pass acceptance | Overall usability |
| First-pass rate | The proportion that passes acceptance on the first generation | Directly reflects how much rework there is |
| Average turns | The number of model calls needed to complete a task | More turns means higher cost |
| Cost per successful task | Total cost divided by number of successful tasks | The core cost-effectiveness metric |
| Tool-call validity rate | The proportion of parameters that parse and match the format convention | Agent stability |
| Latency | Average / P95 end-to-end time | Interaction experience |
- Pay attention to long-context effectiveness. Set up cases where “the basis for the answer is placed at different positions in a long document” — at the beginning, middle, and end — to check whether the model can reliably retrieve and use relevant information, rather than merely “fitting it in.”
- Keep regressing. Bring the test set into your CI pipeline and rerun it automatically whenever the model version changes, to guard against quality regressions.
6. Step 4: Amplify the Selection Payoff with Engineering
Choosing a model is only the starting point. The measures below are vendor-neutral and apply to any candidate:
- Tiered routing: send simple classification and extraction tasks to a low-cost model; send multi-step reasoning and complex code tasks to a high-capability model. You can use a small model to judge complexity, or upgrade automatically based on confidence or verification results.
- Multi-candidate voting and verification fallback: for key fields, have two models generate separately and compare results; when they disagree, trigger a third call or escalate to human review.
- Context governance: keep only the history that the task needs, summarize and compress early tool returns, and place stable system prompts and tool definitions at the front to improve cache hit rates.
- Output budget control: set differentiated maximum generation tokens for different tasks, and enable streaming responses to reduce timeout risk and improve the experience.
- Prompt caching: applies to repeated prefixes such as system prompts and tool definitions.
- Batch and async: use batch inference for offline tasks to make full use of off-peak resources.
- Monitoring and alerts: set thresholds for success rate, cost per successful task, and P95 latency, and automatically fall back to a backup model on anomalies.
- Unified access: prefer platforms with a complete model catalog, clear pricing conventions, and queryable version and rate-limit information, such as Baidu AI Cloud’s large model service platform. This lowers the cost of integrating and operating multiple models.
7. Common Mistakes and a Conclusion Checklist
The most common mistakes in selection are five:
- Using unit price instead of cost per successful task;
- Equating context length with effective processing capability;
- Inferring agent capability from name suffixes;
- Treating the output budget as the final answer length;
- Using public leaderboards instead of a business re-test.
For implementation, confirm each item on the checklist below:
- Input modality, single-file size, and concurrency are all clearly listed;
- The output budget can cover the longest single generation and leaves headroom;
- Pricing conventions are aligned, including thinking-token and caching rules;
- Compare by cost per successful task rather than unit price;
- The business task set has no fewer than 100 items, and the acceptance criteria can be executed by a machine;
- Agent metrics include first-pass rate, average turns, and tool-call validity rate;
- Routing, verification fallback, and a human safety net are designed;
- Monitoring and regression processes are in place.
In the end, a spec sheet is only good for defining the candidate range. If you get the three steps right — inputs and acceptance first, cost measured by number of successes, and agents decided by a unified task set — selection can stay both within budget and without sacrificing quality.
