Choosing a Multimodal LLM: Cost, Capability, and Agent Scenarios

    A practical guide to selecting multimodal large language models for production. It covers defining acceptance criteria before comparing specs, estimating total task cost beyond per-call pricing, evaluating tool calling and agent reliability, and building a repeatable testing and routing process.

    1. The First Principle: Work Backward from Acceptance Criteria, Not from Spec Sheets

    When multimodal LLMs move into production, the most common mistake is to look at model specifications first and then search for a use case. Context length, output budget, and support for images or video matter, but they only describe the capability boundary. They do not prove task success rate. What actually drives model selection are the business acceptance criteria: whether the input includes images, tables, or video; whether the output is a short answer, a set of structured fields, or executable code; whether multi-turn tool calls are allowed; whether failures can be retried automatically; and how much manual review costs.

    Selection should therefore follow the sequence of scenario, acceptance, cost, and re-testing:

    1. Define the input modalities and task boundaries. Decide whether the task is batch image-text extraction or direct video understanding, and whether it is single-turn Q&A or multi-turn tool orchestration.
    2. Define measurable outputs, such as field completion rate, code runnability rate, tool-call success rate, citation traceability, and manual rework rate.
    3. Build a cost model. Look beyond per-call price to include thinking token consumption, retry count, context reuse, and human verification cost.
    4. Re-test with the same task set. Run each candidate with identical prompts, identical tools, and identical budgets, and record the results and call logs.

    Price comparisons only become meaningful once quality is broken down into observable metrics.

    2. Unified Comparison Dimensions: Translating Specs into Business Constraints

    2.1 Input Modalities and Task Boundaries

    Image-text understanding, video understanding, and tool calling are separate capability dimensions. Supporting image input does not mean a model handles complex charts reliably. Supporting video input does not mean it can locate events in long videos. When evaluating, sort inputs by difficulty: simple screenshot recognition, dense table extraction, consistency checks across multiple images, video clip understanding, and cross-modal tool calling. Each level needs its own acceptance criteria. Do not summarize all capabilities with a single phrase such as “supports multimodal input.”

    2.2 Context Window and Output Budget

    Million-token context windows have become a common feature of some multimodal models, but a context window is not the same as effective usable length. Key information in a long document may sit near the beginning or end, so stability in the middle needs to be verified with needle-in-a-haystack style tests.

    The output budget also does not equal the final answer length. Thinking consumes generation budget too. In some modes, chain-of-thought and the answer share the budget. In others, input and the current turn’s generation share the total window. Do not assume a model can produce output of the same length after the context is full.

    When comparing models, record the maximum input, maximum output, whether the thinking budget is independent, whether it is limited by the remaining context, and the truncation policy when limits are exceeded. For tasks such as long-document summarization, code modification, and tool orchestration, these constraints directly affect failure rates.

    2.3 Thinking Mechanisms and Billing

    Models differ in how they think. Some enable deep thinking by default, some can switch to a non-thinking mode, some bill thinking tokens separately from answer tokens, and some combine them. Input and output unit prices in a pricing table are only the baseline. Actual cost also includes thinking consumption, retry inputs, tool return content, and cache hit rates. If the platform offers context caching or batch discounts, include them in the estimate, but do not treat an unverified discount as the default cost.

    2.4 Tool Calling and the Agent Loop

    For agent scenarios, the key question is not whether a model can call tools, but whether it can complete the loop reliably. Observe whether tool selection is correct, whether parameters match the schema, whether the model can correct itself from error messages after a failure, whether multi-turn state stays consistent, whether it falls into unproductive loops, and whether timeouts and budget-based stops are supported.

    Measure tool-call success for each task type, such as query, write, code execution, and multi-tool chains. Do not rely only on an overall average.

    3. Cost Model: Per-Call Price Is Not Total Task Cost

    To estimate total task cost, use this general formula:

    1. Total task cost =
    2. Σ(input tokens × input price)
    3. + Σ(output tokens × output price)
    4. + Σ(thinking tokens × thinking price)
    5. + retry and rework cost
    6. + human verification cost
    7. + tool calls and external resource cost

    Retry and rework costs are often underestimated. A cheaper model that needs three retries can cost more than a model that passes on the first attempt. A model with a higher unit price that produces stable output and accurate tool calls can noticeably reduce manual review cost. Therefore, compare cost per qualified result, not cost per call.

    Three core metrics are useful:

    • Average cost per task: total spend divided by the number of tasks.
    • Cost per qualified result: total spend divided by the number of results that pass acceptance.
    • Rework rate: the share of results that need manual editing or a new call.

    Only by tying price to acceptance results can you answer whether a cheap model passes on the first try, and whether an expensive model needs less rework.

    4. General Trial Order: Establish a Baseline First, Then Expand Modalities

    For high-frequency image-text and structured tasks, start with a cost-controlled candidate that has a moderate output budget. This sets a cost baseline. Then add stronger multimodal candidates gradually. For budget-sensitive scenarios that need a large generation budget, focus on long-output stability, truncation policy, and retry cost. For scenarios involving direct video input, first shortlist candidates that support video understanding and can handle long contexts. Then retest them with clip localization, event extraction, and cross-segment consistency tasks.

    The final choice for a complex agent must come from running the same task set. The trial order can be summarized as follows:

    1. High-frequency image-text extraction, classification, and structured output: establish a low-cost baseline first.
    2. Long-form generation, code modification, and complex reasoning: add candidates with large output budgets and compare first-pass success rates.
    3. Video understanding and multi-image consistency: add multimodal candidates and verify temporal and cross-modal alignment.
    4. Multi-tool agents: retest all candidates with the same tool set, prompt template, and budget.

    5. Re-Testing Method: Fixed Task Set, Unified Scoring, Complete Logs

    Re-testing should not rely on hand-picking a few cases that seem to work. Build a task set that covers typical difficulty levels, and define scoring rules for each task type. Illustrative pseudocode:

    1. tasks = load_eval_tasks() # Includes inputs, tools, budgets, expected outputs, and scoring rules
    2. candidates = load_candidates()
    3. for model in candidates:
    4. for task in tasks:
    5. result = call_model(
    6. model,
    7. input=task.input,
    8. tools=task.tools,
    9. max_output=task.budget,
    10. timeout=task.timeout
    11. )
    12. score = judge(result, task.expected, task.rubric)
    13. log(
    14. model=model,
    15. task_id=task.id,
    16. task_type=task.type,
    17. tokens_in=result.tokens_in,
    18. tokens_out=result.tokens_out,
    19. thinking_tokens=result.thinking_tokens,
    20. latency=result.latency,
    21. tool_trace=result.tool_trace,
    22. retry_count=result.retry_count,
    23. score=score
    24. )
    25. report = aggregate(logs, group_by=["model", "task_type"])

    Scoring should cover at least answer correctness, structured field completion rate, tool-call success rate, code runnability rate, citation consistency, and latency and cost. Logs should preserve tool-call traces and failure reasons, so you can tell whether a problem comes from model capability, the prompt, or the tool schema.

    6. Agent-Specific Evaluation

    Agent scenarios also need attention to these governance capabilities:

    • Budget control: set maximum input, maximum output, maximum tool turns, and timeouts for each task to prevent unproductive loops.
    • Failure recovery: check whether the model can read a tool error and correct its parameters, and whether retries are idempotent.
    • State consistency: check whether key context is lost across multi-turn calls, and whether different task states get mixed up.
    • Observability: confirm that each call’s input summary, output summary, tool trace, and cost can be recorded.
    • Security and permissions: confirm that tool calls follow least-privilege rules, and that sensitive operations require human confirmation.

    If you deploy on Baidu AI Cloud, you can use general capabilities such as log services, monitoring and alerting, and permission management to build auditable call chains and a cost dashboard. The goal is not to add more features. The goal is to make every failure attributable and every retry measurable.

    7. Decision Matrix and Rollout Recommendations

    The final selection can be organized as a four-quadrant matrix. The horizontal axis is cost per qualified result, and the vertical axis is task success rate and stability. Prioritize candidates with high success rates and low cost per qualified result. Limit high-success but expensive candidates to high-value tasks. Use low-cost candidates with high rework rates only for low-risk scenarios that can be checked automatically.

    For rollout, take three steps. First, build a baseline with small-traffic testing. Second, run A/B re-testing on a fixed task set. Third, route models by task type. Do not treat a spec ranking as an effectiveness ranking, and do not treat a single demo as a production conclusion. The goal is not to pick one strongest model. The goal is to build a model routing and evaluation process that can be re-tested, governed, and adjusted as the business changes.