Multimodal LLM Selection: Context, Output Budget, and Agent Re-testing Methods

    A practical guide to selecting a multimodal LLM for production. It argues that end-to-end cost per successful task—not public leaderboard scores or per-token price—should drive the decision, then walks through how to read vendor spec sheets without falling into five common traps: context length vs. usable input, output budget vs. answer length, per-modality capabilities, number notation, and shifting names and versions. It breaks per-call costs down into input, cached input, output, and additional charges with a worked formula, explains how to build a stratified business task set for repeatable evaluation, and shows how to turn re-test results into task routing, failure escalation, and regression testing.

    1. The Real Problem: Choose the Model That Minimizes Rework, Not the “Strongest”

    When you integrate a multimodal LLM into a product, the usual debate is “how much stronger is this flagship model than the lightweight one?” But what actually drives your budget is a different question: for the same instruction—read a file, write code, call a tool—why does one model cost only a few cents per turn while another approaches a dollar?

    The gap comes mainly from three sources:

    • Different task types: plain text extraction and cross-file code edits place very different demands on reasoning depth and output length.
    • Different context strategies: repeatedly stuffing an entire codebase or a full document set into the prompt multiplies cost.
    • Different billing models: input and output unit prices, how reasoning tokens are billed, and the discount rules for cache hits and batch calls all vary.

    So there are two more valuable questions to ask:

    1. Can a cheaper model pass on the first try?
    2. Can a pricier model reduce rework?

    If a cheap model needs three rounds—and re-carries the context each time—its end-to-end cost may end up higher. Conversely, if an expensive model raises first-round success from 70% to 95%, what it saves is often the hidden cost of human review and rework. On this basis, the selection principles are:

    • Re-test candidates head-to-head on real business tasks, not against public leaderboard scores.
    • Compare end-to-end cost—how much you spend per successfully completed task—rather than just unit price.
    • Narrow the candidate list with three filters (cost baseline, large-budget generation, and direct video input), then decide complex agent tasks with the same task set.

    2. Give Candidates Roles, Not Labels

    “Largest” and “lightweight” are just names; they don’t indicate capability. Candidates usually fall into four broad categories:

    • General-purpose multimodal (Candidate A): accepts text, images, and video directly; higher single-call output budget, with different caps depending on whether reasoning mode is enabled; suited to video content analysis, cross-modal retrieval, and long agent flows.
    • Lightweight image understanding (Candidate B): mainly text and images; lower single-call output budget; usually more cost-effective per turn; suited to high-frequency image-text extraction, moderation, classification, and structured generation.
    • Long-generation (Candidate C): text and images; output budget up to hundreds of thousands of tokens; suited to whole-repo refactoring, long report generation, and large-scale single-file code rewrites.
    • Balanced long-context (Candidate D): also supports video input; medium default output budget, with generation constrained by the remaining window; suited to “very long input, medium output” tasks such as long contract comparison and multi-year log analysis.

    This grouping only narrows the candidate list. The task distribution in live traffic often differs from expectations, so the model that performs best in your scenario may not be the one ranked highest in public evaluations. The final decision always rests on re-testing.

    3. Five Traps in the Spec Sheet

    Before selection, request the latest specs from the vendor and check them line by line. These five categories are the most error-prone.

    1. Context length is not the same as usable input.

      • The context window is the total space shared by input and generation. The gap between the single-generation limit and the window is the upper bound on usable input.
      • System prompts, tool definitions, history messages, and retrieved reference material all consume that budget.
      • For a model with a 1M window and a 64K single-generation limit, you can submit at most about 960K of input in theory—and you still need to leave room for the system prompt and tool definitions. In engineering practice, people usually reserve an extra 5%–10% buffer to avoid over-length errors.
    2. Output budget is not the same as answer length.

      • With thinking or reasoning mode enabled, the internal reasoning process also counts as generated tokens, and some platforms bill by generated tokens.
      • Some models share one output budget between the chain of thought and the final answer: max_tokens is consumed by reasoning first, and only the remainder becomes answer space.
      • If a 32K output budget is spent entirely on complex reasoning, little may be left for the answer—enough to trigger truncation.
    3. Confirm each modality separately.

      • “Supports image-text understanding” is not the same as “supports direct video input.” An icon badge usually means images, a play-button badge means video, and .audio means audio.
      • Video is also limited by duration, frame rate, and parsing granularity, and this content is billed by token as well.
      • If your pipeline already has ASR transcription, keyframe extraction, and similar stages, a “text + image” model may be more cost-effective for video; whether to take video as direct input should be part of the cost comparison.
    4. Number notation can’t be compared precisely.

      • Vendors may express tokens in decimal (1M = 1,000,000) or binary (1K = 1,024); the same “1M” is 1,000,000 on some platforms and 1,048,576 on others.
      • For precise calculations, convert to concrete token counts and then to the billing unit, to avoid skewing budget estimates.
    5. Names, versions, and documentation change.

      • The shorthand a user says may not match the API ID; for example, “model 4.1 flash” may map to an interface identifier with a specific version-date suffix.
      • Terms like “GA/preview” and “latest/stable” in a product line name may also point to different interface identifiers.
      • Before integration, check the official model list and pricing page, and record the API ID, version, unit price, and date in your configuration or model registry; run regression tests in sync whenever they change.

    4. Cost Breakdown: What You’re Actually Paying For in a Single Call

    A single-turn cost splits into four parts:

    • Input token cost: system prompt, tool definitions, conversation history, reference documents, and the visual tokens of images or video all count. This is where the cost of long context is most easily underestimated.
    • Cached input cost: the portion that hits a prefix cache is usually billed at a lower unit price.
    • Output token cost: the final answer and internal reasoning are billed together, usually at a higher unit price than input, so the output budget is your main cost lever.
    • Additional costs: retries, calls dropped by timeouts, and failed rollbacks are all really billed.

    You can build a rough model:

    1. Single-turn cost ≈ input tokens × input unit price
    2. + output tokens × output unit price (incl. reasoning)
    3. Per-task cost ≈ single-turn cost × average turns × (1 + retry rate)
    4. Total cost = per-task cost × daily task volume

    Assume the input unit price is $a per million tokens and the output unit price is $b per million tokens. A task uses 12,000 input tokens and 800 output tokens per turn, averages 2 turns, and has a 20% retry rate. The per-task cost is roughly:

    1. (12000a + 800b) / 1,000,000 × 2 × 1.2 ≈ 0.0288a + 0.00192b (dollars)

    When a = 0.8 and b = 2, a task costs about 2.7 cents; when a = 4 and b = 16, it costs about 14.6 cents—a difference of roughly 5.4×.

    If daily volume is 100,000 tasks, monthly costs are about $81,000 and $438,000 respectively, a difference of $357,000. And this is only single-turn logic: if a cheap model averages 3 turns, its per-task cost climbs toward 4 cents, and the gap narrows markedly. Unit price is only the starting point; turns and rework are the amplifiers.

    5. Building a Business Task Set: Make Selection Evidence-Based

    Public evaluations reflect general capability; business selection needs a test set that matches your own distribution. A good approach is stratified sampling across six categories, with at least 50 samples each, covering the real long tail:

    • High-frequency structured extraction: pull fields from invoices, forms, and screenshots; output strict JSON; require complete fields, correct types, and passing business-rule validation.
    • Image-text QA and moderation: multi-image comparison, table recognition, and subtle-difference judgment, with reference answers.
    • Code understanding and modification: cross-file reading, changing code to meet requirements, and completing unit tests; judged by compile results and whether tests pass.
    • Multi-step tool calling: query databases, retrieve, and call internal APIs; check tool-selection correctness, argument validity, and recovery after failure.
    • Long-document tasks: contracts, reports, or logs over 100 pages; check key-information location and cross-section reasoning.
    • Video understanding: only your own model can weigh the trade-offs of direct video input vs. keyframe extraction vs. transcribe-then-analyze.

    The execution flow is:

    1. Version locking: fix the API ID, system prompt, tool definitions, temperature, output budget, and other parameters.
    2. Multi-sample re-runs: run each sample 3–5 times to rule out single-run randomness.
    3. Two-dimensional evaluation: make an objective judgment per task, with blind review by business staff when necessary.
    4. Usage recording: log token usage, latency, and cost turn by turn.

    The following is illustrative code showing how to define specs, wrap an adapter layer, and run a batch while recording usage:

    1. from dataclasses import dataclass
    2. from typing import Any, Protocol
    3. @dataclass(frozen=True)
    4. class ModelSpec:
    5. """Spec registry: all numbers follow the official docs, with a verification date recorded at onboarding"""
    6. name: str # internal codename, e.g. "m1"; the external name and API ID live in the config center
    7. api_id: str
    8. doc_date: str # document verification date
    9. max_context: int # context window (tokens)
    10. max_output: int # max output per single call
    11. supports: tuple # supported input modalities, e.g. ("text", "image", "video")
    12. default_output: int = 4096
    13. # Illustrative specs: values are not real parameters, used only to demonstrate differences in how limits are counted
    14. SPECS = {
    15. "m1": ModelSpec("m1", "<api-id-1>", "2026-11-01", 1_000_000, 262_144,
    16. ("text", "image", "video")),
    17. "m2": ModelSpec("m2", "<api-id-2>", "2026-11-01", 1_000_000, 64_000,
    18. ("text", "image")),
    19. "m3": ModelSpec("m3", "<api-id-3>", "2026-11-01", 1_000_000, 384_000,
    20. ("text", "image")),
    21. "m4": ModelSpec("m4", "<api-id-4>", "2026-11-01", 1_048_576, 131_072,
    22. ("text", "image", "video")),
    23. }
    24. class LlmClient(Protocol):
    25. def chat(self, spec: ModelSpec, messages: list, *, max_output: int,
    26. tools: list | None = None, **kwargs) -> dict:
    27. """Unified adapter interface: hides differences between vendors' APIs in parameter names and response structure"""
    28. def estimate_input(spec: LlmClient, messages: list, tools: list | None = None) -> int:
    29. """Estimate input tokens: count with the specific model's tokenizer (illustrative)"""
    30. text = "".join(str(m.get("content", "")) for m in messages)
    31. text += str(tools) if tools else ""
    32. return count_tokens(spec, text) # placeholder: replace with a real tokenizer
    33. def plan_budget(spec: ModelSpec, est_input: int, want_output: int) -> int:
    34. """Allocate the output budget from the remaining window, reserving a 5% buffer"""
    35. usable = spec.max_context - est_input
    36. reserve = int(spec.max_context * 0.05)
    37. return max(0, min(want_output, spec.max_output, usable - reserve))
    38. def run_task(client: LlmClient, spec: ModelSpec, task: dict) -> dict:
    39. """Single task: re-check the remaining context each turn and record usage"""
    40. messages, tools = task["messages"], task.get("tools")
    41. total_in = total_out = 0
    42. for step in range(task.get("max_steps", 8)):
    43. budget = plan_budget(spec, estimate_input(spec, messages, tools),
    44. spec.default_output)
    45. resp = client.chat(spec, messages, max_output=budget, tools=tools)
    46. usage = resp.get("usage", {})
    47. total_in += usage.get("prompt_tokens", 0)
    48. total_out += usage.get("completion_tokens", 0) # includes reasoning tokens
    49. messages.append({"role": "assistant", "content": resp["text"]})
    50. if resp.get("done"):
    51. return {"ok": judge(task, resp["text"]), "steps": step + 1,
    52. "in": total_in, "out": total_out}
    53. return {"ok": False, "steps": task.get("max_steps", 8),
    54. "in": total_in, "out": total_out}

    After the batch run, aggregate these metrics:

    • First-pass rate: the proportion of tasks that meet acceptance criteria on the first attempt.
    • Average turns, success rate, structured-output validity rate, and human-takeover rate.
    • End-to-end cost per task and latency (P95).

    Only when these sit in one table do you have a complete basis for a decision.

    6. Turning Re-test Results into a Selection Decision

    Apply the re-test conclusions through three types of decisions:

    1. Route by task difficulty: hand high-frequency structured tasks to low-cost models; hand tasks that need deep reasoning, direct video input, or very long output to more capable models.
    2. Set up failure escalation: if a task fails within a limited number of steps, or trips a confidence threshold in its output, escalate it automatically to a stronger model. Cost is then spent only on the genuinely hard tasks.
    3. Run regular regression: vendors upgrade models, and a fixed API ID may switch to a new default version. Plug the task set into continuous integration and re-run after every upgrade or parameter change to prevent silent online regressions.

    On the engineering side, follow three cost disciplines:

    • Bring in long context in segments: retrieve first, then insert material in relevance order, instead of carrying everything every time.
    • Compress history promptly in multi-step tasks: replace verbose tool returns with summaries.
    • Keep high-frequency prefixes stable: put system prompts and tool definitions at the very front to try to hit the cache.

    Multimodal preprocessing also matters: scale images down to what they’re actually displayed at, and extract keyframes from video at a business-appropriate granularity, to avoid needless visual tokens.

    7. Pre-Delivery Selection Checklist

    • Candidates listed by role, with input modalities, context, output budget, and billing units verified for each.
    • Confirmed whether reasoning mode shares the output budget with the answer, and how reasoning tokens are billed.
    • Re-tested with a self-built task set, with each sample run multiple times.
    • Compared by end-to-end cost per task (including turns and retries), not by unit price.
    • A routing and escalation strategy designed and wired into regression testing.

    In one sentence: the spec sheet tells you whether a model can be used; business re-testing tells you whether it’s worth using. Start with a lightweight model to build a cost baseline, then bring in long-generation and direct-video-input capabilities by scenario, and finally let the same task set decide—that’s how you find the balance among cost, capability, and stability.