# LLM Selection Guide: Context Windows, Output Budgets, and Agent Benchmarking

    How to choose an LLM for production workloads by evaluating specs, cost, and real business tasks rather than model names. The guide explains how to read modality limits, context windows, output budgets, and reasoning settings consistently across vendors; how to convert per-token prices into a cost per task that includes cache hits, multimodal conversion, and retry amplification; and how to benchmark agents on tool-selection accuracy, structured output validity, first-pass rate, and P95 latency. Includes specification checklists, token-unit normalization code, a cost model with a worked example, a tiered routing pattern, and an evaluation harness skeleton.

    1. The real problem: pick the right model, not the cheapest or the biggest

    For the same task — reading a contract, editing a function, debugging an API call — switching models can change the cost of a single call by several times over. Most of that gap comes from three things: input and output length, whether reasoning is enabled, and how each vendor defines its pricing units. What matters more is whether a cheap model gets it wrong on the first attempt, and whether an expensive model actually reduces rework and manual cleanup.

    Specs deserve the same scrutiny as prices. Current mainstream multimodal models commonly offer context windows in the million-token range, and input support has expanded from “text + images” to “text + images + video.” But the terms “context,” “output limit,” and “reasoning budget” in vendor documentation do not mean the same thing across providers: some windows hold input and this turn’s generation together, while some output budgets also include the chain of thought. Without a common baseline, any comparison of “who has the longer context” or “who outputs more” is likely to be wrong.

    This guide breaks the problem into three layers:

    • Specifications: verify modality, context, output budget, and reasoning behavior to confirm whether the task is physically possible.
    • Cost: normalize unit prices and estimate the expense of one call, one task, and a full batch run.
    • Quality: evaluate with a business task set and measure first-pass rate, rework rate, and end-to-end latency.

    All three are required. Specs determine what is possible, cost determines what is feasible, and quality determines whether it is worth it.

    1.1 Specification checklist

    Dimension What to verify Why it matters
    Input modality Text, images, video, plus limits on resolution, image count, and duration Determines whether multimodal assets can be sent directly or must be preprocessed
    Context How the window is defined, and whether input has a separate cap Determines whether a long document, long logs, or a whole codebase can be sent in one shot
    Output budget Default value, maximum, and whether it includes the chain of thought Determines whether long code, long reports, and large batches of structured output can be generated in one call
    Context sharing Whether input, this turn’s generation, and the chain of thought share one window Directly determines whether “million-token input + large output” is actually possible
    Reasoning mode Whether it can be switched off, how the budget is measured, how it is billed, how latency changes Affects the tradeoff between reasoning quality on one side and cost and latency on the other
    Pricing basis Input, output, cache hits, async batching, and how multimodal input is converted Determines whether cost estimates are accurate
    Version identifier The vendor’s API model ID and version, and whether older versions can be pinned Affects reproducibility and post-launch regression testing

    1.2 Input modality drives “preprocessing cost”

    In multimodal workloads, anything other than text is usually the hardest part to handle.

    • Images: e-commerce product photos, invoices, screenshots, scanned documents. Confirm the resolution and per-image size limit, the maximum number of images per request, and whether charts and small type can be read reliably. If each image has to be downsampled or split before sending, both recognition accuracy and token consumption change.
    • Video: product demos, screen recordings, surveillance clips. Confirm the duration limit, the sampling frame rate, and whether frames and audio track are understood together. If the model does not accept video natively, the application must extract frames and send them one by one. That increases the number of calls and inflates input tokens.

    So the first step is a coarse filter by input type. For high-volume image-centric workloads, prefer models with native text and image support. For workloads that include video, first verify native video support.

    1.3 Context: distinguish “window size” from “effective utilization”

    Million-token context has real value:

    • An entire contract, financial report, code repository slice, or long conversation can be sent in one shot without chunking and stitching.
    • Cross-document comparison, global search, and maintaining history in a long agent chain all have more headroom.

    Two caveats are unavoidable, though:

    1. A longer context does not mean dumping everything in works. In very long inputs, key information in the middle is easily overlooked — commonly called “lost in the middle.” The longer the input, the higher the per-call token cost and the larger the time to first token. For tasks with high information density and a clear scope, retrieval-augmented generation (RAG) that filters first and then reads is usually more reliable and cheaper than loading everything.
    2. Check how the window is counted. Some vendors count in decimal (for example 1,000,000), others in binary (for example 1,048,576), and in Chinese both get shortened to “1M” or “million.” The difference is about 4.9%, which near the limit decides whether a request succeeds or fails outright. In addition, some models impose a separate cap on input that is slightly below the total window.
    1. # Note: the two readings of "1M" are not the same number
    2. assert 1_000_000 != 1024 * 1024 # 1,048,576 — about 4.9% apart
    3. def fit_check(model_spec, input_tokens, output_tokens):
    4. """Check the window and output budget before sending, so you can degrade
    5. early instead of failing the request or getting a truncated answer."""
    6. # When the total window holds input and this turn's generation together,
    7. # you cannot use the whole window.
    8. if model_spec.shared_window:
    9. if input_tokens + output_tokens > model_spec.window:
    10. return "input_output_exceed_window"
    11. # Some models impose a separate cap on input.
    12. if model_spec.max_input and input_tokens > model_spec.max_input:
    13. return "input_exceed_limit"
    14. # When the output budget includes the chain of thought, the final answer
    15. # needs room reserved separately.
    16. if output_tokens > model_spec.max_output:
    17. return "output_exceed_budget"
    18. return "ok"

    1.4 Per-call output budget is not the length of the final answer

    The output budget is often misread as “how long the model can write.” The reality is more complicated:

    • Default and maximum differ. The budget you actually get depends on how the request is configured, not on the maximum in the documentation.
    • Reasoning consumes the budget. With reasoning enabled, the chain of thought and the final answer usually share the same budget. Reserving room for the answer means accounting for two things:
      • Complex tasks can produce chains of thought thousands of tokens long.
      • Once the limit is hit, the reply may be truncated — and truncation behavior differs across vendors, so it has to be confirmed one by one.
    • The sharing relationship with the total window differs. Some models share one window between generation and input. In that case, “fill a million-token input and then output hundreds of thousands of tokens” is a misreading: the output you can actually use equals the total window minus the input already consumed.
    • In a shared multi-turn window, “input” keeps growing. In tool-calling scenarios, every turn’s message history counts as input for the next turn. The more turns, the faster the context fills up.

    So for workloads that need long code generation, whole reports, or large batches of structured output, verify specifically whether the output limit is enough and how continuation works after a truncation.

    1.5 Reasoning mode: evaluate capability, cost, and stability together

    Models that support a reasoning mode can do more internal reasoning before answering, which often helps with math, logic, and complex code tasks — but not every task benefits.

    • Whether it can be turned off. For simple tasks like classification, extraction, and formatting, disabling reasoning is usually faster and cheaper.
    • How it is billed and counted. Confirm whether chain-of-thought tokens count toward output and whether they are charged at the output price.
    • Whether results are stable. Reasoning has some inherent randomness, so repeated runs on the same input can produce different results. High-precision workloads should require structured output and validate results programmatically.

    1.6 Version identifiers: reproducibility matters more than names

    • Do not treat a version number as an API identifier. The model ID used in a vendor’s API is not always the same as the version name and date suffix shown publicly. For example, some models use a short API ID (of the form xxx-flash) while the public version name includes a version number and release date.
    • Watch for silent updates. The service behind the same API ID can be upgraded by the vendor, so results you tested before launch may not stay reproducible. For workloads with high stability requirements, settle these questions during selection:
      • Can historical versions be pinned?
      • Does the vendor publish a version change log?
      • Are canary rollout and rollback available?

    2. Pricing: from unit price to cost per task

    2.1 Look beyond the unit price

    LLM billing is usually token-based, but the full cost is far more than “input price × input length + output price × output length.” At minimum, also consider:

    • Output is usually priced well above input. For long-output tasks, cost is driven mainly by how much is generated.
    • Cache hit pricing. The portion served from a context cache is usually billed at a lower rate. Multi-turn conversations, fixed prompts, and repeated prefixes benefit the most.
    • Batch or async discounts. For offline work that tolerates latency — data labeling, bulk archiving — evaluate an asynchronous batch API.
    • How multimodal input is converted. An image is typically converted into some number of tokens based on resolution, tiling, or image count, so estimating from file size is not valid.
    • Supporting resource costs. Preprocessing such as document parsing and video frame extraction adds storage and compute overhead, which belongs in the usage of general cloud products like object storage and container platforms.

    2.2 Normalize token units

    1. import re
    2. def to_tokens(text: str) -> int:
    3. """Normalize notations such as 96k / 1M / 1,048,576 into a token count."""
    4. s = text.strip().lower().replace(",", "").replace(" ", "")
    5. if s.endswith("k"):
    6. return int(float(s[:-1]) * 1000)
    7. if s.endswith("m"):
    8. return int(float(s[:-1]) * 1_000_000)
    9. return int(s)
    10. # "1M" from different sources is not equal — normalize before comparing
    11. assert to_tokens("96k") == 96_000
    12. assert to_tokens("1M") == 1_000_000

    2.3 Total cost formula: include rework

    Suppose a task has a first-pass rate of p — the probability that one call passes acceptance — each attempt costs the same, and failures are retried independently until success. The expected number of attempts is then 1 / p, which gives:

    1. Expected cost per task ≈ (input cost per call + output cost per call + vision conversion cost per call) / p
    2. Total batch budget ≈ number of tasks × expected cost per task

    This means a model that is 30% cheaper per token can still cost about 28% more per task if its first-pass rate drops from 0.90 to 0.70 (0.70 / 0.90 ≈ 0.78).

    For an agent with tool calls, each call needs an extra multiplier for the number of turns, k:

    1. Expected cost per task ≈ (input cost per call + output cost per call) × k / p

    A typical ReAct-style loop looks like this: the model first decides which tools it needs and returns a call request, the system executes the tool and writes the result back into the context, and the model then decides the next step. This repeats until an answer is produced, or a step or budget limit is reached.

    1. def run_agent(task, model, tools, max_steps=8, token_budget=200_000):
    2. messages = [{"role": "system", "content": SYSTEM_PROMPT},
    3. {"role": "user", "content": task}]
    4. used = 0
    5. for step in range(max_steps):
    6. resp = model.chat(messages=messages, tools=tools)
    7. used += resp.usage.total_tokens # usually includes the chain of thought
    8. if used > token_budget:
    9. return {"status": "budget_exceeded", "step": step}
    10. if not resp.tool_calls:
    11. return {"status": "done", "answer": resp.text, "step": step}
    12. for call in resp.tool_calls: # independent tools can run in parallel in the same loop
    13. result = dispatch(call) # dispatch using the tool name and arguments from the model
    14. messages.append({"role": "tool", "name": call.name, "content": result})
    15. return {"status": "step_limit"}

    This code deliberately omits retries, timeouts, rate limiting, idempotency, and result validation — the production concerns. It makes two points:

    1. The step limit and token budget must be controlled explicitly, otherwise cost becomes unpredictable when a task fails.
    2. Each turn’s messages becomes the next turn’s input, so more turns mean a more expensive per-call input, and a larger k.

    2.4 A worked example

    The parameters below only demonstrate the calculation method. Replace them with the vendor’s current price list for real work.

    Parameter Value
    Input price 2 CNY per 1M tokens
    Output price (including chain of thought) 8 CNY per 1M tokens
    Task volume 200,000 items, averaging 6,000 input tokens per call (4,000 cacheable) and 800 output tokens
    Cache hit price 20% of the input price
    Average calls per task 2.5
    First-pass rate p 0.90

    First, the cost of one call. Input: 4,000 × 2 × 20% + 2,000 × 2 = 5,600 (CNY per 1M tokens). Output: 800 × 8 = 6,400. Total per call: 12,000, or 0.012 CNY.

    Then the total task cost: 0.012 × 2.5 / 0.90 × 200,000 ≈ 6,667 CNY.

    Sensitivity across three scenarios:

    Scenario Cost per task Total budget for 200,000 items
    Baseline ~0.0333 CNY ~6,667 CNY
    Cache disabled ~0.0417 CNY ~8,333 CNY
    First-pass rate drops to 0.70 ~0.0429 CNY ~8,571 CNY

    The conclusion is clear: enabling the cache and raising the first-pass rate save more money than simply driving down the unit price.

    3. Agent scenarios: naming and context length prove nothing about success

    Plenty of selection write-ups end with “pick the Max tier” or “pick the Pro tier,” justified by “the name is stronger” or “the context is longer.” Whether an agent succeeds depends on a set of measurable indicators:

    • Tool selection accuracy: whether it picks the right tool when several have similar functions.
    • Argument-filling accuracy: whether field types, enum values, and required parameters match the contract.
    • Multi-step planning: whether the task is decomposed into the right sequence of steps, and whether it can recover after a failure.
    • Context management: whether it keeps the goal from drifting and does not drop constraints across a long history.
    • Structured output stability: whether repeated runs reliably produce results a program can parse.

    None of these correlate with the model’s name; they can only be measured. Especially in workloads with “similar tool descriptions, many parameter constraints, long step chains,” a longer context does not automatically turn into a higher completion rate.

    3.1 How output budget and window sharing affect agents

    Agent output is often code patches, SQL, or large batches of JSON. If the per-call output limit is too small, truncation happens constantly, forcing continuation and stitching — which adds calls and adds chances to stitch incorrectly.

    When input and this turn’s generation share the total window, a larger k leaves less room for output. So long-chain agents usually need:

    • Context compression and summarization;
    • Trimming and structured summarization of tool results;
    • Retrieving history instead of reloading all of it.

    3.2 Prefix caching: the cost lever for long agents

    Every agent turn carries the same system prompt, tool list, and early history. If the service supports prefix caching, repeated billing for this content drops substantially, and the benefit grows with k. During selection, confirm whether the cache activates automatically, how long it stays valid, and how cached portions are priced.

    3.3 Multimodal agents: let the model “see” before acting

    Agents that read screenshots, fill forms, and locate UI elements must be evaluated on how visual understanding and tool calling work together:

    • The vision model identifies elements in the interface or image.
    • The tool layer performs clicks, queries, and submissions.
    • Whether the structured description passed between the two is stable directly determines the success rate of the whole chain.

    3.4 Tiered routing: pick separately for simple and complex tasks

    Not every request deserves the most expensive model. A lightweight routing layer in the application can handle this:

    • Short input, a single tool, structured output: route to the cost-baseline model.
    • Multiple tools and long-chain reasoning: route to the flagship model.
    • Primarily visual input: route to a model with native image support.
    • Video input: route to a model with native video support.
    1. def route(req, registry):
    2. # 0. Preprocessing cost: does the asset need transcoding, frame extraction, or splitting?
    3. if req.has_video and not req.model.native_video:
    4. return "preprocess_or_reroute"
    5. if req.input_tokens > req.model.max_input:
    6. return "preprocess_or_reroute" # compress, retrieve, summarize
    7. # 1. Feasibility: window and output budget
    8. if fit_check(req.model, req.input_tokens, req.est_output) != "ok":
    9. return "preprocess_or_reroute"
    10. # 2. Routing rules
    11. if req.needs_multimodal and req.task_type in ("extract", "classify"):
    12. return registry["vision_baseline"] # high-volume image+text, structured extraction
    13. if req.needs_multimodal:
    14. return registry["vision_flagship"] # complex reasoning over image+text
    15. if req.agent_steps >= 4 or req.tool_count >= 5:
    16. return registry["agent_flagship"] # complex agent
    17. if req.est_output > req.model.default_output:
    18. return registry["long_output"] # long code, long reports
    19. return registry["text_baseline"] # default: cost baseline

    One emphasis: routing rules are only an engineering device. Which class of model each branch should use still has to be decided by the evaluation results in Section 4, not guessed from spec sheets.

    4. Evaluation: turn “selection” into a verifiable engineering process

    4.1 Building the evaluation set

    • Real sources: sample from business logs, tickets, and historical conversations, anonymize, and load them in. Do not substitute a public leaderboard for business validation.
    • Layered coverage: cross-stratify by input modality (text only, image + text, video), context length (short, medium, near the window limit), and output length (short answer, long code, large JSON batches).
    • Golden set and challenge set: the golden set drives routine regression; the challenge set collects known failure cases — very long inputs, blurry images, easily confused tools, multi-step dependencies.
    • Version traceability: record the model ID, parameter configuration (temperature, output budget, reasoning on/off), and evaluation time for every sample so results stay reproducible.

    4.2 Metrics

    Dimension Metric Definition
    Quality First-pass rate Share of calls that meet acceptance criteria on the first attempt
    Quality Rework rate Share that needs a second correction or human intervention
    Quality Tool-calling accuracy Joint correctness of tool selection and argument filling
    Quality Structured output validity Share of outputs a program can parse and that pass validation
    Cost Cost per task Including all turns and vision conversion
    Cost Total batch budget Number of tasks × cost per task, with some headroom reserved
    Efficiency P95 latency End-to-end latency for 95% of requests
    Stability Consistency across runs Variance in results for the same input across repeated runs

    4.3 Evaluation script skeleton

    1. import json, time
    2. def evaluate(model, spec, samples, price_in, price_out):
    3. passed = 0
    4. total_cost = 0.0
    5. latencies = []
    6. for s in samples:
    7. t0 = time.time()
    8. raw = model.chat(
    9. messages=s.messages,
    10. tools=s.tools,
    11. max_output_tokens=min(spec.default_output, s.expected_output * 2),
    12. temperature=0, # fix randomness so regression comparisons hold
    13. )
    14. latencies.append(time.time() - t0)
    15. # Programmatic validation: parsing and assertions first, then human or judge-model review
    16. ok = s.validate(raw.text) if s.has_validator else s.manual_review(raw.text)
    17. passed += int(ok)
    18. total_cost += (raw.usage.prompt_tokens * price_in
    19. + raw.usage.completion_tokens * price_out) / 1_000_000
    20. latencies.sort()
    21. return {
    22. "pass_rate": round(passed / len(samples), 4),
    23. "avg_cost": round(total_cost / len(samples), 6),
    24. "p95_latency": round(latencies[int(len(latencies) * 0.95)], 3),
    25. }
    26. # Illustrative call: run every candidate model on the same sample set
    27. # report = evaluate(model_a, spec_a, samples, price_in_a, price_out_a)

    4.4 Acceptance and governance

    • Set thresholds: before launch, agree on minimum first-pass rate, cost per task, and P95 latency. Any candidate that misses one does not enter the canary.
    • Canary rollout: ramp up gradually by traffic share rather than switching everything at once.
    • Continuous monitoring: track first-pass rate, rework rate, cost per task, P95 latency, and error-code distribution through monitoring and alerting, with automatic rollback on anomalies.
    • Logging: write anonymized call logs to a logging service, keeping the request ID and model version for auditing and regression.
    • Offline evaluation: offline evaluation jobs can sit in a queue and be consumed asynchronously against a batch API.

    If you want to avoid the repeated cost of integrating with and evaluating each vendor separately, an enterprise LLM platform such as Baidu AI Cloud lets you compare candidate models through one access and evaluation pipeline and then deploy by business tier. Availability of such platforms varies by region, so check what is available where you operate. Either way, the selection criteria should rest on business evaluation results, not on the length of a spec sheet or a low price.

    5. A five-step selection path

    1. Filter by input modality first: for routine text + image workloads, keep candidates with native text and image support. If video is involved, keep candidates with native video support. Where assets need transcoding, frame extraction, or image splitting, evaluate the preprocessing cost first.
    2. Establish a cost baseline with a lightweight model: for structured tasks such as high-volume image and text understanding, information extraction, and format classification, run the cheaper candidates end to end and measure the first-pass rate.
    3. Add candidates based on output needs: for long code, long reports, and bulk JSON, focus on candidates with a generous output budget and a larger default output, and verify how continuation after truncation works.
    4. Evaluate video tasks separately: compare native video understanding against “extract frames + understand frame by frame” on both accuracy and cost.
    5. Race complex agents head to head: run all candidates on the same agent task set (multiple tools, long chains, failure recovery), then decide using first-pass rate, rework rate, cost per task, and P95 latency. If needed, apply tiered routing so simple tasks go to the baseline model and complex ones to the flagship.

    6. Common pitfalls

    • Treating context as output: context is a shared space for input and generation, not an extra allowance on top of output.
    • Treating specs as a ranking: parameters only decide what is possible; quality has to be validated with a business task set.
    • Looking at input price only: ignoring output price, cache hit pricing, vision conversion, and retry amplification can underestimate total cost by several times.
    • Ignoring unit differences: 1M vs. 1,048,576, and 96k vs. 96,000 — near the limit, this decides success or failure.
    • Treating a version name as an API identifier: when calls fail or results drift, check the API identifier in the vendor’s documentation first.
    • Overlooking silent model updates: pin versions or build a regression baseline so you avoid “it passed yesterday, it fails today.”
    • Switching everything at once: canary first, ramp later, and keep rollback available.

    7. Conclusion

    Selecting a multimodal LLM comes down to one principle: align the units first, measure cost second, and let business tasks decide the winner. Spec sheets answer whether a task is possible. The cost model answers whether it pays off, and its key variables are first-pass rate and number of calls per task. The evaluation system decides who fits best, and it has to cover first-pass rate, rework rate, cost per task, and P95 latency.

    “Max” or “Pro” in a name, and “million-token” in a context line, are not reasons to choose. Put those numbers through the same business tasks and the same units of measurement, and the answer surfaces on its own.