Multi-Model Benchmark: Why the Same Task Consumes Several Times More Quota

    This article reports a side-by-side benchmark of multiple large language models running the same tasks, and explains why identical workloads can consume several times more quota depending on model choice. It walks through a reproducible four-step evaluation method, a scoring rubric, observed results across three task types—long-document structured extraction, runnable code generation, and web-search writing—and how auto-routing behaves in practice. The main takeaway is that cost reduction comes primarily from engineering constraints rather than swapping in a "smarter" model: context trimming, task-tier routing, prompt-template caching, output-structure constraints, retry budgets, and off-peak scheduling can bring quota consumption down to a fraction of the original.

    1. Background: From Single-Model to Multi-Model

    Over the past two years, the way assistants and LLM APIs were consumed was essentially one-to-one: a product was bound to a single model, users had no choice, and therefore no decision to make. Once the platform matured, that changed. Under a single entry point, models from multiple vendors now sit side by side—from small, fast in-house models to large, high-parameter third-party reasoning models. In the dropdown, users see roughly ten options, including automatic routing.

    The real shift this introduces is not “more choices” but a change in the billing model:

    1. A unified quota pool. Models are no longer billed independently. Calls to different models are converted into a single “billing unit,” with the pool shared and the balance pooled together.
    2. Multiplier coefficients. Each model carries a multiplier tag; the higher the multiplier, the more quota is deducted for the same amount of work. A multiplier is essentially the “relative cost per token” and is the first yardstick for comparing value across models.
    3. Time-based discounts and free micro-tasks. Specific time windows on workdays and holidays discount quota consumption, while extremely short, low-complexity requests are not billed at all. Cost is therefore a function not only of model attributes but also of time.
    4. Automatic routing. The system assigns models based on task complexity, in principle making “cheap models handle simple work, expensive models handle hard work” the default behavior.

    This raises a question that must be answered: if models are interchangeable, how do you use them most economically? Switching by gut feeling will not be economical, because intuition is often mismatched—users tend to hand every “important task” to the most expensive model, whereas testing frequently shows that these tasks can be delivered perfectly by mid- and low-tier models as well.

    2. Evaluation Method: Turning “Which Model Is Better” into a Reproducible Experiment

    To compare cost, you first have to turn “quality” into comparable numbers. We use a four-step approach that anyone can replicate in their own setting.

    2.1 Task Set Design: Covering Three Cost Structures

    The three task types differ not in difficulty but in the composition of token consumption, which determines how sensitive cost is to model choice:

    Task type Input tokens Output tokens Tool calls Cost driver
    Long-document structured extraction Very high (whole document in context) Medium None Input length
    Runnable code generation Medium High None Output length and retry count
    Web search + writing Medium Medium Multiple rounds Retrieval-round amplification

    Testing only one task type yields a one-sided conclusion. Testing only long documents overstates the benefit of “switching models”; testing only code mistakes retry count for a model-capability problem.

    2.2 Controlling Variables

    • The same input material (for example, a simulated management report on the order of 20 pages), the same prompt, with no targeted fine-tuning;
    • Identical sampling parameters (temperature and maximum output length set uniformly) to avoid randomness polluting the comparison;
    • All calls compressed into the same time window, avoiding discount periods, so that “cheapness” comes from the model rather than the discount;
    • Three models plus one automatic-routing run per question, for twelve calls across the three task types, forming a complete control matrix.

    2.3 Scoring Rubric: Breaking Subjective Feel into Checkable Items

    Take the long-document task as an example. The scoring items are fixed at six, each judged independently to avoid an overall impression score:

    1. {
    2. "rubric": [
    3. {"id": "R1", "desc": "Summary length does not exceed the set cap", "weight": 1},
    4. {"id": "R2", "desc": "All five questions answered with no omissions", "weight": 2},
    5. {"id": "R3", "desc": "Key metric values match the source exactly", "weight": 2},
    6. {"id": "R4", "desc": "YoY change calculations correct and direction labeled", "weight": 2},
    7. {"id": "R5", "desc": "Each figure tagged with source page number, traceable", "weight": 2},
    8. {"id": "R6", "desc": "Output is an openable table file with complete columns","weight": 1}
    9. ]
    10. }

    Scoring must be automatically or semi-automatically verifiable. Numerical consistency can be compared by script, word count can be measured programmatically, and page-number traceability can be spot-checked; only summary quality requires human judgment. The more automated the scoring, the more repeatable the experiment and the more it lends itself to a long-term dashboard.

    3. Observed Results Across Three Task Types

    3.1 Long-Document Structured Extraction: Converging Capability, Diverging Bills

    This question required two things: first, a summary of no more than 800 words covering business performance, changes in revenue and profit and their causes, profit quality (the impact of non-recurring gains and losses and accounting treatment), cash-flow and accounts-receivable risk, and any material events after the reporting period; second, a key-metrics table containing operating revenue, net profit attributable to parent, net profit attributable to parent excluding non-recurring items, net cash flow from operating activities, R&D investment and its share, and accounts receivable, with the current-period value, prior-period value, YoY change, and source page number for each, output as a table file.

    The result was clear: all four solutions scored full marks. Every summary stayed within the word cap, the metric values matched the source, the YoY directions and percentages were correct, page numbers were traceable, and the table column structure was complete.

    This shows that for tasks like “long document + structured output,” mainstream models’ capabilities have fully converged; there is no case where “you must use the top tier to get it right.” Yet the bills were completely different: with the same input and the same output requirements, quota consumption between the highest and lowest differed by several times—close to 6 to 7x in this round of observation.

    The engineering conclusion is direct: for this kind of task, output quality is weakly correlated with model tier and strongly correlated with input length. So the right cost-cutting action is not “pick a smarter but more expensive model” but to compress the input—feed only the sections relevant to the metrics into the context, strip out headers, footers, tables of contents, and boilerplate notes, which usually cuts input tokens by more than half and brings cost down almost linearly. Constraining the output to a fixed JSON structure also helps, both reducing useless explanatory text and making it easy to write directly to a database.

    3.2 Runnable Code Generation: Quality First, Cost Multiplied by Retries

    The second question asked for a reimbursement tool that finance can use directly: form entry, amount and date validation, over-limit alerts, itemized summary, and table-file export, runnable on a single page.

    The cost structure of a code task is the opposite of a document task: output tokens dominate, and retries multiply cost. If a model’s first generated version has a minor flaw, rerunning it doubles consumption. So the key metric for this question is not the per-call price but the first-pass success rate. When evaluating, you should record: whether the first generation runs directly, whether boundary validation is complete, whether amounts are computed with fixed-point or smallest currency units (floating-point accumulation is a fatal flaw in reimbursement scenarios), whether there are accessibility defects (form labels, keyboard reachability), and code readability.

    The practical recommendation is to use a high-quality tier for code generation, but to add constraints at the same time—make explicit “no explanations, output only the complete file content,” specify the input/output contract, and prohibit using floating point for money calculations. The harder the prompt constraints, the lower the retry probability, and the expensive model’s actual total cost may end up lower than a cheap model run three times.

    3.3 Web Search + Writing: Cost Amplified by Retrieval Rounds

    The third question required searching the web and writing an industry brief for management. The bulk of the cost here is not generation but the number of tool-call rounds: each round’s retrieved content enters the next round’s context, and three rounds of retrieval can grow input tokens to five or six times the initial amount.

    The evaluation therefore focuses on three points: whether citations are verifiable (able to point to a specific source rather than a vague statement), timeliness (whether recent material is prioritized), and whether the structure is ready to present (conclusions up front, with comparisons and risk flags). The cost-cutting measures are equally clear: set an upper limit on the number of retrievals, summarize retrieval results before feeding them into context, and avoid “pushing full text into history as you search.”

    3.4 How Auto-Routing Actually Performs

    Auto mode is positioned as “worry-free.” In testing, it performed close to the optimal choice on two task types, but on the extraction task that requires stable structured output, it occasionally selected a heavier tier, with cost higher than manually choosing a low-multiplier model. Auto-routing suits the exploration phase and everyday light tasks; once a task type becomes a high-frequency fixed pipeline, it should be locked into an explicit model-selection rule.

    4. The Core of Cost Reduction Isn’t Switching Models, It’s Engineering Constraints

    Breaking down the bills from the twelve calls, the main sources of quota savings rank as follows:

    1. Context trimming. Biggest payoff on long-document tasks, usually contributing more than half of the savings.
    2. Task-tier routing. Fixing the model tier by task type rather than always picking the most expensive one.
    3. Prompt-template reuse and caching. The same prefixes (system prompt, templates, wording conventions) should be designed as stable, unchanging segments so they hit the prefix cache and reduce repeated billing.
    4. Output-structure constraints. Capping length and format to reduce useless generated tokens.
    5. Retry budgets. Setting a retry cap per task type and degrading or handing off to a human once exceeded, to prevent runaway consumption.
    6. Time-based scheduling. Moving batch, offline, non-real-time tasks (running report extraction overnight, archiving historical documents on weekends) into discount windows—a zero-cost optimization.

    A deployable routing-policy sketch:

    1. POLICY = [
    2. # Task type, Trigger condition, Model tier, Output budget, Retry limit
    3. ("Extraction/Summary", "input > 5k tokens", "low-cost tier", 1500, 1)
    4. ("Extraction/Summary", "input <= 5k tokens", "low-cost tier", 1200, 1)
    5. ("Code generation", "must produce runnable file", "high-quality tier", 5000, 2)
    6. ("Search & writing", "needs external real-time info", "balanced tier", 2500, 1)
    7. ("Chat/rewrite", "output < 500 words", "low-cost tier", 800, 1)
    8. ]

    For the accompanying call instrumentation, it is worth logging: task_type, model_tier, input_tokens, output_tokens, tool_rounds, retry_count, rubric_score, quota_cost, timestamp. With this data, model selection is no longer a monthly guess but a weekly look at a “cost per point of score” (quota consumption / score) leaderboard.

    5. Implementation Recommendations

    • Build a baseline before optimizing. Without a scoring rubric and token instrumentation, all cost reduction is guesswork.
    • Prioritize tasks by cost structure. For high-input tasks, do context trimming first; for code tasks, harden prompts first; for retrieval tasks, cap rounds first.
    • Write conclusions into configuration, not memory. Model tiers, output budgets, and retry caps should live in config or policy code—versionable and canary-releasable.
    • Watch out for the “capability premium” trap. Evaluation shows that on most standardized tasks, the output-quality gap between tiers is far smaller than the price gap; reserve the most expensive model for tasks that genuinely need deep reasoning and long-chain generation.
    • Keep a human-review step. Numerical outputs must be spot-checked, and page-number traceability is the most effective signal for whether the model “actually read the source.”

    In one sentence, the value of this benchmark is this: models are interchangeable, but cost does not fall automatically. Classify the tasks, thin out the input, lock down the output, and cap the retries—bringing quota consumption down to twenty or thirty percent of the original for the same amount of work is entirely achievable, and far more reliable than repeatedly probing the model dropdown.