Multi-Model Integration and Routing: Model Selection and Cost Control for LLM Applications
This article explains how teams move from a single-model setup to a multi-model architecture, where model selection becomes a multi-objective optimization problem that balances quality and cost. It examines how model capabilities differ across three task types—long-document extraction, code generation, and web-based research and writing—and how costs are calculated under a unified credit pool. It then covers building reproducible, verifiable evaluations and implementing task-based routing and budget control through rule-based, complexity-based, and cascade strategies. A cost-control checklist and a reference engineering architecture round out the guide, helping teams reserve strong models only where they are truly needed and shift cost control from month-end billing to real-time decisions.
1. From a Single Model to Multiple Models: How the Selection Problem Changes
In the past, most teams approached foundation-model usage with a “decide once” mindset: pick the single most capable model overall and route every task to it. This approach rests on an implicit assumption—that a more capable model is always the better deal in every scenario.
Once a unified platform opens up multiple third-party models to end users and converts each model’s usage into a common pool of credits, that assumption no longer holds, for three reasons:
- Capabilities vary by strength. Different models differ in long-document understanding, code generation, multi-step reasoning, and tool calling; an “all-around champion” rarely exists.
- Pricing differs. Platforms usually assign different billing multipliers to different models. A lower multiplier deducts fewer credits for the same call.
- Versions change dynamically. Models iterate frequently, and after a version upgrade both capability and token consumption can shift, so today’s selection decision may not hold over time.
As a result, “which model to use” is really a multi-objective optimization problem: above a quality threshold, find the combination of models and call paths with the lowest cost per task.
One platform once publicly distinguished between two user demands—“choosing the model” and “choosing the price”—which sparked industry discussion about model routing. Starting from engineering practice, this article addresses three questions:
- How do you design a reproducible comparison evaluation instead of selecting models based on subjective impressions?
- Under a unified credit pool, what factors determine cost, and how do you account for it?
- How do you implement task-based routing and budget-controlled scheduling in a system?
2. Capability Differences: The Model Watershed for Three Task Types
Different task types place very different demands on model capabilities. Three typical scenarios illustrate this.
2.1 Long-Document Structured Extraction: Long Context and Instruction Following
Turning a several-hundred-page public annual report into a clean, structured table may look like just “read the file and pull out information,” but it actually involves several steps:
- Information retrieval within long context. Key figures are scattered across chapters and pages, and you need to align definitions across paragraphs.
- Table-structure understanding. Continued tables across pages, merged cells, and footnotes all interfere with assigning fields correctly.
- Consistency of numbers and units. Currency units, currencies, and statistical bases can switch at any point within the same document.
- Instruction following and output format. Field definitions, field order, and output format must strictly meet downstream requirements.
Beyond fields that can be validated automatically—such as whether an extracted amount matches the source text—judging whether account items are categorized appropriately and whether the interpretation of statistical bases is reasonable often requires domain knowledge. This task type demands long input length and stable output formatting, so it favors models with a large context window and stable instruction following.
2.2 Code Generation: Runnability and Engineering Constraints
Take generating an expense-reimbursement tool as an example. The requirements usually span:
- Form entry and input validation;
- Summing receipt amounts and enforcing expense caps;
- Multi-level approval and status transitions;
- Interfaces to connect with a finance system;
- Audit trails and permission management.
This task type tests four capabilities:
- Code correctness. Does it run on the first try, and are edge cases handled properly?
- Architectural soundness. Does it follow the project’s existing tech stack and directory conventions?
- Security. Are there problems such as concatenated SQL or hardcoded sensitive information?
- Maintainability. Are naming, comments, and error handling up to standard?
If the task also requires multi-file changes and multi-step planning, the weight on the model’s reasoning capability rises noticeably.
2.3 Web Retrieval and Synthesis Writing: Retrieval and Information Reliability
Writing an industry briefing for management is difficult because of:
- Timeliness. Key facts such as market size, policy, and pricing need citations to traceable, up-to-date information.
- Coverage. You need to address multiple dimensions rather than just answering the most obvious question.
- Source quality. The credibility of authoritative institutions, public reporting, and self-media content varies widely.
- Hallucination risk. General-purpose large models may fabricate data or citations that look plausible but do not exist.
This task type favors models or tool combinations with web retrieval capability and traceable citations. In addition, decisions such as “whether model-generated code should go into production systems” or “whether business data is suitable to hand to an external model” are compliance and security judgments and should not be left to the model to decide on its own.
3. Cost Structure Under a Unified Credit Pool
3.1 Cost Model
A unified credit pool can be viewed as a shared account: different models are like meters with different rates, and after converting actual usage they are deducted from the same account. The credit consumption of a single task can be roughly expressed as:
Per-task credit consumption ≈ model billing multiplier × Σ(input + output tokens per call)
There are two additional adjustment factors:
- Capability coefficient. Capabilities such as web retrieval, deep reasoning, and tool calling may add extra consumption.
- Time-of-day coefficient. Some platforms offer credit discounts during specific hours on workdays or holidays, so shifting calls off-peak can directly reduce cost.
The simple intuition of “charging by number of calls” masks three key facts:
- The multiplier is a coefficient, not a fixed price. Under the same multiplier, different request lengths can widen actual consumption several times over.
- Multiple retries and repeated output revisions are all billed repeatedly.
- Additional capabilities may cost more than basic conversation.
3.2 An Illustrative Calculation
Suppose a platform publishes the following multipliers, used only to demonstrate the accounting method:
- Lightweight model: 1.0;
- Standard model: 4.0;
- High-reasoning model: 12.0.
Also suppose that, to complete the same task, the models differ in input, output, and number of retries:
| Model tier | Input/output per call (illustrative) | Retries | Credit consumption (illustrative) |
|---|---|---|---|
| Lightweight | 1500 / 400 | 1 | 1900 |
| Standard | 1500 / 400 | 1 | 7600 |
| High-reasoning | 1500 / 2500 (incl. reasoning) | 2 | 32000 |
In this illustrative example, consumption for the same task differs by about 16.8×. About 12× comes from the multiplier difference, and the rest comes from output length and retries. This also explains why “the same work, but credits differ several-fold” has become a common phenomenon. When selecting models, plug measured token usage and call counts into the formula rather than looking only at the multiplier.
3.3 Four Levers for Cost Optimization
Ordered by impact, you can adjust four levers:
- Lower the multiplier. Among candidate models that meet the quality bar, prefer the one with a lower multiplier.
- Shorten the length. Trim input context, cap output length, and avoid letting the model expand without constraints.
- Reduce the count. Cut trial-and-error retries, and replace “let the model think again” with deterministic validation.
- Choose the time window. Schedule batch, non-real-time tasks during discount periods.
4. Evaluation Methods: Making Selection Data-Driven
4.1 Evaluation Design Principles
Comparison evaluations easily fall into the trap of “going by feel,” so follow four principles:
- Reproducible. Fix the model version (to avoid behavioral drift from rolling upgrades), the prompts, the input files, and generation parameters (such as temperature and max output length), and log the complete requests and responses.
- Verifiable. Prefer evaluation sets with ground-truth answers from public materials, such as listed-company annual reports, public policy documents, and runnable test cases.
- Multi-task coverage. A single task easily leads to one-sided conclusions; cover different capability dimensions such as documents, code, and retrieval.
- Weigh cost and quality together. Record quality scores and credit consumption in parallel to form a two-dimensional “quality–cost” view, rather than ranking by quality alone.
4.2 Structured Output and Automated Validation
Take a document-extraction task as an example: you can define a structured output format for the model and then validate it automatically with a script:
import csvimport jsonREQUIRED_FIELDS = ["period", "item", "current_amount", "prior_amount", "yoy_change"]def evaluate_extraction(raw_output: str, reference: list[dict]) -> dict:# 1. Parse the structured output and check that all required fields are presenttry:rows = json.loads(raw_output)rows = [r for r in rows if all(f in r for f in REQUIRED_FIELDS)]except json.JSONDecodeError:return {"format_ok": False, "field_match": 0.0, "value_match": 0.0, "extra_rows": 0}# 2. Compare against the ground truth field by field: field presence and value equalityref_map = {(r["period"], r["item"]): r for r in reference}hit = value_hit = 0for r in rows:key = (r["period"], r["item"])if key in ref_map:hit += 1for f in ["current_amount", "prior_amount", "yoy_change"]:if abs(float(r[f]) - float(ref_map[key][f])) < 0.01:value_hit += 1# 3. Hallucination check: the more rows that don't exist in the ground truth, the more fabricationreturn {"format_ok": True,"field_match": hit / len(reference),"value_match": value_hit / (len(reference) * 3),"extra_rows": len(rows) - hit,}
For code-generation tasks, you can use “compile + unit tests” for automated scoring: first compile and run, then execute the preset test cases, and the pass rate becomes the quality score. For retrieval-and-writing tasks, you can build a “fact–source” list and manually or semi-automatically verify whether each key fact’s source is real and whether it matches the original text.
4.3 The Evaluation Matrix
You can ultimately organize the results into an evaluation matrix like the one below. The model tiers and multipliers in the table are illustrative; substitute the actual published data and measured results:
| Task type | Recommended model tier (illustrative) | Enable web/deep reasoning | Main criteria | Multiplier (illustrative) |
|---|---|---|---|---|
| Long-document extraction | Long context + stable output format | No | Field completeness, value accuracy | 1.0–4.0 |
| Code generation | Stronger reasoning | Depends on task complexity | Compilation pass rate, unit test pass rate | 4.0–12.0 |
| Retrieval and briefing | Web-retrieval capable | Web enabled | Fact accuracy, source traceability | 4.0–12.0 |
| Intent classification, format conversion | Lightweight | No | Rule checks or a small test set | 0–1.0 |
Another common case is the “auto” or intelligent routing mode. It usually selects a model automatically based on request features and can serve as a default baseline, but it must be included in the same evaluation process. Only when auto mode meets the quality bar is it worth handing scheduling over to the platform; otherwise, the team needs to build its own routing strategy at the application layer.
It’s worth stressing that the validity of evaluation conclusions is bounded by the task, the prompts, and the model version. It’s recommended to:
- Keep the internal evaluation set under version control, and rerun a regression round after a model upgrade;
- Refresh cost data whenever a model is updated;
- Note the execution date and version information in evaluation results.
5. Routing Strategies: From a Single Entry Point to Layered Routing
5.1 Three Routing Strategies
- Rule-based routing. Static mapping by task type—simple and reliable. For example: route classification, rewriting, and format conversion to lightweight models, and route code generation and complex reasoning to more capable models.
- Semantic/complexity routing. First use a lightweight model or classifier to judge request difficulty, then decide which model to use. This is more flexible but adds an extra upfront call.
- Cascade scheduling. First handle the task with a low-cost approach and use a validator to judge whether it meets the bar; if not, escalate to a stronger model. This is a key technique for balancing cost and quality:
# Illustrative pseudocode: cascade calls + validation + budget controldef run_task(task: Task, check: Validator, gateway: Gateway, budget: Budget) -> Result:# Ladder: start with low cost, then higher capabilitytiers = [("light-model", False, False), # (model, use web, use deep reasoning)("mid-model", False, False),("reasoning-model", task.need_web, True),]for model, use_web, deep in tiers:if budget.remaining() < budget.reserve_for_fallback():raise BudgetExceeded(task) # If the budget is insufficient, stop escalating to a higher-cost tierresult = gateway.call(model=model,messages=build_messages(task),use_web=use_web,deep=deep,max_output_tokens=task.max_output,)if check(task, result):return result # Return as soon as it passes; no further escalation# If it fails, move to the next tier; failure details can be written into the next prompt to help the model correct# ...return escalate_to_human(task) # If no tier meets the bar, escalate to human handling
Validators select different rules by task type:
- Document extraction: field completeness, value ranges, unit consistency.
- Code generation: compilation results, unit tests, static analysis, and security-rule scans.
- Retrieval and writing: citation accessibility, completeness of the fact list, and traceability of key numbers.
5.2 Budget and Cost Control
Cascade scheduling must work together with budget control; otherwise, “retry–escalate” can amplify cost instead. Specific measures include:
- Set caps on the number of calls and on credit consumption for each task type, so escalations cannot stack without limit.
- Allocate quotas by team, project, or interface, and set usage-threshold alerts.
- Log the full call chain—task ID, model version, input and output length, call count, credit consumption—to provide a basis for later tuning.
6. Cost Control Checklist
- Layer by task. Route simple tasks to lightweight models and complex tasks to models with stronger reasoning, avoiding “using a sledgehammer to crack a nut.”
- Trim context. Deduplicate retrieval results and truncate by relevance; for long documents, chunk and summarize first, then extract, to cut wasted tokens.
- Cap output length. Specify the output structure and number of fields to avoid free-form expansion.
- Enable result caching. Use the hash of “input content + prompt + model version” as the cache key, set a reasonable expiry, and serve repeated requests directly from cache.
- Idempotency and deduplication. Merge identical requests within a short time window to avoid duplicate billing.
- Off-peak batch processing. Schedule batch tasks during discount periods, and prioritize response speed for real-time tasks.
- Prompt slimming and version management. Long, repetitive prompts increase input tokens per call; trim them regularly and validate changes with A/B tests.
- Build a regression mechanism. Re-evaluate after a model upgrade to prevent “quality declining without anyone noticing” or “playing it safe by always using the strongest model.”
7. Engineering Architecture
From a systems-engineering perspective, a multi-model application can be split into several layers:
- Model gateway layer. Uniformly wrap the call protocols of different models, hide the API differences between vendors, and provide timeout, retry, rate limiting, and fallback capabilities.
- Routing and scheduling layer. Implement rule-based routing, complexity judgment, and cascade escalation strategies.
- Validation layer. Perform deterministic validation of outputs to decide whether to escalate or hand off to a human.
- Storage layer. Persist documents, vector indexes, cached results, and evaluation data.
- Observability layer. Monitor call volume, credit consumption, success rate, average latency, and cost per task.
Under this architecture, the basic capabilities typically offered by cloud platforms such as Baidu AI Cloud can be combined as needed to support stable operation:
- Model invocation. Access multiple models through the cloud vendor’s large-model service platform.
- Document and cache storage. Use object storage to hold original files and intermediate results.
- Async tasks. Use a message queue to carry batch evaluations, offline report generation, and other tasks, making off-peak execution easier.
- Retrieval. Use a vector database to manage semantic indexes.
- Runtime. Deploy the gateway and scheduling services on a container platform for elastic scaling.
- Operational support. Use log services, monitoring alerts, and billing management for usage analysis and quota control.
When enterprises put multi-model applications into production, they should also attend to three requirements:
- Data classification. Content involving personal information, financial data, or trade secrets should be desensitized before any call.
- Audit trails. Record call records and results to meet compliance and traceability needs.
- Access control. Distinguish which model tiers different roles can use.
8. Conclusion
The growing number of foundation models has not made selection easier; it has turned it from “picking one” into “managing a batch.” The real value of multi-model integration lies not in giving users more options, but in enabling teams to:
- Quantify the quality and cost of each task with reproducible evaluations;
- Reserve strong models for the steps that truly need them through cascading and routing;
- Move cost control forward from “checking the bill at month-end” to “deciding at call time” with budgets and monitoring.
As in common software-engineering practice, assigning the right task to the right tool is often more efficient than pursuing a single “all-in-one tool.” Model selection should be the same. Above a quality threshold, completing tasks at the lowest cost is what “model freedom” truly means in the multi-model era.
