Multi-Model Collaboration: Engineering Practices for Combining LLMs by Task Capability
Relying on a single LLM often means compromising on cost, speed, or quality. This article details engineering practices for combining large language models based on specific task capabilities. It covers evaluating model dimensions, building custom evaluation datasets, designing routing and combination strategies, ensuring completeness in large-scale document processing, and provides a practical deployment checklist.
1. Why a Single Model Cannot Cover All Scenarios
Large language models (LLMs) vary in reasoning depth, long-context handling, code generation, multimodal understanding, response latency, and call cost. A model excelling in one task may be unstable in another. If a team expects one model to handle all requirements, they will typically sacrifice at least one of cost, speed, or quality.
A more feasible approach is to break down tasks first, then match the appropriate capabilities for each sub-task. For instance, simple text classification and summarization can be assigned to fast, low-cost models; mathematical or logical problems requiring multi-step deduction can go to models with stronger reasoning; and Q&A over ultra-long documents requires models with long-context support combined with retrieval. The core of this approach is not finding an “all-around model” but establishing a selection mechanism that can be continuously validated.
2. Breaking Down Tasks into Evaluable Capability Dimensions
Before selecting a model, the concept of “good” must be quantified. Common evaluation dimensions include:
- Reasoning and planning capability: Accuracy of multi-step problems and self-consistency of intermediate steps.
- Long-context understanding: Accuracy of locating key information in long materials and the ability to identify contextual contradictions.
- Structured output: Compliance rate of returning results in a specified format (e.g., JSON) and the ratio of missing fields or type errors.
- Code generation and debugging: Percentage of generated code passing unit tests and success rate of fixing known bugs.
- Multimodal input: Recognition accuracy for non-text inputs like images, tables, and audio.
- Latency: Time to first token and P95 total time, which determine user experience and concurrency limits.
- Cost per call: Actual billed cost per 1,000 tokens or per call, rather than just the list price.
- Data compliance: Whether data leaves the intranet and meets retention and desensitization requirements.
Each dimension should correspond to a quantifiable metric. If a dimension cannot be quantified, it usually means the task definition is not clear enough, and you need to return to the business scenario to redefine it.
3. Building a Custom Evaluation Dataset
Public leaderboards only reflect general capabilities and cannot directly represent business performance. Teams need to build their own evaluation datasets. The process typically includes the following steps:
- Collect real samples: Extract inputs from online logs or historical tickets and desensitize them.
- Annotate golden answers: Domain experts provide standard answers or judging rules. For open-ended tasks, judging criteria should be specified, such as “must contain three key points and no factual errors.”
- Stratified sampling: Stratify by difficulty, length, and business category to avoid sample concentration in simple scenarios.
- Run multiple times for statistical values: LLM outputs are stochastic; the same input may yield different results across runs. Record the mean and variance rather than just a single result.
- Record versions and parameters: Save the model version, temperature, system prompt, and evaluation date to ensure reproducibility.
Below is a vendor-agnostic evaluation framework example. It accepts any callable model function and outputs the mean and variance of the pass rate:
from dataclasses import dataclassfrom statistics import meanfrom typing import Callable@dataclassclass EvalCase:prompt: strchecker: Callable[[str], bool] # Checker function defined by the business sidedef evaluate(model_call: Callable[[str], str],cases: list[EvalCase],runs: int = 3) -> dict:pass_rates = []for _ in range(runs):passed = sum(case.checker(model_call(case.prompt)) for case in cases)pass_rates.append(passed / len(cases))return {"mean_pass_rate": mean(pass_rates),"spread": max(pass_rates) - min(pass_rates),}
The checker function should ideally use deterministic rules, such as verifying if JSON can be parsed, if key fields exist, or if code passes tests. For tasks requiring semantic judgment, a review model or manual sampling can be introduced, but the reliability of the review results must be calibrated regularly.
4. Routing and Combination: Dispatching by Task
After determining the capability boundaries of each model, the next step is to design a dispatch strategy. There are four common combination patterns:
1. Rule-based routing: Directly maps to the target model based on task type or input length. The rules are simple and predictable, suitable for systems with clear task types.
ROUTES = {"summarize": "fast_model","reason": "deep_model","code": "code_model",}def route(task_type: str, text_length: int) -> str:if text_length > 100_000:return "long_context_model"return ROUTES.get(task_type, "fast_model")
2. Cascading calls: First try a low-cost model. If the output fails validation or self-evaluated confidence is low, escalate to a high-capability model. This can significantly reduce average costs, but worst-case latency will increase. A timeout should be set for the escalation chain.
3. Pipeline collaboration: A planning module breaks down the task, an execution module completes it step by step, and finally, a review module checks the results. Pipelines make it easy to locate error points, but each added node increases latency and cost, potentially amplifying upstream errors.
4. Parallel voting: The same question is sent to multiple models, and the output is decided by a majority vote or review results. This is suitable for high-risk decision scenarios, at the cost of multiplied expenses.
Regardless of the pattern used, supporting degradation and fault tolerance mechanisms are required:
- Set a timeout for each call; switch to a fallback model upon timeout.
- Use exponential backoff for transient errors, with a maximum retry limit.
- Validate output format and content; do not return directly to the user upon validation failure.
- Enable circuit breaking when a model fails consecutively to prevent fault propagation.
5. Large-Scale Material Processing: Completeness and Traceability
In scenarios like scientific research, legal, or auditing, a common requirement is generating reports based on a large number of reference files. The most common issue here is omission: the model may only reference a few files while the rest remain unprocessed.
Prompting the model to “read all materials thoroughly” does not guarantee completeness. A reliable approach is to guarantee completeness through engineering workflows rather than relying on the model’s consciousness. The recommended process is as follows:
- Build an inventory: Traverse all input files, recording path, size, and hash value to form a verifiable inventory.
- Chunk extraction: Split files into semantically complete chunks, saving the source file and location info for each chunk.
- Per-file summarization (Map): Independently generate structured summaries for each file or chunk, annotating whether it is relevant to the current task.
- Aggregation generation (Reduce): Summarize into a final document, requiring every assertion to include source citations.
- Coverage report: List used, excluded, and unprocessed files, and state the reason for exclusions.
The following example demonstrates the basic structure of inventory building and coverage statistics. In practice, large files should be read in chunks for hashing to control memory usage:
import hashlibfrom pathlib import Pathdef build_inventory(root: str) -> dict[str, dict]:inventory = {}for path in Path(root).rglob("*"):if path.is_file():digest = hashlib.sha256(path.read_bytes()).hexdigest()inventory[str(path)] = {"sha256": digest, "used": False, "reason": ""}return inventorydef coverage_report(inventory: dict[str, dict]) -> str:total = len(inventory)used = sum(1 for item in inventory.values() if item["used"])unused = [path for path, item in inventory.items() if not item["used"]]lines = [f"Used {used}/{total} files."]lines += [f"Unused: {path}" for path in unused]return "\n".join(lines)
During the aggregation phase, a self-verification step can be added. The model can be prompted to ask follow-up questions about the draft, such as “Which part of the material is this assertion based on?” or “Is there any contradictory material?” Every follow-up question must be answered using the original source; assertions that cannot be answered should be flagged or removed. This method improves the verifiability of conclusions without relying on the model’s consciousness.
6. Deployment Checklist
- Pin model versions and record parameters and prompts used in each evaluation.
- Integrate the evaluation dataset into CI/CD for automated regression after model or prompt changes.
- Track call volume, latency, and cost by model and business line; set budget alerts.
- Verify desensitization and retention requirements for input data, ensuring data flow complies with regulations.
- Set up manual review nodes for high-risk outputs and retain complete call logs for traceability.
- Regularly update the evaluation dataset by adding new online failure cases to keep evaluations close to real business scenarios.
