Choosing LLMs by Scenario: Building Model Combinations and Full-Coverage Processing for Long Documents
Many teams default to picking a single "best" large language model, but in practice no single model excels at everything. This article explains how to break workloads into stages, evaluate candidate models against a reproducible framework, combine multiple models through routing, pipeline, or cross-validation architectures, and process large document collections with full coverage using ledgers, chunking, and staged summarization. Includes practical Python code for evaluation and document processing workflows.
From One “Best” Model to Scenario-Based Model Combinations
When selecting a large language model, technical teams often look for one that leads on every benchmark. In real-world work, that assumption rarely holds. Different models have different strengths in reasoning depth, long-context handling, code generation, instruction formatting, and cost per response—no single model fits every scenario.
A more pragmatic approach is to break the task down first. Take a research writing task as an example; it typically includes these stages:
- High-level planning: defining the argument structure, deciding which sections to keep, and allocating length
- Source material review: reading through large volumes of reference files and extracting information relevant to the question
- Detailed drafting: generating section-by-section content while keeping terminology and notation consistent
- Review and validation: checking that claims match evidence and identifying logical gaps
Each stage demands different model capabilities. Mapping stages to capability requirements is far more useful than relying on overall leaderboard rankings.
Building a Reproducible Evaluation Framework
When evaluating models, fix a set of test cases relevant to your business and feed identical inputs to every candidate model. Common evaluation dimensions are listed below.
| Dimension | Key Question | Test Method |
|---|---|---|
| Reasoning and planning | Can it break down multi-step problems and stay consistent? | Present a planning problem with constraints; manually verify each step |
| Long-context comprehension | Can it cover information scattered across a long document? | Plant facts throughout a long text and measure recall rate |
| Instruction following | Does it strictly honor format, length, and structure requirements? | Design prompts with hard constraints; validate output with scripts |
| Structured output | Can the generated content be parsed directly by programs? | Request JSON output and measure parse success rate |
| Cost and latency | Are time and cost per task acceptable? | Log latency and token usage for the same batch of tasks |
Below is a generic evaluation script framework. The model invocation function is encapsulated by your team; the script itself does not depend on any specific service.
from typing import CallableTestCase = tuple[str, str, Callable[[str], bool]] # (name, prompt, check function)def run_eval(models: dict[str, Callable[[str], str]],cases: list[TestCase],) -> dict[str, float]:"""Run the same test suite against each candidate model and return pass rates."""scores: dict[str, float] = {}for model_name, call in models.items():passed = 0for _, prompt, check in cases:answer = call(prompt)if check(answer):passed += 1scores[model_name] = passed / len(cases)return scores
Check functions should be programmatic wherever possible—for example, verifying word count ranges, testing whether JSON parses, or checking that key facts appear. Items that cannot be checked programmatically should be scored manually against a unified rubric.
Record the model version, parameter settings, and test date alongside results. Once a provider updates a model, the same inputs may produce different outputs; conclusions without version information are hard to reproduce.
Three Architectures for Multi-Model Collaboration
Once you know the capability requirements of each stage, you can pick an appropriate collaboration architecture. Here are three common patterns.
3.1 Routing Pattern
A lightweight classification step first determines the task type, then dispatches it to the corresponding model. For example, format conversion goes to a low-cost model, while complex reasoning goes to a high-capability one.
- Pros: lower overall cost, faster response times
- Risks: classification errors send tasks to unsuitable models; fallback rules are needed
3.2 Pipeline Pattern
Tasks flow through stages in sequence: planning, execution, review. Each stage has well-defined input and output structures, and each stage’s output becomes the next stage’s input.
- Pros: clear responsibilities make it easy to locate which stage a problem came from
- Risks: errors from earlier stages propagate downstream, so structural validation should be added between stages
3.3 Cross-Validation Pattern
Multiple models independently answer the same question, and their key conclusions are compared. When conclusions agree, adopt them directly; when they diverge significantly, escalate to human review.
- Pros: reduces the blind spots of any single model
- Risks: invocation costs multiply; majority agreement does not guarantee correctness—key facts still need to be traced back to source material
Choose the architecture based on the task’s fault-tolerance requirements. Batch tasks with low fault tolerance suit the routing pattern; long workflows requiring step-by-step checkpoints suit the pipeline pattern; high-stakes judgments suit the cross-validation pattern.
Full-Coverage Processing of Long Materials
4.1 The Problem: Sampling-Based Reading Causes Omissions
When reference materials reach hundreds of files, models tend to search for keywords first and then read only a subset of files rather than reading everything. Useful information gets missed, and conclusions may be biased toward the few files that were read.
Users often add forced instructions to prompts, such as “read all files, do not skip anything.” Such instructions can influence model behavior, but they cannot prove that every file was actually covered. Coverage must therefore be verified through engineering processes—not through prompts alone.
4.2 The Three-Step Method: Ledger, Chunking, Summarization
Step one: build a ledger. List all files, compute a hash for each, and track processing status.
import hashlibfrom pathlib import Pathdef build_ledger(root: str) -> list[dict]:"""Build a processing ledger for every file under a directory."""ledger = []for path in sorted(Path(root).rglob("*")):if not path.is_file():continuedigest = hashlib.sha256(path.read_bytes()).hexdigest()ledger.append({"file": str(path),"sha256": digest,"status": "pending", # pending / summarized / used / skipped"reason": "",})return ledger
Step two: chunk. Long files should be split into segments, with adjacent segments overlapping slightly so sentences are not cut off mid-thought.
def chunk_text(text: str, max_chars: int = 4000, overlap: int = 200) -> list[str]:"""Split text by character count, keeping an overlap between adjacent chunks."""if overlap >= max_chars:raise ValueError("overlap must be smaller than max_chars")chunks = []start = 0while start < len(text):end = min(start + max_chars, len(text))chunks.append(text[start:end])if end == len(text):breakstart = end - overlapreturn chunks
Step three: summarize. Generate structured summaries file by file and chunk by chunk, and record for each file whether it was used or excluded—along with the reason for exclusion. Once all processing is complete, verify that no items in the ledger remain in the pending state.
def summarize_all(ledger, read_file, summarize):"""Generate summaries file by file and confirm no ledger items are left behind."""for item in ledger:if item["status"] != "pending":continuetext = read_file(item["file"])partial = [summarize(chunk) for chunk in chunk_text(text)]item["summary"] = "\n".join(partial)item["status"] = "summarized"unfinished = [i["file"] for i in ledger if i["status"] == "pending"]if unfinished:raise RuntimeError(f"Files not fully processed: {unfinished}")return ledger
Note that these examples assume files are directly readable text. Binary files, scanned documents, and spreadsheets must be converted to text first, with conversion results marked in the ledger. Files that cannot be converted should be marked as skipped with a stated reason—never silently discarded.
4.3 A Prompt Structure for Socratic, Layer-by-Layer Questioning
An iterative, layer-by-layer questioning approach helps deepen the analysis.
