Preventing LLMs from Taking Retrieval Shortcuts in Large Document Tasks

    When processing hundreds of files, Large Language Models (LLMs) often rely on retrieval shortcuts instead of reading all content, leading to incomplete coverage in tasks like audits, code migrations, or compliance checks. This article explores the root causes of this behavior and introduces a constrained workflow to enforce full document reading. By using explicit file manifests, context-based batching, structured prompts, and automated coverage validation, developers can ensure that LLMs process every file and produce verifiable, traceable results.

    1. Background: Why Do Models Retrieve Instead of Reading Everything?

    When handling tasks involving hundreds of files, many users find that the model’s answers appear complete but actually only cover a small fraction of the documents. The model first uses keywords or semantic retrieval to find “seemingly relevant” paragraphs, and then answers based on these fragments. While efficient for Q&A scenarios, this approach directly causes omissions in tasks requiring full coverage, such as audits, codebase migrations, compliance checks, and knowledge base consolidation.

    What makes this trickier is that retrieval-based results cannot easily self-prove their completeness. The smoother the answer, the easier it is for users to mistakenly assume that all materials have been considered. Therefore, the key to solving this problem is not switching to a more powerful model, but rather redesigning the task execution method and acceptance criteria.

    2. Root Cause Analysis

    Lazy retrieval behavior is usually caused by a combination of the following factors:

    1. Limited Context Budget: A model has an upper limit on the amount of text it can process at once. Faced with massive files, the cost of reading them one by one is much higher than retrieval. Without explicit requirements, the model tends to choose the low-cost path.
    2. Instructions Lack “Must Read One by One”: If the prompt simply says “analyze these files and find risks,” the model will decide for itself which files need to be read. Its criteria may not align with the user’s.
    3. Lack of Completion Criteria: If the task does not define what “done” means, the model will stop when it thinks it has gathered enough information, rather than when all files are processed.
    4. Output Format Hinders Verification: Free-text conclusions make it difficult to reverse-engineer which files were actually processed, leaving users unable to check coverage.

    Thus, the essence of the problem is an “invisible execution process and unclear completion standards.”

    3. Design Principles

    Addressing the causes above, we can decompose bulk reading tasks into a constrained workflow based on four principles:

    • Explicit Coverage: Every file has a unique identifier and status. By the end of the task, each must be explicitly marked as processed or unprocessed.
    • Verifiable: The final result includes coverage statistics. The number of processed files must equal the total number in the manifest; otherwise, the task is considered incomplete.
    • Hierarchical Processing: First establish a file manifest, then generate structured summaries for each file, and finally perform cross-file aggregation. The artifacts at each level can be independently inspected.
    • Retryable: If a batch fails, only that batch is rerun, without affecting the completed parts.

    4. Workflow Implementation

    4.1 Step 1: Inventory Files and Generate a Manifest

    The manifest is the foundation of the entire process. It records the path, size, and status of each file, with the initial status set to “pending”. The manifest itself should be saved as a file for easy checkpoint recovery later.

    1. from pathlib import Path
    2. import json
    3. def build_manifest(root: str, exts=(".md", ".py", ".txt")) -> list[dict]:
    4. """Traverse the directory to generate a manifest of files to be processed."""
    5. manifest = []
    6. for p in sorted(Path(root).rglob("*")):
    7. if p.is_file() and p.suffix in exts:
    8. manifest.append({
    9. "path": str(p),
    10. "size": p.stat().st_size,
    11. "status": "pending",
    12. })
    13. return manifest
    14. def save_manifest(manifest: list[dict], target: str) -> None:
    15. Path(target).write_text(json.dumps(manifest, ensure_ascii=False, indent=2), encoding="utf-8")

    4.2 Step 2: Batch Processing Based on Context Budget

    A model’s single input is limited, so files need to be split into batches based on a budget. Byte count is a rough proxy for estimating token count; the actual threshold should be adjusted according to the context and output length of the model in use. The 200,000 bytes in the example is for demonstration purposes only.

    1. def make_batches(manifest: list[dict], max_bytes: int = 200_000) -> list[list[dict]]:
    2. """Split the manifest into batches based on the byte budget.
    3. A single oversized file will occupy its own batch and needs to be segmented separately."""
    4. batches, current, used = [], [], 0
    5. for item in manifest:
    6. if current and used + item["size"] > max_bytes:
    7. batches.append(current)
    8. current, used = [], 0
    9. current.append(item)
    10. used += item["size"]
    11. if current:
    12. batches.append(current)
    13. return batches

    4.3 Step 3: Explicit Reading Instructions

    The prompt must explicitly prohibit retrieval substitutions and mandate the output structure. Below is a ready-to-use template.

    1. Please read each file in the following list one by one. Do not use keyword search or sampling as a substitute for reading.
    2. For each file, output a fixed structure:
    3. - path: File path
    4. - status: done or failed
    5. - summary: Key content of the file relevant to the task, no more than 200 words
    6. - findings: Issue items found; if none, provide an empty list
    7. After processing all files, separately output the "number of processed files" and the "total number of files in the manifest".
    8. If the two do not match, list the paths of the unprocessed files.

    The benefit of a fixed structure is that every file must leave a record. A missing record is in itself a signal of an omission.

    4.4 Step 4: Coverage Validation

    Before aggregation, the program should first check coverage. When coverage is incomplete, the task should not proceed to the next stage.

    1. def check_coverage(manifest: list[dict], results: dict[str, dict]) -> dict:
    2. """Compare the manifest with the results returned by the model to find uncovered files."""
    3. missing = [m["path"] for m in manifest if m["path"] not in results]
    4. failed = [p for p, r in results.items() if r.get("status") != "done"]
    5. return {
    6. "total": len(manifest),
    7. "done": len(manifest) - len(missing) - len(failed),
    8. "missing": missing,
    9. "failed": failed,
    10. }

    Only when both missing and failed are empty should the aggregation phase begin. Otherwise, the corresponding batches should be rerun.

    4.5 Step 5: Aggregation and Cross-Validation

    The aggregation phase processes structured summaries rather than the original files. This significantly reduces the input scale. For key conclusions, relevant files can be selected for re-verification, mapping the aggregated results back to specific paths to ensure every conclusion can be traced to its source.

    5. Common Issues and Considerations

    Oversized Single Files: When a single file exceeds the context limit, it should be segmented by chapters or fixed line counts. Each segment should generate its own summary, which are then merged into the file’s summary. Segment boundaries should retain a small overlap to avoid truncated sentences.

    File Dependencies: For example, configuration files referenced in multiple places, or terms defined across several documents. In such cases, common definitions should be extracted first to form a glossary or interface list, which subsequent batches can then reference.

    Sampling Review Cannot Replace Full Coverage: Sampling can be used to check the quality of summaries, but it cannot serve as proof of coverage. Coverage proof can only come from a line-by-line comparison between the manifest and the results.

    Balancing Cost and Latency: The overhead of full reading is significantly higher than retrieval. You can classify tasks by value: apply the full coverage workflow to high-risk directories, and use retrieval plus sampling for low-risk content. Classification rules should be written into the task instructions to prevent users from mistakenly assuming all content has undergone full inspection.

    Traceable Results: Every conclusion should be accompanied by its source path and location for manual review. Conclusions without a source should be treated as unverified.

    6. Implementation Recommendations

    In an enterprise environment, this workflow can be deployed on general-purpose infrastructure. The file manifest can be stored in object storage, batch tasks can be scheduled via message queues, execution status recorded in a database or logging service, and monitoring alerts can be configured to notify the owner when coverage consistently falls below a threshold or batch failures exceed the limit. General-purpose capabilities provided by cloud platforms like Baidu AI Cloud can be combined and utilized in this manner.

    During implementation, a three-step approach is recommended. First, validate whether the manifest, batching, and coverage checks work properly using a small-scale directory. Second, finalize the prompt template and output structure, accumulate a batch of manually reviewed samples, and evaluate the summary quality. Third, integrate the workflow into production tasks, retaining the manifest and results from each run as an audit trail.

    In summary, ensuring that a model fully processes massive files relies not on stronger prompt engineering tricks, but on a visible execution process and verifiable completion standards. By turning “did it finish reading?” into a programmatically checkable condition, lazy retrieval naturally has nowhere to hide.