Using LLMs for Humanities Research: Evaluation Criteria and a Two-Stage Model Selection Workflow

    Large language models are increasingly used in humanities and historical research for literature retrieval and argument mapping, yet models differ sharply in speed, output length, and factual accuracy. This article presents a reproducible methodology for evaluating LLM performance on scholarly tasks, classifies common model behaviors into three types, and introduces a two-stage workflow—clue generation followed by evidence verification—combined with a literature verification checklist to balance efficiency and reliability while controlling hallucination risk.

    Introduction: The Tension Between Efficiency and Reliability

    In humanities research, using large language models to search literature and organize scholarly viewpoints has become common practice. While running cross-tests on the same ancient Chinese history topic, the author found that models differ markedly in response speed, output length, and factual accuracy. Some models answer quickly but superficially; others produce long outputs that may contain fabricated citations. These experiences raise an unavoidable question: how should we evaluate these tools and use them safely in research? This article offers a reproducible evaluation methodology and a two-stage workflow that balances efficiency with reliability.

    1. Standardize Test Conditions Before Comparing Speed

    Before comparing model speed, you must fix the test conditions. At minimum, record the following:

    • Model name and version number;
    • Test date;
    • Identical prompts across all runs;
    • First-token response time and full completion time, using the median rather than a single run.

    Network conditions and server load also affect latency, so it is best to run the same batch of questions within a similar time window. The tests described here are small in scale and only illustrate the general behavior of different model types; they are not sufficient to establish rankings.

    The value of speed depends on the task. When the goal is quickly locating keywords or establishing a general direction, a fast-response model is more appropriate. When the task requires exhaustive literature coverage or multi-angle argumentation, a slower model is often more useful. Determine the task stage first, then decide on speed requirements.

    2. Longer Output Does Not Mean Higher Information Density

    In testing, the model with the longest output did not necessarily provide the most useful information. One model, asked to list literature, produced a very long bibliography with many entries and detailed source information; another returned only three to five core references, but each was more directly relevant to the question.

    Both output styles have their uses. If the goal is to maximize the range of leads, a long list is more valuable. If the goal is direct citation, a short, precise answer saves more time. When judging output quality, consider both the task goal and the downstream verification cost. Long lists usually require far more verification work—a factor easily overlooked during evaluation.

    3. Why Hallucinations Are Especially Dangerous in Humanities Research

    The most serious problem observed in testing was hallucination. Some models provided bibliographic entries that do exist, but upon checking, several papers’ core arguments did not match—or even contradicted—the question’s direction. In another case, the title, author, and journal information were all real, but the paper never discussed the topic at all—a “wrong attribution.”

    Such errors have an outsized impact in humanities research. Once fabricated or mismatched references enter a citation list, the result ranges from wasted lookup time to a complete loss of credibility. The specific mechanism behind hallucination cannot be determined from the outside, so no speculation should be treated as fact. What can be confirmed is that when a question involves a specific dynasty or historical figure, models are more likely to produce seemingly relevant classic references without verifying whether their content actually supports the argument.

    Another concern is output instability. Asked the same question twice, the same model may produce one broadly usable answer and one clearly fabricated answer. If a model consistently errs on the side of omission, users can clearly understand its capability boundary. If answers fluctuate between good and bad, every output requires item-by-item verification, which increases the manual burden.

    4. Three Typical Model Behaviors

    Based on this testing, common models can be roughly grouped into three categories. This classification describes behavioral patterns only and is not an evaluation of any specific product.

    • Exhaustive type: Slower responses, long output lists, broad coverage. Useful as a source of leads, but with higher hallucination risk—every output item must be verified.
    • Precise type: Moderate response speed, concise output, higher relevance between references and the question. Limited by knowledge coverage, typically providing only a small number of core references.
    • Unstable type: Inconsistent performance; different answers to the same question may exhibit the characteristics of either of the first two types. Treat each answer as a new task rather than relying on previous judgments.

    5. A Two-Stage Workflow: Find Leads First, Then Verify Evidence

    Combining two model types can balance efficiency and reliability.

    Stage 1: Clue generation. Use an exhaustive-type model and ask it to list as many relevant references, scholars, and arguments as possible. Place few constraints in this stage—the goal is to build a lead pool. Every item produced here is only a lead to be verified and must never be written directly into a bibliography.

    Stage 2: Evidence verification. Feed the items from the lead pool one by one to a precise-type model, asking it to judge whether each source’s argument supports the claim in question based on the provided abstract or excerpt. Use the following prompt template:

    Please read the abstract or excerpt of the following reference and judge whether its core argument supports “【claim to be verified】,” explaining your reasoning.
    Requirements:

    1. Answer based only on the text I provide; do not add anything beyond it.
    2. Quote sentences from the original text as evidence.
    3. If the text is insufficient to make a judgment, or you cannot confirm the content, answer “uncertain” directly—do not speculate.

    Before use, pay attention to copyright and compliance. Only upload literature or abstracts you are authorized to use, and anonymize them beforehand by removing personal information unrelated to the research. For copyrighted full texts, confirm the permitted scope of use first.

    In one investigation into the causes of regional military governorship (fanzhen) fragmentation in the Tang dynasty, the author first used an exhaustive-type model to gather a batch of leads, then submitted several doubtful items to a precise-type model for verification. The results showed that some items argued in a direction opposite to the research need, some mainly discussed other periods, and only a few were directly relevant. This process significantly reduced manual verification time and lowered the risk of miscitation.

    6. Assign Models by Task Stage

    AI-assisted work in humanities research can be broken into several stages, each suited to a different model:

    • Topic analysis and concept definition: A faster model works well for quickly generating multiple interpretations.
    • Material organization and viewpoint synthesis: A precise-type model suits in-depth analysis of a small set of materials.
    • Exhaustive literature search: An exhaustive-type model expands the lead pool, but must be paired with duplication checks and source verification.
    • Language polishing and format checking: Delegate to dedicated editing tools or rule-based scripts.

    No single model covers every stage. The practical approach is to build your own model assignment table, route each type of task to the most suitable tool, and record each tool’s actual performance on that task type.

    7. Literature Verification Checklist

    Before citing any AI-recommended reference, complete the following checks for each item:

    Check Method Action if Failed
    Reference existence Search library catalogs, academic databases, or publisher websites Treat as suspected fabrication; delete
    Author and journal information Verify character-by-character against the original source Correct or delete
    Argument relevance Read the abstract or full text to confirm its relationship to the claim Mark as irrelevant or contrary; do not cite as supporting evidence
    Full-text accessibility Confirm whether the full text or a reliable abstract can be obtained If unobtainable, mention only as a lead, never as evidence

    Only when all four checks pass can the reference be cited. If a reference cannot be found through any public channel, treat it as suspected fabrication.

    8. Allow the Model to Say “Uncertain”

    Users need to accept “uncertain” as a valid answer. If you always demand a definitive response, the model is more likely to fabricate content. Therefore, prompts should explicitly state: “If you cannot confirm, answer ‘uncertain’.” When you receive such an answer, treat it as a normal result, not a failure.

    9. Common Mistakes

    • Copying AI-generated lists directly into a bibliography.
    • Verifying only titles and authors without checking content.
    • Treating “uncertain” as a capability deficiency and repeatedly pressing the model until it gives a definitive answer.
    • Uploading unauthorized full texts or personal information without anonymization.
    • Accepting a result after asking a question only once, ignoring output variability.

    Conclusion: Adjust by Task, Verify Continuously

    The performance of large language models in humanities research assistance shows a clear tension between efficiency and reliability. Fast models often lack depth, while deep models are more prone to hallucination. By standardizing evaluation criteria, assigning models by task, adopting a two-stage workflow, and insisting on item-by-item verification, model weaknesses can be kept within manageable bounds.

    Disciplines differ considerably in literature structure and evidentiary requirements, so selection criteria and verification intensity should be adjusted accordingly. Researchers can gradually build their own records of model capabilities and adapt their usage by task type. This is one of the main ways to improve the reliability of AI-assisted research—and a necessary measure for maintaining the integrity of scholarly citation.