Why LLMs Guess Instead of Admitting They Don't Know: A Practical Guide to Selective Refusal Evaluation

1. The Root Cause: Models Don’t Lack Intelligence, They Lack the Ability to Say “I Don’t Know”

Over the past two years, conversational AI has become a daily entry point for real work. Users turn to it to look up information, write code, and run a first pass on compliance checks, and they expect direct answers. But in real business settings, a wrong answer often does more harm than no answer at all. This is especially true in multi-document Q&A, retrieval-augmented generation (RAG), customer support tickets, and codebase Q&A. These answers must be cross-verified across many pieces of evidence. A single missing or contaminated fact is enough to make the model, driven by linguistic momentum, “fill in” the incomplete information and produce a conclusion that is grammatically perfect and well-formatted, but factually wrong.

The technical root cause of these failures is the lack of selective refusal (also called selective prediction). A mature answer system needs three kinds of judgment:

1. Capability boundary: Is the question covered by the model’s own knowledge or by the materials provided for this request?
2. Evidence sufficiency: Even if relevant material exists, is it enough to support a definitive conclusion?
3. Expressing uncertainty: When evidence is insufficient, can the model give a partial answer with explicit gaps called out, rather than guessing or refusing outright?

Mainstream benchmarks have long focused on whether a model can answer correctly. They have almost no systematic way to measure whether it should answer at all. When accuracy is the only optimization target, the model’s rational strategy is to always respond, because refusing can only lower the score. This is misaligned incentives caused by flawed evaluation design, not simply a lack of model capability.

2. Three Failure Modes and the Metrics That Expose Them

In practice, weaknesses in selective refusal show up in three forms. They must be measured separately; otherwise, an improvement on one metric can mask a regression in another.

Failure mode	What it looks like	Typical business impact	Key metric
Missed refusal (should refuse, doesn’t)	Still answers when evidence is missing; fabricates numbers, clauses, and citations	Factual errors, compliance risk	Unanswerable detection rate, refusal recall
Over-refusal (could answer, refuses)	Answers “I can’t answer that” to questions that are fully supported by the available information	Collapse in user experience, surge in human-handled tickets	Over-refusal rate
Mis-targeted or misattributed refusal	Answers the wrong sub-question, or the stated reason for refusing doesn’t match the actual defect	More misleading, harder to diagnose	Defect-type attribution accuracy

At a minimum, you should track these four metrics:

• Refusal accuracy: the proportion of unanswerable samples correctly identified as unanswerable.
• Over-refusal rate: the proportion of answerable samples incorrectly refused. This one needs its own dashboard to prevent teams from gaming refusal accuracy with a “better silent than wrong” approach.
• Selective accuracy: the accuracy of only the answers the model chose to give when it was confident, which measures the quality of the answered portion.
• Calibration error (ECE): the gap between the model’s stated confidence and its actual accuracy, which is used to set decision thresholds.

A more practical approach is to plot a risk–coverage curve: answer coverage on the x-axis and error rate on the answered portion on the y-axis. The closer the curve sits to the bottom-left, the better, and the area under the curve can serve as a single composite score. On the business side, you can use the harmonic mean of refusal accuracy and answer accuracy on answerable questions as a release gate. This prevents either side from being sacrificed.

3. Why Static Benchmarks Fall Short: From Memorizing a Test Set to Generative Evaluation

The traditional approach is to build a fixed Q&A test set, label which questions are answerable and which are not, and run the benchmark repeatedly. This approach quickly hits a wall:

• Data contamination: Public benchmark sets easily end up in pretraining and instruction-tuning corpora. The model learns the answers and question patterns rather than the ability to judge boundaries.
• Overfitting to the evaluation distribution: Once the way unanswerable samples are constructed is fixed (for example, always using “not mentioned in the text” phrasing), the model can score well just by recognizing the surface template.
• No coverage of long-tail defects: In real business settings, the forms of evidence defects are nearly infinite, so any static set has inherently limited coverage.
• Version drift: Retrieval corpora, knowledge bases, and business rules change every week. What was “answerable” yesterday may be “unanswerable” today.

Evaluation therefore needs to move from a question-bank model to generative evaluation: use parameterized perturbation rules to synthesize new test instances on the fly for each run. The value isn’t a larger question count. It comes from three structural advantages: natural resistance to contamination, controllable difficulty, and the ability to attribute failures to specific defect types.

4. A Perturbation Strategy Framework: Turning Missing Evidence Into a Controlled Variable

The core component of generative evaluation is the perturbator. It takes an original document set and an answerable question, modifies the material or the question according to a specified strategy, and outputs a new labeled instance. In practice, these operations fall into six categories. Teams typically maintain hundreds of atomic operations, grouped into three difficulty levels by combination and intensity.

Six Perturbation Categories

1. Evidence removal: Delete the key passages, table rows, or citations that carry the answer so it can no longer be derived from the material.
2. Evidence conflict: Insert conflicting numbers, versions, or conclusions across multiple documents to test whether the model flags the conflict.
3. Outdated information: Replace facts with older versions of an entity, retired standards, or expired data to test time sensitivity.
4. Out-of-scope questions: Construct domain-external questions that are unrelated to the knowledge base’s topic (for example, across industries or product lines).
5. Referential ambiguity: Introduce entities with the same name, missing subjects, or vague references so the question cannot be resolved unambiguously.
6. False premises: Presuppose facts that don’t hold (nonexistent fields, retired features) in the question to test whether the model corrects the premise.

Three Difficulty Levels

• L1 (local flaw): The answer can be reconstructed from the remaining material, and the ideal behavior is to answer while flagging the uncertain source.
• L2 (partially unanswerable): The evidence covers only sub-questions, and the ideal behavior is to answer in sections with explicit gap labels.
• L3 (fully unanswerable): No sufficient evidence exists, and the ideal behavior is to refuse clearly and state what is missing.

A minimal viable perturbation can be expressed as a declarative configuration, which makes version control and regression comparison easier:

from dataclasses import dataclass, field
@dataclass
class PerturbationSpec:
    op_id: str                 # Atomic operation ID, used for attribution statistics
    category: str              # Evidence removal / conflict / outdated / out-of-scope / ambiguity / false premise
    level: int                 # 1..3, difficulty level
    target_span: str | None    # Locator key for the passage to rewrite
    payload: dict = field(default_factory=dict)
    def apply(self, corpus: list[dict], question: str) -> tuple[list[dict], str, dict]:
        """Return the perturbed corpus, the perturbed question, and the expected-behavior label."""
        ...
# The expected-behavior label determines how the sample is scored, so correctness
# isn't judged by brittle string matching.
# expected_behavior: "refuse" | "partial_answer" | "answer_with_caveat"

There is one key convention when running the evaluation: every sample must carry an expected-behavior label, not just a reference answer. For refuse samples, score whether the model refused and whether the reason it gave identifies the defect type. For partial_answer samples, score whether it correctly draws the line between what is known and what is not.

5. From Offline Evaluation to Online Guardrails: Four Production Components

Offline evaluation only reveals the upper limit of capability; online traffic needs engineering safeguards. Build these four layers in order.

1. Retrieval Sufficiency Check (the First Gate, Before Evaluation)

Before the model generates an answer, a lightweight discriminator decides whether the evidence is sufficient. It looks at the relevance distribution of the retrieved chunks, whether all key entities are present, and whether there are conflicting values. If the verdict is “insufficient,” the request goes directly to refusal or clarification, so the generative model doesn’t fill an information vacuum on its own. This step costs far less than post-hoc verification and offers the best return on investment.

2. Answer-to-Evidence Consistency Check (the Second Gate, After Generation)

Once the answer is generated, extract each assertion sentence by sentence and check whether evidence supports it (attribution check). Unsupported assertions should be removed or downgraded to “speculation” and explicitly labeled in the reply. In practice, use two channels, extractive checking and model-based checking. If they disagree, apply the more conservative option.

3. Refusal and Clarification Wording

Refusal isn’t the same as an error message. A high-quality refusal includes three elements: what information is missing, which conclusions have already been confirmed, and how to get the missing information. Offer “clarification follow-up” as a middle state. A single clarifying question narrows the scope and can typically turn 20%–30% of over-refusals into productive interactions.

def decide_answer(evidence_hits: list[float], conflict: bool, question_scope_ok: bool) -> str:
    """Illustrative decision logic; calibrate the thresholds on business data
    using the risk-coverage curve."""
    if not question_scope_ok or conflict:
        return "clarify"                      # Out of scope or conflicting evidence: clarify first
    if not evidence_hits or max(evidence_hits) < T_LOW:
        return "refuse"                       # Missing evidence: state the gap explicitly
    if max(evidence_hits) < T_HIGH:
        return "partial"                      # Insufficient evidence: partial answer with uncertainty flagged
    return "answer"

4. Wire Evaluation Into CI and Online Monitoring

Turn generative evaluation into a regression gate that runs whenever prompts, retrieval strategies, or model versions change. Fix the random seed so results are comparable. Produce reports broken down by defect category and difficulty level, and block the release if any layer regresses beyond a threshold. On the production side, track three signals: refusal rate, over-refusal appeals, and the rate at which humans rewrite model answers. An abnormal spike in the refusal rate usually means something broke in the retrieval index or chunking strategy, not that the model has degraded.

6. Common Pitfalls and Engineering Advice

• Don’t use a general-purpose LLM as the sole judge: LLM judges tend to favor “polite refusals,” so they easily mistake well-phrased wording for a correct refusal. Combine rule-based checks, an LLM judge, and human spot checks so all three cross-validate.
• Don’t optimize a single metric: The over-refusal rate must be tracked on a long-term dashboard, or teams will drift toward “say less, be wrong less.”
• Keep perturbations close to the real distribution: Synthetic conflict and outdated-information defects should match the forms of real defects in production. Mine high-frequency defect patterns in reverse from zero-hit query logs and human-correction records.
• Don’t ignore multilingual and domain adaptation: Chinese long-form documents have pronoun omission and table and footnote structures that differ significantly from English corpora, so perturbation rules need separate calibration.
• Budget for cost and latency: Sufficiency checks, consistency checks, and clarification follow-ups all add calls. Include this compute in capacity planning during architecture design, and, where needed, enable them in phases by business value after a small-traffic rollout.

7. Key Takeaways

Teaching a model to say “I don’t know” is not a prompt-engineering trick. It’s a system capability that requires both an evaluation framework and a full engineering pipeline. The core conclusions come down to three points. Evaluation goals must move beyond “answer accuracy” to “boundary judgment quality”; otherwise, models always have an incentive to guess. Test sets must shift from static question banks to parameterized, tiered generative evaluation to resist data contamination and attribute defects. And production needs a complete closed loop of retrieval sufficiency checks, answer-to-evidence consistency checks, tiered refusal and clarification handling, and regression gates with monitoring. Miss any one of these three, and a model that looks great in demos will still package incomplete information as a confident answer in real business use.