Why Retrieval Fails: Relevant ≠ Required

A benefits assistant receives this question:

Can part-time contractors receive the wellness stipend?

Its retriever returns passages about employees, contractors, and wellness benefits. Every result looks relevant. Only one may contain the worker category, employment status, benefit, date, and exception needed to answer the question.

That gap is the central retrieval problem: relevant material is not necessarily required evidence.

Related and required answer different questions

In this lesson, related means semantically or topically close to the query. Required means necessary to justify the specific answer.

Consider three candidate chunks:

Retrieved passage	Why it may rank well	Does it answer the question?
Employees receive a wellness stipend after 90 days.	Same benefit and workplace domain	No. It covers the wrong worker category.
Contractors may receive equipment reimbursement.	Same worker category and benefits domain	No. It covers the wrong benefit.
Part-time contractors are excluded from wellness stipends.	Same worker category, status, and benefit	Yes, if it is current and authoritative.

The first two are not random noise. Their relatedness is exactly why the failure can be difficult to notice. An embedding comparison or other ranking signal may reasonably place them near the query while lacking a direct representation of what evidence the final task requires.

How the silent failure forms

Follow the first passage through a simplified RAG path:

question about part-time contractors
-> retriever ranks an employee wellness passage highly
-> application injects that passage into context
-> model sees fluent, on-topic policy text
-> model generalizes the employee rule to contractors
-> answer appears grounded because a source was present

Nothing in that path has to crash. Search returned a result. Context assembly completed. The model generated valid text. A citation may even render correctly.

The missing signal is adequacy: the supplied passage does not establish the rule for the entity in the question.

A model cannot reliably identify a document that never entered its context. It may notice that the evidence is incomplete and abstain, but it does not have automatic access to the missing contractor policy or knowledge of which unseen passage should have ranked higher.

Retrieval failure has several locations

“The retriever was wrong” can still be too broad. Inspect where required evidence fell out of the path.

Failure surface	What happened	What the model receives
Corpus coverage	The current policy was never ingested	No path to the required source
Parsing or chunking	A rule was separated from its exception	An incomplete evidence unit
Metadata or permissions	A date, tenant, product, or access filter excluded the right record	Candidates from the wrong scope
Query representation	The search emphasized wellness but lost part-time contractor	Broadly topical candidates
Ranking and cutoff	The right chunk exists but falls below the returned top results	Higher-ranked related material only
Freshness	An older policy outranks the current version	Plausible but stale evidence
Selection and injection	The right candidate was returned but not placed in context	A context-assembly miss
Generation	The right evidence was supplied but misread or ignored	Adequate context, incorrect use
Citation	The answer cites a nearby source that does not support its claim	A provenance or support mismatch

The rows demand different fixes. Expanding the candidate count may help a ranking cutoff but not a missing source. Rechunking may preserve an exception but not repair stale metadata. A stronger model may use good evidence better but cannot recover a document excluded by permissions.

Similarity ranks candidates, not obligations

An embedding vector represents learned relationships that are useful for comparison. It does not encode a task-specific contract saying which legal clause, version, jurisdiction, or exception must appear before an answer is allowed.

The query itself may also be underspecified. Can contractors receive the stipend? could depend on country, contract type, start date, or program version. Retrieval can only rank against the information and filters the system has.

Useful retrieval systems therefore combine several controls when appropriate:

• lexical and semantic search;
• metadata filters for date, tenant, locale, or document status;
• reranking that considers the full query and candidate text;
• rules that require certain source types;
• abstention or clarification when evidence is incomplete;
• evaluation against known query-to-evidence pairs.

No combination is universally best. Each adds assumptions and its own failure surface.

More data can make selection harder

Adding documents increases possible coverage. It can also add duplicates, obsolete versions, contradictions, low-quality summaries, and near matches that compete with the required source.

The relevant quantity is not simply how much information exists. It is whether the system can select the right evidence under realistic requests and constraints.

This is why retrieval evaluation should include varied domains and query types rather than a small set of polished examples. BEIR formalizes that concern for information-retrieval research by evaluating across heterogeneous datasets. A production corpus needs its own representative tests because its documents, permissions, and questions differ from a public benchmark.

A prompt cannot manufacture missing evidence

Suppose the application adds this instruction:

Answer only from the retrieved context. Be accurate and cite the source.

That instruction can improve behavior. It may encourage abstention and reduce unsupported extrapolation. It cannot make the current contractor policy appear if retrieval supplied only an outdated employee passage.

Worse, the model can follow the instruction faithfully and still produce the wrong answer from wrong context. Prompt quality and evidence quality are separate controls.

The precise claim is not that prompts are useless. It is that a prompt constrains generation from available input; it does not repair corpus coverage, freshness, permissions, ranking, or context injection by itself.

Citation presence is not citation integrity

A generated answer may contain a source marker and still be unsupported. Check at least three relationships:

1. Source identity: Does the citation resolve to the intended document and version?
2. Claim support: Does the cited passage actually entail or justify the answer’s claim?
3. Scope fit: Does it cover the right entity, date, jurisdiction, and conditions?

The wellness-policy example can produce a perfectly real citation to an employee policy. The citation’s existence does not extend that policy to contractors.

This separates two evaluation layers:

• retrieval evaluation asks whether required evidence was found and ranked usefully;
• answer evaluation asks whether generation used supplied evidence correctly and represented its support honestly.

Passing one layer does not force the other to pass.

What retrieval metrics can and cannot tell you

Retrieval teams often measure whether relevant or labeled-required items appear among the top candidates and how highly they rank. Metrics such as recall at a cutoff can reveal that required documents are frequently absent from the returned set. Ranking-oriented measures can compare candidate order across systems.

Those measurements require a defensible relevance judgment set. If labels mark every topical document as relevant while the product needs one specific clause, the metric can look healthy while answers fail.

Retrieval metrics also stop before generation. They do not show whether the application injected the candidates, whether the model followed them, or whether the citation supports the wording of the final claim. End-to-end checks must reconnect those stages.

Diagnose from the trace

For a consequential answer, retain enough structured evidence to ask:

1. Which corpus version and filters were active?
2. What query representation or search request ran?
3. Which candidates returned, with what metadata and rank?
4. Which passages actually entered the model context?
5. Which answer claims depend on which passages?
6. Was required evidence absent, ignored, contradicted, or misquoted?

This trace separates a retrieval miss from a reasoning or generation miss. Without it, a team may keep changing prompts when the current policy was never indexed.

“If RAG returns a relevant source and the prompt says to use it, the answer is grounded.”

Relatedness is a candidate signal. Grounding requires evidence that supports the exact claim under the right scope. Missing, incomplete, stale, conflicting, or untrusted context can produce a fluent answer even when the model follows its instructions.

The rule to carry forward

Before trusting a retrieved passage, ask what the answer would be unable to justify without it. That identifies the evidence that is required rather than merely adjacent to the topic.

RAG creates a route from external sources to current context. Reliability depends on the quality of every handoff along that route. The next architectural question is when a system should use that route at all, and when a maintained fact, structured lookup, curated context, or general model knowledge is a better fit.