# RAG Autopsy: When the Knowledge Base Disagrees With Itself

Independent research by Muhammad Usman Mateen.
Canonical article: https://usmanmateen.com/research/rag-autopsy
Published 2026-08-16, last revised 2026-09-04.

> This is a machine-readable summary of the full article, generated from the same
> frozen results the article renders from. It is a summary, not the whole paper.

## The question

Most RAG evaluation asks whether a system retrieves the right evidence. This study
starts one step later. Every condition receives an identical evidence pack, and the
documents in it disagree with each other, in the way a real company knowledge base
disagrees with itself: superseded policies left in the index, an FAQ that has drifted
from the policy it summarised, a regional rule that contradicts the global one.

## Method

Retrieval is removed from the experiment so that any difference between conditions is
attributable to presentation rather than to what was found. Six conditions differ only
in which governance metadata the prompt may state:

- A baseline: title and body text only
- B temporal: adds effective dates, last-updated and superseded status
- C authority: adds source tier and owner
- D temporal + authority: both
- E conflict-first: full metadata, model must assess the conflict before answering
- F contradiction gate: a separate consistency check runs first

155 held-out cases, 1061 valid generations, 13 excluded,
0 parse failures. Model google/gemma-4-12b-qat at temperature 0.
Hypotheses were preregistered and prompts hashed and frozen before the held-out run.

## Results

| Condition | Answer accuracy | Conflict acknowledgement | False abstention |
|---|---|---|---|
| A Baseline | 59.8% | 100.0% | 40.2% |
| B Temporal metadata | 84.4% | 76.8% | 8.1% |
| C Authority metadata | 63.0% | 100.0% | 37.0% |
| D Temporal + authority | 84.2% | 73.9% | 8.3% |
| E Conflict-first | 79.1% | 93.5% | 14.9% |
| F Contradiction gate | 78.5% | 90.0% | 10.0% |

## Findings

1. **Showing effective dates is the single effective intervention.** Accuracy rose
   +25.0pp from baseline. Authority
   labels alone moved it +3.8pp.

2. **The dominant failure was false abstention, not a wrong answer.** At baseline the
   model refused to answer 40.2% of questions the supplied
   evidence could settle.

3. **Accuracy was bought with transparency.** Conflict acknowledgement fell
   −25.2pp as accuracy rose
   +24.4pp. The system got better at
   answering and quieter about the knowledge base being broken.

4. **The main prediction was contradicted.** Stale adoption was
   0.0% at every duplicate count. Filling the context with
   obsolete copies did not drag the answer stale.

5. **Two failures resisted everything.** Recency versus authority stayed at
   0% accuracy in
   all six conditions, and scope conflicts never rose above about a fifth.

6. **Position mattered at baseline.** Moving the governing document changed accuracy by
   35.9 percentage points. Metadata removed the effect in the sub-study.

## What this does not show

One model was tested, so none of this is established as general. The corpus is
synthetic. Per-archetype samples are small and carry no confidence intervals. The rule
that authority outranks recency is this benchmark's stated convention rather than a
universal law, and the headline residual failure depends on it. The study measures
interpretation of fixed evidence and says nothing about retrieval quality in production.

## Related work

The problem statement is inherited, not invented. Conflicting sources in RAG are
established (Xu et al., 2024; Su et al., 2024; Cattan et al., 2025). That detection and
resolution come apart is reported by Yu et al. (2026). Supersession as a retrieval
objective is addressed by Bacellar (2026) and Huwiler et al. (2025). This study's
narrower contribution is holding retrieval constant and measuring the paired cost of
metadata exposure on the same cases.