RAG Autopsy

Most RAG evaluation quietly assumes the retrieved documents agree with each other. Real company knowledge bases do not work like that.

Policies get replaced but the old version stays in the search index. An FAQ drifts away from the policy it was summarising. A regional rule contradicts the global one. Duplicate copies survive a migration. Sometimes two current documents simply disagree and there is no correct answer at all.

So this study removes retrieval from the experiment. There is no retriever here and no retriever comparison. Every condition gets an identical set of documents. That means any difference between conditions comes from how the conflict was presented, not from what was found.

There was no existing benchmark for this, so I built one. AutopsyBench is a set of 200 cases about a fictional company, with invented policies and answers that can be checked automatically. 45 cases were used for development and 155 were held out for the real run.

The cases cover 14 kinds of conflict across four domains: expenses, information security, access control and incident management. Some conflicts are resolvable, like an expired policy against a current one. Some are genuinely not, like two current policies of equal standing that disagree.

That last group matters. Because unresolvable cases are included, refusing to answer is scored as a correct outcome when it is correct, and as a failure when it is not. The study measures two opposite errors instead of chasing a single accuracy number.

Six conditions, one variable. Each condition sees the same evidence and differs only in what metadata is visible and what the model is asked to output.

I expected the model to be fooled by out of date documents. It was not. Its actual failure was the opposite one.

At baseline it refused to answer 40.2% of questions that the evidence could actually settle. Not wrong, just unhelpful. It saw two numbers, could not tell which was current, and gave up.

Here is one of those cases in full. Same model, same documents. The only thing that changes is whether the documents show their dates.

Showing effective dates moved accuracy from 59.8% to 84.4%, a paired difference of +25.0pp with a 95% interval of [17.4, 32.6]pp.

Showing source authority instead, so the model could see which document was an official policy and which was an FAQ, moved accuracy only to 63.0% (+3.8pp). That is roughly a tenth of the effect.

You can see why in the per conflict breakdown. On stale_vs_current cases the baseline scored 0% and the metadata condition scored 100%. Without dates the model cannot separate an expired policy from a live one. With dates it is close to perfect.

This is the result worth sitting with. The baseline flags a conflict on every conflicting case, not because it is perceptive, but because it cannot resolve one and says so.

Give it dates and it resolves the conflict, then stops mentioning that the sources ever disagreed. Conflict acknowledgement fell −25.2pp.

For anyone putting RAG inside a company, that is the uncomfortable part. A system can get better at answering while getting worse at telling you the knowledge base is broken. The conflict first protocol was the best balance here: 79.1% accuracy while still reporting 93.5% of conflicts.

A smaller sub study re ran 24 cases three times, moving the governing document to the top, the middle, then the bottom of the context. Nothing about the information changed, only its position.

At baseline that alone moved accuracy by 35.9 percentage points. With metadata visible, the effect was gone in this sub study.

Two kinds of conflict resisted every intervention.

The first is recency_vs_authority: a recent informal note contradicts an older policy that is still in force. The benchmark rule says the policy wins. Accuracy was zero in all six conditions, and the model spotted the conflict every time, but it did not fail the same way throughout. Without dates it simply refused: baseline and authority-only abstained on every case. Once dates were visible it started answering, and each of those 35 answers picked the memo rather than the policy. Metadata did not correct the judgement here. It turned a refusal into a confident wrong answer.

The second is scope. When a regional policy should override a global one, accuracy never rose above about a fifth: 18% at baseline and 18% with full metadata.

The original prediction was that stacking up obsolete duplicates would drag the model toward the stale answer. It did not happen at any duplicate count, in any condition.

This is a real measurement rather than a broken metric. The answer extraction is symmetric, so a stale value is exactly as matchable as the correct one, and the metric fires correctly against test fixtures.

The interventions did not remove failure. They moved it. Baseline fails by refusing to answer. The metadata conditions fix that and introduce a new failure, resolving conflicts silently. Wrong authority and scope errors stay roughly constant throughout.

Ten held out cases with every condition's response. Cases were picked by behaviour, using written criteria, not by which ones looked good. Six of the ten contain a failure.

• One model. Everything here describes google/gemma-4-12b-qat. The zero stale adoption result in particular may not hold for a different model.
• Synthetic benchmark. Fictional policies are cleaner and better structured than real company documentation, and the conflicts are deliberately constructed.
• The resolution policy is a convention. Authority outranking recency is a rule this benchmark states, not a universal law. The headline residual failure depends on it.
• Small samples per conflict type. Between 4 and 17 cases each across 14 conflict types. Those numbers are descriptive and carry no confidence intervals.
• Ceiling effects. Appropriate abstention was 100% and false confidence 0% in every condition, which made two of the preregistered hypotheses untestable.
• Fixed evidence, not live retrieval. This measures interpretation. It says nothing about retrieval quality in production.
• Conflict acknowledgement is a definition. It measures whether the model set conflict_detected on cases the benchmark marks as conflicting. A system that resolves silently scores zero by construction.

Conflicting evidence in retrieval is not a new problem and this study did not find it. Xu et al. (2024) separate knowledge conflict into three families: retrieved text disagreeing with what the model already knows, retrieved sources disagreeing with each other, and the model disagreeing with itself. This study sits entirely in the second family. Su et al. (2024) established that constructed conflict corpora are a legitimate instrument, which is the method used here. Cattan et al. (2025) are closer still, with a conflict taxonomy specific to retrieval and an expert-annotated benchmark. The problem statement here is inherited, not invented.

Two papers bound what I can claim, and I found both only after the results existed, while reading specifically for work that would undermine the contribution rather than support it.

Yu et al. (2026) run more than 50,000 turn-level evaluations across four model families and report that contradiction acknowledgement is uncorrelated with safe resolution. That is close to the trade-off reported here, and it means the observation that detection and resolution come apart is theirs, not mine. What this study adds is narrower and directional: on the same cases, paired, the interventions that raised accuracy by +24.4pp also lowered conflict acknowledgement by −25.2pp. A price, rather than an absence of correlation.

Bacellar (2026) argues that where authorities supersede one another the retrieval objective should be the currently active authority frontier rather than semantic similarity, and Huwiler et al. (2025) model document evolution as a version graph, lifting accuracy on versioned documents from 58 percent to 90 percent. Supersession is therefore already a named problem with existing engineering answers, and this study did not identify it. The difference is where the fix sits. Both of those works intervene at retrieval, so the wrong document is never fetched. This study deletes retrieval from the experiment and asks what the generator does when the superseded document and the current one are both already in front of it.

Three further results frame the findings. Kirichenko et al. (2025) show abstention is unsolved across twenty frontier models, and that reasoning fine-tuning makes it worse by 24 percent on average. Their benchmark asks whether models abstain when they should; this study measures the opposite error, refusing when the evidence does settle the question, which was the dominant failure here at 40.2%. Yao et al. (2026) show models can perceive source authority and that filtering on it improves accuracy, which sits in productive tension with the result here that simply labelling the tier in the prompt left authority selection unchanged at 56.0%. Estimating authority outside the model appears to be a different intervention from stating it inside the prompt. Ge et al. (2025) and Hwang et al. (2024) both act on externally estimated source credibility, and the latter aggregates by majority voting across sources, which is precisely the mechanism this study predicted would drag answers stale and did not.

Gokul et al. (2025) find contradiction detection hard for frontier models, which is worth stating because it did not reproduce here: detection sat at 100.0% at baseline. The likely reason is task construction. These conflicts are blunt numeric disagreements between documents answering the same question, so noticing them is easy and only adjudication is hard. Liu et al. (2023) and Hsieh et al. (2024) are the reason document order is a controlled variable here rather than an accident.

Four gaps come out of this reading, and they are an agenda rather than a claim of territory. Nothing I found reports the transparency cost of an accuracy intervention, which is the trade-off measured here. False abstention on evidence that is resolvable has no dedicated benchmark. Scope conflicts are treated as a retrieval objective by Bacellar but nothing addresses them at the generation step, which matters because metadata did not move them here at all. And every one of these results describes a single model, which given the finding on reasoning fine-tuning is the weakest part of the study.

References

• Bacellar, A. (2026) ‘Controlling Authority Retrieval: A Missing Retrieval Objective for Authority-Governed Knowledge’, arXiv preprint arXiv:2604.14488. Available at: https://arxiv.org/abs/2604.14488
• Cattan, A., Jacovi, A., Ram, O., Herzig, J., Aharoni, R., Goldshtein, S., Ofek, E., Szpektor, I. and Caciularu, A. (2025) ‘DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMs’, arXiv preprint arXiv:2506.08500. Available at: https://arxiv.org/abs/2506.08500
• Ge, Z., Wu, Y., Chin, D.W.K., Lee, R.K.-W. and Cao, R. (2025) ‘Resolving Conflicting Evidence in Automated Fact-Checking: A Study on Retrieval-Augmented LLMs’, arXiv preprint arXiv:2505.17762. Available at: https://arxiv.org/abs/2505.17762
• Gokul, V., Tenneti, S. and Nakkiran, A. (2025) ‘Contradiction Detection in RAG Systems: Evaluating LLMs as Context Validators for Improved Information Consistency’, arXiv preprint arXiv:2504.00180. Available at: https://arxiv.org/abs/2504.00180
• Hsieh, C.-Y., Chuang, Y.-S., Li, C.-L., Wang, Z., Le, L.T., Kumar, A., Glass, J., Ratner, A., Lee, C.-Y., Krishna, R. and Pfister, T. (2024) ‘Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization’, arXiv preprint arXiv:2406.16008. Available at: https://arxiv.org/abs/2406.16008
• Huwiler, D., Stockinger, K. and Fürst, J. (2025) ‘VersionRAG: Version-Aware Retrieval-Augmented Generation for Evolving Documents’, arXiv preprint arXiv:2510.08109. Available at: https://arxiv.org/abs/2510.08109
• Hwang, J., Park, J., Park, H., Kim, D., Park, S. and Ok, J. (2024) ‘Retrieval-Augmented Generation with Estimation of Source Reliability’, arXiv preprint arXiv:2410.22954. Available at: https://arxiv.org/abs/2410.22954
• Kirichenko, P., Ibrahim, M., Chaudhuri, K. and Bell, S.J. (2025) ‘AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions’, arXiv preprint arXiv:2506.09038. Available at: https://arxiv.org/abs/2506.09038
• Liu, N.F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F. and Liang, P. (2023) ‘Lost in the Middle: How Language Models Use Long Contexts’, Transactions of the Association for Computational Linguistics. Available at: https://arxiv.org/abs/2307.03172
• Su, Z., Zhang, J., Qu, X., Zhu, T., Li, Y., Sun, J., Li, J., Zhang, M. and Cheng, Y. (2024) ‘ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLM’, arXiv preprint arXiv:2408.12076. Available at: https://arxiv.org/abs/2408.12076
• Xu, R., Qi, Z., Guo, Z., Wang, C., Wang, H., Zhang, Y. and Xu, W. (2024) ‘Knowledge Conflicts for LLMs: A Survey’, arXiv preprint arXiv:2403.08319. Available at: https://arxiv.org/abs/2403.08319
• Yao, Z., Zhang, H. and Bi, K. (2026) ‘AuthorityBench: Benchmarking LLM Authority Perception for Reliable Retrieval-Augmented Generation’, arXiv preprint arXiv:2603.25092. Available at: https://arxiv.org/abs/2603.25092
• Yu, Z., Xing, W., Ye, C., Teng, X., Yang, B., Lin, C. and Han, M. (2026) ‘Detecting Is Not Resolving: The Monitoring Control Gap in Retrieval Augmented LLMs’, arXiv preprint arXiv:2605.27157. Available at: https://arxiv.org/abs/2605.27157

Four decisions in the code determine whether any of the numbers above mean anything. Each excerpt below is sliced out of the file that actually ran, at build time, so it cannot drift away from the experiment. The rest of the harness is in the repository.

1const EXPOSURE = {2  A: { temporal: false, authority: false },3  B: { temporal: true, authority: false },4  C: { temporal: false, authority: true },5  D: { temporal: true, authority: true },6  E: { temporal: true, authority: true },7  F: { temporal: true, authority: true },8};

1function renderDoc(doc, exposure) {2  const lines = [`[${doc.source_id}] ${doc.title}`];34  if (exposure.authority) {5    lines.push(`Source type: ${AUTHORITY_LABEL[doc.authority] ?? doc.authority}`);6    lines.push(`Owner: ${doc.department}`);7    if (doc.scope) lines.push(`Applies to: ${doc.scope}`);8  }910  if (exposure.temporal) {11    if (doc.version) lines.push(`Version: ${doc.version}`);12    lines.push(`Effective from: ${doc.effective_from ?? "not stated"}`);13    if (doc.effective_to) lines.push(`Effective to: ${doc.effective_to}`);14    lines.push(`Last updated: ${doc.last_updated ?? "not stated"}`);15    lines.push(`Status: ${doc.status}`);16  }1718  lines.push("");19  lines.push(doc.text);20  return lines.join("\n");21}

1async function callGuarded(provider, payload) {2  let res = await callWithRetry(provider, payload);3  if (res.truncated) {4    res = await callWithRetry(provider, { ...payload, maxTokens: MAX_TOKENS_RETRY });5    res.retried_for_truncation = true;6  }7  return res;8}

1export function extractAnswerValue(testCase, answerText) {2  const cands = candidateValues(testCase);3  const found = new Set(numericTokens(answerText).filter((n) => cands.has(n)));4  if (found.size === 1) return { value: [...found][0], status: "ok" };5  if (found.size === 0) return { value: null, status: "none" };6  return { value: null, status: "ambiguous" };7}

The pattern running through all four is that the harness should never be able to produce a flattering result by accident. The six conditions share one rendering path, the scorer treats a stale value exactly as it treats a correct one, and an infrastructure failure is recorded as an exclusion rather than quietly converted into a model error.

The benchmark is generated from a fixed seed, so it rebuilds byte for byte. The model ran at temperature 0 with schema constrained output. The protocol was frozen, including SHA-256 hashes of every prompt, before the held out run started.

Every generation records the provider, model, decoding settings, prompt hash, condition, document order, timestamp and token usage. Truncated responses are recorded as exclusions rather than scored as wrong answers: 13 of 1074 generations were excluded that way, with 0 parse failures among the valid rows.

This is independent research, done outside any institution. Here is exactly who did what, including where AI tools were involved.