I expected the model to believe old documents. It did something else.

There was nothing off the shelf that modelled enterprise document governance, so I wrote a generator. It builds 200 cases about a fictional company, with invented policies, effective dates, authority tiers and supersession notes.

The important part is that ground truth is decided by the generator, not by a model. Each case ships with a rule that says why one document wins: later effective date, higher authority, narrower scope, explicit supersession. That makes scoring automatic and means the benchmark's opinions are written down where people can argue with them.

Some cases have no defensible answer at all. Two current policies, equal authority, flatly contradicting each other. For those, refusing to answer is the correct behaviour, which turned out to matter more than I expected.

I ran the model locally through LM Studio. Two things went wrong before I had a single valid result, and both would have quietly corrupted the numbers.

First, my provider asked for JSON output the way the OpenAI API does. This build rejected that and wanted a schema instead. That was a loud failure, so it was easy to fix.

The second one was not loud. Gemma returns its reasoning separately from the answer content, and reasoning ate most of the token budget. At my original limit the JSON was getting cut off mid-object. A truncated response parses as nothing, and my first instinct was to score that as a wrong answer.

That would have been wrong. A truncated response is my harness failing, not the model being incorrect. I changed it to record truncation as an exclusion, raised the budget after measuring actual reasoning length, and re-ran. About 20 percent of the smoke test had been silently truncating.

My main prediction was majority bias. Fill the context with five copies of an obsolete policy and one current one, and the model would follow the crowd. I had already sketched the animation for it.

Stale adoption came back at zero. Not low, zero, at every duplicate count, in every condition.

I spent a while assuming the metric was broken. It was not. The extraction rule treats a stale value exactly like a correct one, and it fires correctly against test fixtures. The model simply never took the old number.

The real failure was the opposite of what I designed for. At baseline the model refused to answer 40 percent of questions the evidence could settle. It saw two numbers, could not tell which was current, and gave up.

Showing effective dates fixed most of that and moved accuracy up by about 25 points. Showing authority labels instead barely moved anything.

Here is the part I keep thinking about. Once the model could resolve the conflict, it stopped mentioning that a conflict existed. Conflict reporting dropped by about 25 points.

So the system got better at answering and worse at telling you the knowledge base is broken. If you are running this inside a company, the second thing might matter more. A wrong answer gets caught. A silently resolved contradiction does not.

Two things resisted everything. When a recent informal note contradicted an older policy that was still in force, accuracy was zero in all six conditions. What changed was how it failed. With no dates in the context the model refused to answer at all. Once I showed the dates it started answering confidently, and every one of those answers picked the note over the policy. The intervention that fixed most of the benchmark converted a refusal into a wrong answer here.

Scope was the other one. When a regional policy should override a global policy, accuracy never got above about a fifth, with or without metadata.

There was also a large position effect at baseline. Moving the governing document to the bottom of the context cost about 36 points of accuracy. Metadata removed that effect entirely in the sub-study.

For the write-up I wanted the animation to do actual explanatory work rather than decorate. The rule I settled on: if a state change is the finding, animate it, and keep the object identity stable so you can see what changed.

The false abstention figure is one real case. Toggle the dates and the metadata appears on the same two cards, then the recorded answer changes underneath. Nothing is regenerated, nothing is faked, and the outcome shown is the one from the frozen run.

The position figure reorders the same cards rather than redrawing them, because the entire point is that nothing about the information changed.

A second model, first. Every number here describes one local model, and the zero stale adoption result is exactly the kind of thing that might not generalise.

After that, whether you can get the accuracy of the metadata conditions while keeping the conflict reporting of the baseline. The conflict-first protocol got closest, so that is where I would start.