Can Retrieval Tell Which Source to Trust?

Picture a support RAG bot that retrieves this year's refund policy and last year's policy in the same set. Both pages talk about refunds. The ranker has done its job. The answer still cites the old rule.

Manoj Kumar Vohra asked the question in the title of this page in a Medium article of 9 October 2026 and measured it on synthetic documents: one refund question with eight long documents in his main test, and three questions (the refund window, the length of a free trial and an uptime guarantee) in his chunk-level test. The short answer is not on its own.

TL;DR

• Retrieval scores relatedness, not a source's right to decide. Every stage of a typical pipeline, in Vohra's words, "answers the same question: how related is this document to what was asked?", and "An outdated policy page is highly related to a question about the current policy."
• Picking the right source takes the source's type, its status and the right one of three dates. In his small-chunk test on synthetic documents, with each source's title, type, date and status removed, every system's freshness AUC, averaged over his three questions, fell to between 0.51 and 0.72, where 0.50 is a coin flip: a direction, not a rate for your corpus.
• A newer document that explicitly supersedes older ones should win even when several older copies still agree. This page adds one condition: it wins from its effective date, not its publication date.

Three dates do three different jobs:

• The effective date is the date from which a document's rule applies. It decides whether the document applies on the day of the question.
• The publication date is the date a document was first made public. It orders versions and can come weeks before the effective date.
• The last-modified date is the date a document was most recently changed in any way. A typo fix moves it as much as a rewrite does.

Explicit supersession is a statement, recorded with the newer document, that it replaces a named older one. In a reply of 9 October 2026 to a comment by this page's author, Vohra proposed it as the rule: "If a newer, approved policy explicitly supersedes an older version, it should take precedence even when multiple older documents still agree with each other."

Mnemoverse, which publishes this library, makes one of the memory systems named below.

What does retrieval actually score?

Vohra describes a typical hybrid pipeline in four stages: keyword matching, embedding search, rank fusion and an optional reranker. Each scores how related a document is to the question; none scores authority. In his words, a reranker will rank the outdated page "right behind the correct page, and a language model reading both may pick the wrong one."

Ranking has not failed here: as usually built, it measures relatedness, and the outdated page uses the same nouns as the question. So does a draft, and so does a forum thread that argues with the policy.

His long test set shows it. "The long set is one question and eight documents." On it, a reranker "ranked the unapproved draft and the superseded policy in positions 3 and 4, ahead of the official-but-unrelated documents. They are on topic, so it scores them as related. It cannot know they are wrong."

Vohra is fair to the tool: "A reranker is fast and cheap, and relevance is the one thing it does well." And "The rerankers received only the question and the text." Relatedness gets a document into the candidate set. Something else has to decide whether that document is allowed to settle the answer.

A knowledge graph has the same limit: "Graph traversal answers “what is connected to this?” It does not answer “which of these values should I trust?”" If a draft, a live policy and a superseded policy all sit next to "refund window", the walk finds all three.

What does a pipeline need to know to pick the right source?

Source details are the fields that describe where a passage comes from rather than what it says: its title, its type, its date and its status. The name is Vohra's. In his test, "Every document was sent to the decision models with four fields next to its text: a source title, a source type (official, internal, forum, blog), a date and a status (current, superseded, draft, unverified)."

His Finding 8 takes them away. "The small-chunk tests were run in two variants: with the source title, type, date and status, and with only the text." With the text alone, "Freshness fell for every system, to between 0.51 and 0.72. Some of that is unavoidable: a superseded policy cannot be recognised without a date or a version number, so this measures missing information as much as model skill." The figures are AUC computed per question and averaged over the three questions, which the caption of his Figure 11 calls three scenarios; on his scale "1.00 means always, 0.50 is a coin flip, and near 0 means backwards."

The test removed all four fields (title, type, date and status), not the date alone; the date or version number is his explanation of why freshness cannot be read from wording. The test is small, and "The documents are synthetic."

His advice: "Send real source details (title, type, date, status) if your index has them." A pipeline that stores those fields and drops them before the model reads recreates the without-source-details condition on its own documents. A prompt cannot invent a status the index never stored.

Then there is the document with no date. Christopher Ramm, in a Medium reply of 7 October 2026 to this page's author, about the knowledge base Claude Code maintains for him: "What I'll fix first is dating: many conflicts have an undated document on one side, so I can't tell whether they are conflicts or just old versions."

So record type, status and dates when a document is ingested, not when a question arrives. When two documents disagree and no date settles which is newer, mark the pair undated instead of forcing it into "conflict" or "old version". That is this page's recommendation, not Ramm's wording. A wrong guess either way hides the problem.

Effective date vs publication date: which date says whether a document is current?

The three dates answer three different questions:

Date	Question it answers	Use
Effective date	From when does this rule apply?	Decide whether the document is current
Publication date	When was this document first made public?	Order published versions
Last-modified date	When did anything in this document last change?	Keep the edit history

Shared vocabularies keep them apart. Schema.org defines datePublished as "Date of first publication or broadcast.", dateModified as "The date on which the CreativeWork was most recently modified or when the item's entry was modified within a DataFeed." and validFrom as "The date when the item becomes valid." Its legislationDateOfApplicability, derived from the ELI ontology (European Legislation Identifier), separates applicability from entry into force: "a text may come in force today, and state it will become applicable in 3 months." The example comes from legislative vocabulary: it shows that the distinction exists, not that policy pages use it.

A memory store adds a fourth date: when the record was written. Zep's ingestion guide asks for the original event time in reference_time and adds: "If it is absent, Zep uses ingestion time." The two axes are often called valid time (when a claim holds in the world) and transaction time (when the store learned it). How a memory record can carry both is covered in the library page on outdated patterns after a correction.

The working rule: whether a document is current is decided by its effective date and the date it stops applying. The publication date orders versions. The last-modified date says nothing about the rule.

Superseded documents: do several older documents that agree outweigh a newer one?

Not by count. In Vohra's Finding 7, when the contradiction question was phrased the natural way, asking whether a document contradicts any other evidence, "The correct document, D1, got marked as contradicting by several systems, because three other documents disagreed with it."

The stricter phrasing he used for the long set asks something else. Under it, "a document only counts as contradicting if its own claim is wrong or outdated compared with the most authoritative current evidence", and the contradiction scores he reports for the correct document under it are low. The natural phrasing counts disagreement. The stricter one asks which source is allowed to decide.

In his reply he proposes not relying on the count alone: "I think the more robust solution is to make contradiction resolution time-aware rather than relying solely on the number of sources supporting a claim." The supersession sentence quoted at the top of this page follows it.

Agreement is weak evidence when the agreeing documents share an origin. The old terms, an FAQ written from them and a support macro written from the FAQ are one witness repeated: one decision, photocopied. Who Wins When Agents Disagree? treats this as an authority problem, and The Independence Illusion explains why copies counted as votes mislead.

What should an answer say when a new policy is published but not yet in effect?

Take the rule as written: newer, approved, explicitly supersedes. A policy published today that takes effect in a month meets all three conditions on the day it is published, so the rule as written makes it current that day. To apply the rule, a pipeline also has to know when the replacement takes effect.

The setup: three documents (the 30-day terms, an FAQ and a support macro) allow refunds within 30 days. A 14-day policy is approved and published on 1 October, takes effect on 1 November, and explicitly supersedes the 30-day terms. On 10 October someone fixes a typo in the old FAQ.

rule	asked on 15 October	asked on 5 November
newest publication wins	14 days, wrong: not yet in effect	14 days, right
most documents agree	30 days, right by count	30 days, wrong: three outdated pages still agree
latest edit wins	30 days, right by accident	30 days, wrong
explicit supersession, from the publication date	14 days, wrong	14 days, right
explicit supersession, from the effective date	30 days, with "14 days from 1 November", right	14 days, right

Illustration, not data: the dates and documents are invented to show how each rule behaves.

On 15 October the new policy is approved and published but does not apply yet, so both rules that count from publication answer 14 days, wrong for a customer who asks that day. Counting documents is right on 15 October only because the change has not happened; on 5 November the three outdated pages still agree. The latest-edit rule follows the typo fix: right by accident, then wrong. Only the last row is correct on both dates.

That row needs two things: a superseding document that names what it replaces, and an effective date stored apart from the publication date. What to build:

• Store the replacement and its effective date with the superseding document.
• Express "current" as a predicate, not as a guess by the model, evaluated in code or in the database, with the result handed to the model:

valid_from <= asked_at and (valid_to is null or valid_to > asked_at)

• Keep future-dated documents. Serve a policy that takes effect later as announced, with its effective date, next to the rule that applies today.

Here asked_at is the moment the answer is for, valid_from the effective date and valid_to the date the document stops applying, null while it still applies. The check makes the question date an explicit input. A document with a future effective date should be neither discarded nor treated as current.

Why can a typo fix make an outdated page look current?

Because the date that moves is the modification date, and it does not describe the rule. Like schema.org's dateModified, HTTP dates the document: RFC 9110 says the Last-Modified header field "provides a timestamp indicating the date and time at which the origin server believes the selected representation was last modified". A typo fix is a modification; so is a rewrite, and the field does not say which. Zep's ingestion guide gives the same warning: "Do not use filesystem modification time unless it represents the source event."

Vohra keeps freshness as a fallback: "Where supersession isn't established, freshness should remain one of the signals considered alongside authority and specificity." That holds if freshness comes from the right date. Read from the modification date, the typo fixed on 10 October makes the old FAQ newer than the policy published on 1 October, so the freshness signal points to the outdated rule. On 5 November that is the wrong rule, delivered with a recent timestamp.

Take freshness from the effective date or a version number, and keep modification time as the history of the record. As a primary signal, the edit date is a trap.

Is every disagreement between sources an old version?

No. Ramm's reply names a third kind: "A wrong date and a changed decision are different things. But I found a third kind: different views held at the same time, like a business go live date and a technical go live date. Nobody is wrong there." His handling: "Changed decisions link to the newer source, and parallel views stay visible with their perspective."

So a pipeline needs at least three outcomes for two disagreeing sources:

• Replacement: one source supersedes the other, from its effective date.
• Parallel views: both stand, with their owners named.
• Undated pair: no date settles the order, so the pair goes to a person.

A list ordered by score has nowhere to put the second and third outcomes. Memory API Response Shape: Nowhere to Put a Disagreement sets out the different operations hidden in the word "update" and why a ranked list cannot express them.

What do memory systems record for this?

For example, from each system's own documentation, read on 17 October 2026.

Cognee

In its conflict review guide, each source's external_metadata "carries the date the source’s claims hold as of, under the key the review reads (effective_date by default, configurable with IMPROVE_EFFECTIVE_DATE_KEY)." Facts get conflict marks "that record each fact’s status within the conflict: current, superseded, or conflicting." The review is opt-in: "review_conflicts=True opts in to the review_conflicts stage of improve(), which is off by default".

Mem0

On Mem0 Platform, "The Memory Timestamps feature allows you to specify when a memory was created, regardless of when it’s actually added to the system." Temporal reasoning uses event dates, which "are when something described in a memory happened or will happen"; of that feature the same page says, "It is not available in the OSS SDK." Under Dream, "Superseded memories are not deleted and not hidden by default.", and latest_only=true "narrows the result to active memories only, meaning the current truth, with superseded and merged both excluded."

Mnemoverse

On the REST API, a write can pass supersedes to name one or more earlier records it replaces: "A target is not deleted; it gains a superseded_by pointer to the new atom and stays fetchable by ID". "The filter that hides a replaced revision from default reads is off by default." For that case the superseded_by field is the signal: "while filtering is off, this field is how you tell a replaced revision from a live one". Details: API reference, POST /memory/write.

Supermemory

A document can carry a date field: "When the source content is from, in ISO 8601 format". In its graph memory, "New fact replaces what was true before for search purposes; history can remain for audit." For contradictions, "updates win for “what’s true now.”"

Zep

Each fact carries timestamps such as valid_at, "The real world time that the fact started being true", and created_at, "The time when Zep learned the fact was true". For a published document, the recommended timestamp is "The publication or effective date". In graph construction, "contradictions invalidate older facts rather than leaving conflicting truths side by side." Of valid_at and invalid_at, Zep's facts page states the limit: "The fields do not establish that the source content or the derived fact is true."

How do I test my own pipeline for this?

Write the expected answer down before the run, as Vohra did with his labels: "We wrote down the right labels for each before running any model." Four small sets show which date a pipeline actually uses. Each is an illustration to build from your own documents, not a benchmark.

1. Supersession beats agreement. One official 14-day chunk that explicitly supersedes the old terms; three official 30-day chunks with older dates (the old terms, an FAQ, a support macro), still marked "current"; a forum post and a blog post. Expected: 14 days.
2. Published, not yet in effect. The same set, with the 14-day policy taking effect a month after the question date. Expected: 30 days, with a note that 14 days applies from the effective date.
3. The typo trap. As set 1, but the old FAQ's modification date is later than the new policy's publication date. Expected: 14 days.
4. Control. As set 1, but the 14-day chunk has no date. Expected: a flag that the pair is undated, not a confident answer.

Set 1 returning 30 days points to agreement still outweighing an explicit supersession. Set 2 returning 14 days points to the publication date being treated as the effective date. Set 3 returning 30 days points to freshness read from the modification date. Set 4 returning a confident answer means the pipeline is guessing.

If the expected answer cannot be written down, the pipeline is being asked to invent a policy.

What this page does not claim

The page is method, not a leaderboard. It gives no accuracy, latency or benchmark figure of our own. The numbers quoted come from one published test, small and on synthetic documents, with the limits its author states, and the without-source-details condition removed all four fields, not only the date. The systems named are examples, not a complete list, and none is ranked. Every quotation was read on 17 October 2026. The refund timeline and the test sets are illustrations, not data.

Common questions

Can Retrieval Tell Which Source to Trust?

Not on its own. Retrieval scores how related a passage is to the question, and an outdated policy page is highly related to a question about the current policy. Choosing a source also takes its type, its status and the right one of three dates. In one published test on synthetic documents, freshness judged from the text alone fell to between 0.51 and 0.72 AUC, where 0.50 is a coin flip.

How do I make sure my agent drops outdated instructions instead of getting confused?

Mark the outdated instruction superseded, store what replaces it and the date the replacement takes effect, then filter on that mark when the agent reads, keeping the old record as history. Closing a record with a validity window and a successor is covered in the library page on stale instructions.

Which date decides whether a document is current?

The effective date, together with the date the document stops applying. The publication date orders versions, and a policy can be published weeks before it takes effect. The last-modified date moves when someone fixes a typo and says nothing about when the rule changed.

Should several older documents that agree outweigh one newer document?

Not when the newer document explicitly supersedes them and is already in effect. Copies of one outdated text share an origin, so their number is not independent evidence. In one published test, under a natural phrasing of the contradiction question, several systems marked the correct document as contradicting because three other documents disagreed with it.

What should an agent answer when a new policy is published but not yet in effect?

The rule in effect on the day of the question, with the announced change and its date: for example, 30 days, and 14 days from 1 November. A document with a future effective date should be neither discarded nor treated as current.

Why can a small edit make an outdated page look current?

Because a modification date records when the page last changed, not when its rule changed. HTTP defines Last-Modified as the time the origin server believes the representation was last modified. If freshness is read from that date, fixing a typo in an old FAQ makes it look newer than a policy published after it. Take freshness from the effective date or a version number.

Sources

Every page below was read on 17 October 2026. Each quotation on this page is contiguous text of the page it comes from.

Practitioners

• Manoj Kumar Vohra, Relevant Is Not the Same as Right: Can Retrieval Tell Which Source to Trust?, Medium, 9 October 2026: the question, the refund scenario and every measurement quoted here.
• Manoj Kumar Vohra, reply of 9 October 2026, Medium, to a comment by this page's author: explicit supersession and time-aware contradiction resolution.
• Christopher Ramm, reply of 7 October 2026, Medium, to a comment by this page's author on his article about a notes vault Claude Code maintains: undated documents and parallel views.

Memory systems

• Cognee: Review conflicts.
• Mem0: Memory timestamps; Temporal reasoning; Dream; the documentation index llms.txt, which marks the pages that describe Mem0 Platform.
• Supermemory: Graph memory; Add document, API v5.
• Zep (documentation v4): Searching the graph; Prepare data for ingestion; How graph creation works; Facts.

Standards

• Schema.org (V30.1, 2026-09-16): datePublished; dateModified; validFrom; legislationDateOfApplicability.
• RFC 9110, HTTP Semantics, section 8.8.2, Last-Modified.

Mnemoverse

• API reference, POST /memory/write: supersede semantics and the read-side filter.

Related

• Memory API Response Shape: Nowhere to Put a Disagreement
• Who Wins When Agents Disagree? The Authority Problem
• The Independence Illusion: When AI Agents Agree
• Closing stale instructions with a validity window and a successor
• Why does my LLM agent keep suggesting outdated library patterns after being corrected?
• AI assistant keeps giving outdated answers after codebase refactoring, how to update its learned knowledge?
• Bitemporal Memory for AI Agents: The Missing Axis
• Agent Memory Consolidation Compared

Edward Izgorodin · Mnemoverse · 2026-10-16

Mnemoverse is a persistent-memory API for AI agents. Free key: console.mnemoverse.com · Plans and limits · Docs: Getting Started