LoCoMo — Long-Term Conversational Memory
LoCoMo (Maharana et al., ACL 2024) is the de-facto standard for cross-session conversational memory and the headline benchmark vendors compete on. It asks whether a system can remember and reason over facts spread across long multi-session dialogues — single-hop recall, multi-hop chains, temporal ordering, and open-domain questions.
This is the canonical LoCoMo result: the full 10-conversation public release (1,986 questions — not the 50-conversation paper corpus), bench-v1, locked 2026-09-02. Per-cell judge prompts, dataset hashes, and the forensic vendor write-ups behind the retractions below live in our internal benchmarks registry. The earlier single-conversation matrix (k-sweep, four judges) is superseded by this page and kept for reference at the June 2026 archive. For where LoCoMo sits among all memory benchmarks — and its known answer-key problems — see the AI Memory Benchmarks Field Guide.
TL;DR
- bench-v1 (locked 2026-09-02) runs the full 10-conversation public LoCoMo release — 1,986 questions, all five categories — not the 50-conversation paper corpus, and not a slice.
- Two axes, always labeled, with two different denominators: the LLM-judge axes score all 1,986 questions; recall@k is computable only for the 1,536 questions that carry an evidence label (1,540 questions are non-adversarial, categories 1–4; four of them have no evidence label in the dataset); the adversarial category is excluded from this axis by protocol.
- Our canonical headline is the stock (shipping-default) run; tuned-B (
PRIORS_BEFORE_CUT=false, disclosed) is a labelled tuned-frontier ceiling, never the headline. Tuning closes most of the gap to our own naive-cosine floor, but we do not yet beat it at 95% confidence — we say so plainly.- Vendor rows are dated: supermemory is a valid same-scale measurement; zep's and mem0's earlier public figures are retracted, and mem0's full-corpus run isn't measurable until its quota resets.
What it tests
LoCoMo is a conversational-memory benchmark over machine-generated, persona-grounded dialogues with temporal event graphs. Questions span five categories: single-hop (recall one fact), multi-hop (chain facts across sessions), temporal (when/order), open-domain (world knowledge plus the conversation), and an adversarial category-5 set (unanswerable / should-abstain). The original paper scored answers with token-overlap F1 and ROUGE/FActScore; most downstream work — including ours — now uses an LLM-as-judge for semantic correctness, which is exactly where scores become judge-dependent.
Dataset
10 conversations, 369–689 turns each (mean 588) and 9K–26K tokens each, up to ~35 sessions; 1,986 QA pairs in total, across all five question categories. This is the full 10-conversation public LoCoMo release, not the paper's own 50-conversation / 7,512-question full corpus. bench-v1 evaluates all of it — every conversation, every category, not a single-conversation slice, unlike the archived June 2026 matrix, which scored one conversation at a time and excluded the adversarial category. The conversations fit inside a modern context window, so LoCoMo does not cleanly separate a memory system from a long-context LLM — a caveat we carry on every LoCoMo number.
Our protocol
We score answers on two axes, always reported together and labeled:
- recall@k (k=10) — a mechanical, judge-free check: did the system's top-k retrieval actually surface the memory the answer key requires? No LLM in the loop, nothing to inflate. Recall has a smaller denominator by construction: it is scored over the 1,536 questions that carry an evidence label — the 1,540 non-adversarial questions (categories 1–4) minus four that have no evidence label in the dataset; the adversarial category asks about things that are not in the conversation and is excluded from this axis by protocol (the manifest records this as
n_scored). - LLM-judge (mnemoverse) — our in-house grader returns a binary CORRECT/WRONG verdict per question, over all 1,986 questions including the adversarial category; a cell score is the fraction marked correct. We also record a second, harsher grader, LLM-judge (strict), for contrast.
Judge prompts, the dataset hash, and the run configuration are frozen under bench-v1, locked 2026-09-02 — see How We Measure AI Memory Honestly for the comparability-key discipline behind that lock. The public comparability key — engine git SHA, dataset hash, judge-prompt hashes, and the embedding model actually served — is published as the bench-v1 manifest.
Results — full 10-conversation release (bench-v1)
Two configurations of our own engine, our naive-cosine floor, and three vendors — all on the same 10-conversation release unless noted. Denominators per axis: LLM-judge columns score all 1,986 questions; the recall@k column scores the 1,536 questions with an evidence label (see above). Stock is the canonical headline (shipping defaults, regression paradigm); tuned-B is the disclosed tuned-frontier ceiling.
| System | recall@k (k=10) | LLM-judge (mnemoverse) | LLM-judge (strict) |
|---|---|---|---|
| Mnemoverse (stock) | 0.6316 | 0.5468 | 0.3384 |
| Mnemoverse (tuned-B) | 0.6939 | 0.5826 | 0.3706 |
| naive-cosine (floor) | 0.7138 | 0.5695 | 0.3595 |
| supermemory | — | 0.5181 | 0.3066 |
| zep | — | 0.2956 | 0.1596 |
| mem0 (conv-26 only, n=199) | — | 0.5075 | 0.2613 |
naive-cosine is the row name we use for the floor here — the archived June 2026 matrix labels the same baseline method (plain cosine similarity, no learning) naked_cosine; the name changed, the method didn't.
recall@k also has a stricter reading — delivered-context recall, checked against what the reader actually received after retrieval post-processing rather than the raw top-k set: stock 0.6746, tuned-B 0.7417, floor 0.7138 (identical to recall@k for the floor, which has no post-processing step to diverge from). The precise definition lives with the raw cells in the registry; we surface both numbers here rather than pick the more flattering one.
tuned-B is our disclosed tuning config: PRIORS_BEFORE_CUT=false — feedback priors influence ranking within the retrieved set but do not decide top-k membership. It is not a hidden ceiling; the flag and its effect are public.
mem0's full-corpus number is not available before 2026-10-01 — their free-tier quota was exhausted mid-measurement. The conv-26 figure above (n=199) is the only valid mem0 comparison point until then, dated as such rather than substituted with an estimate.
How to read this table
- Recall and judge are different questions. recall@k asks whether the engine found the right memory; the judge score additionally grades whether the written answer reads as correct, so it inherits the grader's leniency. Don't average them — read both, labeled.
- We publish our own floor. naive-cosine is not a competitor's number to beat — it's a self-critical baseline we compute and publish ourselves. If tuning can't clear it, that's a finding worth reading, not a result to bury.
- The honest verdict on tuning. Paired bootstrap over the same questions (LLM-judge, mnemoverse grader): tuned-B beats stock by +3.58pp (CI [+1.81, +5.39] — significant), and closes to within +1.31pp of the floor (CI [−0.60, +3.22] — the interval crosses zero). Stock alone sits −2.27pp below the floor (CI [−4.33, −0.15] — significant). Plainly: tuning closed the gap to the floor; we do not yet claim beating naive cosine at 95% confidence.
Same answers, four graders
Published LoCoMo headlines — Mem0 92.5% (memory-benchmarks README), Zep 94.7% (getzep.com/research), MemMachine 91.7% (memmachine.ai), Backboard 90.1% (news.backboard.io) — are each produced under their own grading recipe, and none of those recipes is the one this page's headline uses (the tables here are 0–1 fractions; 0.698 reads as 69.8%). The two generous columns below are Mem0's published judge prompt, which marks an answer CORRECT if it contains any one item of a multi-part gold answer and counts paraphrases as matches — the only vendor grader published in full, which is why we re-graded our own answers with it. Zep grades with GPT-5.4 and does not publish its prompt; MemMachine grades with GPT-4.1-mini, Backboard with GPT-4.1. Our headline grader requires the full answer and is strict about dates and entities; our adversarial grader marks any partial answer WRONG. Read a row left to right and you see how much of a score is the judge and how much is the system.
| System | mem0-4o (generous, GPT-4o) | mem0 (generous, GPT-5) | mnemoverse (ours) | strict (adversarial) |
|---|---|---|---|---|
| Mnemoverse (tuned-B) | 0.6979 | 0.6309 | 0.5826 | 0.3706 |
| naive-cosine (floor) | 0.6858 | 0.6193 | 0.5695 | 0.3595 |
| Mnemoverse (stock) | 0.6566 | 0.5972 | 0.5468 | 0.3384 |
| supermemory | 0.6083 | 0.5629 | 0.5181 | 0.3066 |
| mem0 (conv-26 only, n=199) | 0.6583 | 0.5829 | 0.5075 | 0.2613 |
| zep | 0.4230 | 0.3756 | 0.2956 | 0.1596 |
Same 1,986 questions, same saved answers, four gradings — the generous columns were produced by re-judging the stored answers of every run behind this table — our 30 registered runs and the 21 vendor cells alike — with Mem0's verbatim prompt on GPT-4o and GPT-5 (the manifest records each grader's model, prompt hash and, for the generous pair, the upstream prompt source under spec.judges). One asymmetry closed on the way: the generous prompts skip an empty answer while ours scores it WRONG, so an empty answer now counts 0 under every grader — otherwise a system that answers nothing would get a smaller denominator and a flattering mean.
Why this table reads 0.55 and vendor pages read 0.92
In May 2026 we ran one LoCoMo conversation the way the published headlines are produced: GPT-5 as the reader, 200 memories in its context, Mem0's grading prompt on GPT-5, the adversarial category left out. That run — LCM-ANCHOR2, 2026-05-21, conversation 26, 152 questions, an earlier engine build with a Qwen3-Embedding-0.6B embedder — scored 0.9013 under Mem0's grader: the same neighbourhood as Mem0's published 92.5% and the 91.56% in the results file they committed. The identical answers read 0.4803 under our strict grader and 0.7566 under our in-house one (rejudge_20260521_235650.json). That number has been on this site since June, in Judges, Good & Evil, as our own example of the soft metric; the k-curve there is this run. Having checked that we land where the headlines land, we changed the recipe on purpose. The table walks each change on that same conversation, with the naive-cosine floor — a plain embedding search, no memory system — beside every point where we ran it.
| Recipe step | GPT-5-mini reader, current build (July 2026) — stock / floor | GPT-5 reader, earlier build (May 2026) | recall@k (May) |
|---|---|---|---|
| Our grader, all 199 questions, 10 memories to the reader | 0.5327 / 0.5075 | — | — |
| Same answers, Mem0's grader prompt on GPT-5, all 199 | 0.6935 / 0.6080 | — | — |
| Drop the 47 adversarial questions (152 from here down), 10 memories | 0.7368 / 0.6382 | 0.7697 | 0.669 |
| 20 memories | 0.7039 / 0.7039 | 0.8289 | 0.763 |
| 50 memories | 0.7500 / 0.7434 | 0.8618 | 0.840 |
| 100 memories | 0.7829 / 0.7500 | 0.8750 | 0.883 |
| 200 memories — the budget Mem0's harness runs at | 0.7763 / 0.7566 | 0.9013 | 0.927 |
| 200 memories, graded by GPT-4o instead of GPT-5 | 0.8355 / 0.8224 | 0.9145 | — |
Read down the July column (GPT-5-mini reader, stock): the grader is worth 16 points, the question subset 4, the depth another 4 — and the floor climbs with us at every step, trailing by at most ten points and, at 20 memories, by nothing at all. In the July column every rung is a recipe field, not the engine; the rows are single passes, so neighbouring depths move within noise (100 and 200 memories swap order). Read across: with GPT-5 as the reader, the May build scores 3 points higher at 10 memories and 12.5 at 200. That column changes the reader and the build together — a different embedder and retrieval stack, which is why it carries recall@k — so the table does not isolate the reader: those are the combined effect of both changes, and the two series are not one ladder. Everything on this page is produced with the same 547-character answer prompt (locomo/evaluate.py); Mem0's is 7,055 characters of seven-step instruction, and both tell the reader never to refuse. What the table does not contain: that prompt difference, and whatever runs inside a managed pipeline that, in the vendor's own words, "includes proprietary optimizations not available in the open-source SDK". One caveat we owe the reader: this is one conversation of ten, and one pass of it. The registered August pass of the same cell reads 0.6645 at the third row against this July pass's 0.7368, with the 1,540-question mean of that registered run at 0.6825 — single passes move by several points, and the ladder's absolute level belongs to one conversation and one pass, not to the corpus. We have not run the other nine conversations this way; the full-corpus canon at the top of this page, all 1,986 questions under our grader, is 0.5468, and on this one conversation the registered cell reads 0.54.
Why we do not report this recipe. Three reasons, all visible above. A plain embedding search with no memory system climbs every rung with us and never falls off the ladder, and on the full corpus at our own depth it is 2.3 points ahead of stock: the climb measures the recipe, not the memory. At 200 retrieved memories — one dialogue turn each, in this harness — nearly half of conversation 26 is in the reader's context (200 of its 419 turns) — and Mem0's own paper reports a full-context baseline (72.9%) beating every memory system it tested (66.9%). And an independent audit found the standard grader accepts 62.8% of deliberately wrong answers, which puts an effective ceiling near 93.6% — where the published cluster sits. So the table at the top of this page is our number, for ourselves and for comparing others; the manifest publishes every recipe field and the four grader prompts by hash for the registered k=10 canon; the ladder cells and the May run are cited above by exact file and are not yet in the registry.
Where a published number sits
Every LoCoMo headline is the output of a recipe. The rows below record the fields each source states; a dash means the cited page does not state it.
| Claim | Grader | Reader | Memories to reader | Questions | Runs | Source |
|---|---|---|---|---|---|---|
| Mnemoverse stock, bench-v1 canon: 0.5468 | mnemoverse (in-house, GPT-5-mini) | GPT-5-mini | 10 | 1,986, all five categories | 1 | manifest |
| Mnemoverse stock, Mem0's grader, non-adversarial: 0.6825 (floor 0.7117) | Mem0 prompt, GPT-5 | GPT-5-mini | 10 | 1,540 | 1 | same saved answers, re-graded (this page) |
| Mnemoverse, run the published way (May 2026, conversation 26 only): 0.9013 | Mem0 prompt, GPT-5 | GPT-5 | 200 | 152 (conversation 26, four categories) | 1 | run LCM-ANCHOR2, re-judge rejudge_20260521_235650.json; k-curve in Judges, Good & Evil (this page, table above) |
| Mem0 92.5% | Mem0 prompt, GPT-5 | GPT-5 | 200 | 1,540 | — | memory-benchmarks README (1,425/1,540 at top-200; 91.8% at top-50); the committed results file at memory-benchmarks@4b61c5d records 1,410 of 1,540 correct (91.6%) with answerer_model: gpt-5, judge_model: gpt-5, top_200 |
| Mem0 66.9% (paper, April 2025) | "a separate, more capable LLM" — not named | GPT-4o-mini (the paper's inference engine) | — | four categories; count not stated | 10, temperature 0 | arXiv:2504.19413 |
| Full-context baseline 72.9% (same paper) | as above | — (the whole conversation in context; model not named for this row) | all | four categories; count not stated | 10 | arXiv:2504.19413 |
| Zep 75.14% | GPT-4o-mini | GPT-4o-mini | 20 nodes + 20 edges | 1,540 | — | score: blog.getzep.com; recipe fields: the evaluation code that post links |
| Zep 94.7% (1,459/1,540) | GPT-5.4, chain-of-thought grading; prompt not published | GPT-5.4 (reasoning: medium) | 20 edges + 10 nodes + 10 episodes + 5 thread summaries + 5 observations, cross-encoder reranked | 1,540 | — | getzep.com/research (2026-08-27) |
| MemMachine 91.7% (0.9169, agent mode) | GPT-4.1-mini | GPT-4.1-mini | — | 1,540, four categories | — | memmachine.ai (2025-12-01) |
| Backboard 90.1% | GPT-4.1 | — | — | "the standard task set" | — | news.backboard.io (2025-11-03) |
| Hindsight 89.61% | — | — (the abstract reaches 89.61% by "scaling the backbone further"; its 20B open-source model's own figure is 83.6%, a combined LongMemEval + LoCoMo accuracy) | — | — | — | arXiv:2512.12818 |
| Human answer-F1 ceiling 87.9% (LoCoMo paper) | none — token-overlap F1 | humans | — | — | — | arXiv:2402.17753 |
| Grader ceiling ≈93.6% | GPT-4o-mini accepts 62.8% of wrong answers | — | — | 1,540 | — | Penfield Labs audit (April 2026, one team, not peer-reviewed) |
Vendor rows, dated
Benchmark rows age, because the services behind them change. What we show here and why:
- supermemory — valid: same corpus, same judge protocol, no caveat.
- zep — the row above is a fresh bench-v1 measurement. An earlier zep figure we had held from a June 2026 run is retracted and deliberately not restated here: that run returned recall@k of 0.0 in every category, which is the signature of a harness that retrieved nothing rather than a measurement of zep, and it was recorded on 2026-06-11, after zep's 2026-06-09 Batch API deprecation broke the integration our old harness used. Quoting the number would attribute an integration failure of ours to their system.
- mem0 — the number we previously held for mem0 was measured against an under-materialized store and is retracted. The conv-26 figure above is the current, valid one; the full-corpus run is blocked until the quota resets (2026-10-01).
The public bench-v1 manifest carries the comparability key and per-run scores for all 30 runs; full per-cell verdicts and the forensic write-up behind each retraction live in the benchmarks repository — its raw cells are private today, opening under consideration.
API instability is a parameter — including ours
Every row in this table depends on a vendor API that changes underneath it, and that cuts both ways: we are not exempt. Our own retrieval contract changed its top_k handling under a stricter validation path on 2026-03-09 — a run against the old behavior and a run against the new one are not the same cell, exactly like zep's Batch deprecation above. Any LoCoMo number, ours or a competitor's, carries an implicit "as of" date on the API it was measured against, not just on the dataset.
Where the June 2026 matrix went
The earlier card scored one conversation at a time (conv-26, then conv-47) across a k-sweep and four judges, from a lenient-judge-era harness. It is archived, not deleted, with its own imperfections disclosed: see the June 2026 matrix. bench-v1 above is the canonical number going forward.
Common questions
What does LoCoMo measure?
LoCoMo is a conversational-memory benchmark over machine-generated, persona-grounded dialogues with temporal event graphs. Questions span five categories: single-hop, multi-hop, temporal, open-domain, and an adversarial should-abstain set.
How much of LoCoMo does Mnemoverse's bench-v1 cover?
The full 10-conversation public LoCoMo release — all five categories, 1,986 QA pairs — not a single-conversation slice, and not the LoCoMo paper's own 50-conversation / 7,512-question full corpus. bench-v1 was locked 2026-09-02. The earlier conv-26/conv-47 single-conversation matrix is archived, not current.
What is the difference between recall@k and the LLM-judge score?
recall@k is judge-free and answer-key-checked — did the system's retrieval actually surface the right memory. The LLM-judge score additionally grades whether the written answer reads as correct, so it also carries the grader's leniency. We report both, always labeled, and lead with recall.
Does tuning beat the naive-cosine floor?
Partially, and we say so plainly. Paired bootstrap over the same questions: tuned-B beats stock by +3.58pp (significant) and closes to within +1.31pp of our own naive-cosine floor — but that gap's confidence interval crosses zero, so it is not yet significant at 95%. Stock alone sits significantly below the floor. We do not yet claim beating naive cosine.
Are the vendor rows (supermemory, zep, mem0) comparable?
supermemory's row is a valid same-corpus, same-judge measurement. zep's and mem0's earlier public figures are retracted — zep's 2026-06-09 Batch API deprecation broke the integration our old harness used, and mem0's prior number came from an under-materialized store. mem0's full-corpus run is blocked until its free-tier quota resets (2026-10-01), so we show its conv-26 (n=199) figure instead, clearly scoped.
Where did the old conv-26/conv-47 matrix go?
Archived, not deleted — see the June 2026 matrix for the k-sweep, four-judge view, and its own disclosed imperfections. bench-v1 above supersedes it as the canonical LoCoMo number.
Links
- Live, cell-by-cell matrix: benchmarks.mnemoverse.com — serves bench-v1 as its default scope since 2026-09-04, with the four-grader "By judge" view since 2026-09-05; the June 2026 grid remains selectable as a historical scope.
- Public manifest: bench-v1-manifest.json — run ids, hashes, per-run scores; raw cells live in the benchmarks repository (private today, opening under consideration).
- Benchmarks overview — the hub and the other benchmark cards.
- June 2026 matrix (archived) — the retired conv-26/conv-47 k-sweep, kept for reference.
Related
- AI Memory Benchmarks: A Field Guide — where LoCoMo sits among all memory benchmarks, and its answer-key problems.
- Judges, Good and Evil — why a single LoCoMo number depends on the grader.
- How We Measure AI Memory Honestly — the discipline behind these numbers.
- Building Memory That Scales — the engineering story behind the engine these numbers measure, including the 2026-09-03 erratum on the numbers it originally quoted.
— Mnemoverse is a persistent-memory API for AI agents. Free key: console.mnemoverse.com · Docs: Getting Started