Archived (2026-09-03) — superseded by bench-v1
This page is the retired June 2026 LoCoMo matrix (conv-26, then the fairer conv-47 sweep), kept for reference and reproducibility, not as a current result. The canonical LoCoMo number is now the full-corpus bench-v1 result (n=1,986, locked 2026-09-02).
Known imperfections in what follows, disclosed rather than quietly dropped:
- Lenient-judge era, n=152. This matrix predates the strict-judge protocol used in bench-v1, and the adversarial category (category 5) was excluded rather than scored — the question count here (152) is not the full 199-question conv-26 set.
- Subset mixing. "The matrix" below is described as conv-26 throughout, but the individual manifests behind these numbers mix conv-26 and the later conv-47 run inconsistently — exactly the dataset-slice confound the bench-v1 protocol was built to close.
- An embedding confound in the Mnemoverse API row. The
k=10cell for the Mnemoverse API configuration ran againstnv-embedqa@1024, not thetext-embedding-3-smallthe row's documentation claims — tracked as debt-registry item C1. Treat that cell as unverified against its stated config. - Vendor rows forensically refuted. The mem0, supermemory, and zep figures below were re-examined during the 2026-09 bench-v1 audit; the mem0 and zep numbers from that era do not hold up (see the vendor-validity notes on the bench-v1 card). Read every competitor figure on this page as unverified by that later audit, not as a live comparison.
LoCoMo — June 2026 Matrix (Archived)
LoCoMo (Maharana et al., ACL 2024) is the de-facto standard for cross-session conversational memory and the headline benchmark vendors compete on. It asks whether a system can remember and reason over facts spread across long multi-session dialogues — single-hop recall, multi-hop chains, temporal ordering, and open-domain questions.
This is a short card. The full run catalog, per-cell judge prompts, dataset hashes, and the cross-provider protocol lived in the interactive dashboard at the time. For where LoCoMo sits among all memory benchmarks — and its known answer-key problems — see the AI Memory Benchmarks Field Guide.
TL;DR
- LoCoMo is the de-facto standard for cross-session conversational memory — the headline benchmark vendors compete on.
- It tests five question categories: single-hop, multi-hop, temporal, open-domain, and an adversarial should-abstain set.
- Our results are a cross-system matrix (one conversation, conv-26, 152 questions) scored by four LLM judges across retrieval depths k=10–200; we headlined the strict in-house judge.
- The judge swings these scores by tens of points, so read the leader off the full matrix — the fairer symmetric conv-47 run was where the lead changed hands.
What it tests
LoCoMo is a conversational-memory benchmark over machine-generated, persona-grounded dialogues with temporal event graphs. Questions span five categories: single-hop (recall one fact), multi-hop (chain facts across sessions), temporal (when/order), open-domain (world knowledge plus the conversation), and an adversarial category-5 set (unanswerable / should-abstain). The original paper scored answers with token-overlap F1 and ROUGE/FActScore; most downstream work — including ours — now uses an LLM-as-judge for semantic correctness, which is exactly where scores become judge-dependent.
Dataset
10 conversations, roughly 300 turns and 9K–26K tokens each, up to ~35 sessions; about 1,986 QA pairs, of which ~1,540 are the non-adversarial subset conventionally reported (the rest are category-5 adversarial). The conversations fit inside a modern context window, so LoCoMo does not cleanly separate a memory system from a long-context LLM — a caveat we carry on every LoCoMo number.
Our protocol
We scored answers with LLM-as-judge prompts that return a binary CORRECT / WRONG verdict per question; a cell score is the fraction of questions a judge marked correct. Reader, embedding model, judge prompts, and conversation were held identical across every cell of a matrix — but given the subset mixing and the C1 embedding confound disclosed above, movements in this table cannot be attributed solely to system or k.
Our results — the cross-system matrix
One LoCoMo conversation (conv-26), 152 questions after symmetric removal of the adversarial category (originally 199), evaluated with four LLM judges. Scores below are on the strict in-house judge (our dedicated strict-grader is harsher still; note that this June-era "strict" grader predates and differs from the bench-v1 LLM-judge (strict) grader on the canonical card). k is the retrieval top-k passed to the reader; higher is better.
| System | k=10 | k=20 | k=50 | k=100 | k=200 |
|---|---|---|---|---|---|
| Mnemoverse (research prototype) | 0.691 | 0.684 | 0.711 | 0.664 | 0.638 |
| Mnemoverse (API) | 0.296 | 0.362 | 0.408 | 0.526 | 0.539 |
| mem0 (cloud) | 0.572 | 0.625 | 0.625 | 0.664 | 0.664 |
| supermemory (cloud) | 0.342 | 0.421 | 0.651 | 0.638 | 0.645 |
| zep (cloud) | 0.217 | 0.355 | 0.480 | — | — |
| naked_cosine (floor) | 0.592 | 0.599 | 0.632 | 0.664 | 0.704 |
Two Mnemoverse rows appear because both configurations are real:
- Research prototype — the internal engine evaluation (in-process call into
MemoryEngine), full algorithmic stack,strategy=Nonewith engine-config defaults. - API — the version available at the time at
core.mnemoverse.com(thinner request/response contract), invoked withtwo_pass=Trueandstrategy="auto", routing throughStrategyClassifieron the query text only (noqa.category, evidence, ordia_idleakage — verified by grep on the adapter).
The spread between the two rows is engineering work in flight, not a fixed gap: as features move from the research path into the public API, the API row converges toward the prototype — though given the subset mixing and the C1 embedding confound disclosed above, movements in this table cannot be attributed solely to system or k. The em-dashes on the zep row mark cells where every query returned empty; the bench-v1 audit (2026-09) traced this to zep's Batch-API code path, the same path zep deprecated on 2026-06-09 — not a quota issue (that is the mem0 story, see the canon card). We published a single contiguous sweep under identical conditions rather than splice in an earlier run.
The sixth row, naked_cosine, is the floor: plain cosine similarity over the same embeddings — no learning, no graph, no reranking. It was competitive, and at k=200 (0.704) it edged past our research prototype (0.638). That is not a knock on cosine; it is the reminder that one high LoCoMo number does not by itself prove sophisticated memory. What separates systems is how they behave across k, across judges, and on the harder operations — temporal reasoning, multi-hop, contradiction.
For most tasks a good vector store already takes you a long way, and we would rather say so than pretend otherwise. A memory engine earns its place precisely where a similarity index cannot fake it: those harder operations, and holding a lifetime of history at 10M-token scale. Pick your benchmark by the operations your agent actually needs.
Why this card stayed on conv-26 — and where it led. This table is one conversation (conv-26, 152 questions) on our strict in-house judge: a real, reproducible result, and the slice where our engine led. Like any vendor, we were showing a result that flatters us — which is exactly why it was never meant to be the last thing you read. The fairer symmetric conv-47 run (190 questions, adversarial category included, and our competitor-adapter bug fixed) carried the lead changing hands. The judge alone swings these scores by tens of points.
Multi-judge view
The same 152 questions were also scored by mem0, gpt-4o-as-judge, and a strict-grader variant:
| Judge | Tendency |
|---|---|
| Mnemoverse (in-house) | strict — off-by-one dates, partial enumerations, and lost qualifiers fail |
| mem0 | lenient — Mem0's "be generous, same topic = correct" rubric; accepts paraphrases and partial credit our judge rejects |
gpt-4o | lenient — passes partial answers and date drift our judge rejects |
| strict-grader | very strict — penalises any token outside ground truth |
We published the strict in-house judge as the headline; the others were recorded per cell for reproducibility but inflate by ~5–20 points on the same answers, mostly by accepting wrong dates or half-recalled lists. Why a single LoCoMo number is fragile — and how a prompt swap alone moves it ~40 points — is the subject of Judges, Good and Evil.
Known asymmetries (disclosed)
Per our methodology, we disclose measurement asymmetries rather than strip them to manufacture parity. After the PR #290 ASYM-024 closure (HTTP adapter now sends two_pass + strategy symmetric with the in-process baseline), an asym=0.25 residual gap persisted and its direction favoured the API (HTTP) row: the classifier preset (PPR / gap_filling / entity_chain) could engage on HTTP but not on the strategy=None in-process baseline. Documented algorithm advantages on the Mnemoverse side (Hebbian feedback, two-pass, reranker, μ=0 ingest, between-session consolidation, edge pruning) were kept and disclosed, not hidden.
Reproducibility
All cells (six systems × five k) — with judge prompts, dataset hash, judge-prompt hashes, embedding model, reader model, post-filter question count, and the pinned config.git_sha — were committed as JSON under experiments/benchmarks/matrix/cells/ in mnemoverse-core, alongside BASELINE_FROZEN_2026-06-08.md. The HTTP (API) row required a core deploy exposing POST /api/v1/memory/read-batch accepting two_pass / strategy (post-PR #290); older deploys reproduce the pre-closure asymmetry.
Common questions
What does LoCoMo measure?
LoCoMo is a conversational-memory benchmark over machine-generated, persona-grounded dialogues with temporal event graphs. Questions span five categories: single-hop, multi-hop, temporal, open-domain, and an adversarial should-abstain set.
How many questions are in LoCoMo?
The dataset has about 1,986 QA pairs across 10 conversations, of which ~1,540 are the non-adversarial subset conventionally reported. This archived matrix uses one conversation (conv-26), 152 questions after removing the adversarial category.
Why does this archived card report only one LoCoMo conversation (conv-26)?
conv-26 is a real, reproducible result and the slice where our engine led — but like any vendor we were showing a flattering result, which is why this matrix also carries the fairer symmetric conv-47 run (190 questions), where the lead changed hands. bench-v1 supersedes both with the full 10-conversation public release.
Why are there two Mnemoverse rows in the LoCoMo matrix?
Both configurations are real: the research prototype is the internal engine evaluation (full algorithmic stack), and the API is the version available at the time at core.mnemoverse.com with a thinner contract. As features migrate from the research path into the public API, the API row converges toward the prototype.
Does the LLM judge change the LoCoMo score?
Yes — profoundly. The same 152 answers scored by different judges diverge by ~5–20 points; the lenient judges (mem0, gpt-4o) accept paraphrases and date drift the strict in-house judge rejects. We published the strict in-house judge as the headline.
Links
- Current canonical result: LoCoMo — full-corpus bench-v1
- Benchmarks overview — the hub and the other benchmark cards.
Related
- LoCoMo — full-corpus bench-v1 — the current canonical result superseding this archived matrix.
- AI Memory Benchmarks: A Field Guide — where LoCoMo sits among all memory benchmarks, and its answer-key problems.
- Judges, Good and Evil — why a single LoCoMo number depends on the grader.
- How We Measure AI Memory Honestly — the discipline behind these numbers.
- Building Memory That Scales — the engineering story behind the engine these numbers measure.
— Mnemoverse is a persistent-memory API for AI agents. Free key: console.mnemoverse.com · Docs: Getting Started