MnemoverseDocs

Appearance

Edward Izgorodin·Published May 18, 2026Updated Sep 3, 2026

Benchmarks ​

Evaluation results from the Mnemoverse memory engine on established benchmarks. All runs are automated, configurations are documented, and raw results are committed as JSON in the repository.

Interactive dashboard: benchmarks.mnemoverse.com — live leaderboard with detailed per-question breakdowns.

TL;DR

  • LoCoMo is our primary, fully-committed cross-system matrix; BEAM, HotpotQA, MuSiQue, and LongMemEval each test a different memory operation.
  • Every result carries its config, date, and limits — a score is a property of the test rig as much as of the system under test.
  • Most of these benchmarks are graded by an LLM-as-judge; judge model, judge prompt, dataset subset, and reader backbone can each swing a score by double digits.
  • We report judge-free recall@k alongside two labeled LLM-judge graders — LLM-judge (mnemoverse) and LLM-judge (strict) — read every number next to the exact rig that produced it.

How we read a memory benchmark ​

Memory evaluation is not one number. A score is a property of the test rig as much as of the system under test, so we report each result with its config, its date, and its limits attached. Three things shape how we read every benchmark — ours and everyone else's. The full treatment is in How to Evaluate AI Agent Memory; the short version:

  • Direct vs indirect. Most public memory scores — including ours below — are direct recall: does the store hold the right thing and surface it? That is distinct from the indirect question (does memory make an agent better at its actual task), which is barely measured by any public benchmark. We benchmark the direct path and say so.
  • Write → manage → read. Memory is a loop: filter and canonicalize on write; deduplicate, consolidate, and resolve contradictions on manage; retrieve and use on read. Each stage fails on its own. Recall benchmarks mostly probe read; the manage operations (consolidation, forgetting, contradiction resolution) have no widely-adopted public benchmark, so we flag where a number does not cover them.
  • The judge and the harness move the score. Most of these benchmarks are graded by an LLM-as-judge, which is lenient by construction — see The Judge Says Yes Too Easily. Judge model, judge prompt, dataset subset, and reader backbone can each swing a score by double digits, and one benchmark name can point at two different datasets. So a number is only meaningful next to the exact rig that produced it. That is why every score on this page carries its grader label — LLM-judge (mnemoverse) or LLM-judge (strict) — alongside judge-free recall@k; the retired four-judge comparison lives in the archived June 2026 matrix.

LoCoMo (Primary Benchmark) ​

LoCoMo (Maharana et al., ACL 2024) evaluates conversational memory across multi-session dialogues. The dataset labels five question categories: single-hop factual recall, temporal reasoning, multi-hop inference, open-domain knowledge, and an adversarial category designed to trigger hallucinations. bench-v1 scores all five categories, adversarial included (n=1,986). Excluding the adversarial category for cross-system symmetry was a property of the archived June 2026 matrix, not of bench-v1 — the current matrix lives on the LoCoMo benchmark card.

Note on "LoCoMo." The benchmark name points at two different datasets: the full paper corpus is 50 conversations / 7,512 questions, while the industry-standard vendor subset — what most vendors run, ours included — is the full 10-conversation public LoCoMo release (n=1,986, all five categories). The matrix on the card now covers that full public release (bench-v1, locked 2026-09-02) rather than a single conversation. Read any "LoCoMo 90+%" claim by asking which LoCoMo.

Headline. On the full 10-conversation public LoCoMo release (bench-v1, locked 2026-09-02), our canonical headline is the stock run — shipping defaults, the regression paradigm our methodology commits to: LLM-judge (mnemoverse) 0.5468 over all 1,986 questions (five categories, adversarial included) and recall@k 0.6316 over the 1,536 questions that carry an evidence label (the 1,540 non-adversarial questions, categories 1–4, minus four without a label; the adversarial category is excluded from this axis by protocol). Alongside it, labelled and never the headline: the disclosed tuned-frontier ceiling tuned-B (judge 0.5826, recall 0.6939) and our own naive-cosine floor (judge 0.5695, recall 0.7138). On the judge axis, tuning closed most of the gap to the floor, but a paired bootstrap shows that gap (+1.31pp, CI [−0.60, +3.22]) is not yet significant at 95%; on recall@k, tuned-B remains 1.99pp below the floor. We do not claim beating naive cosine. Full two-axis matrix, dated vendor rows, and the honest read: LoCoMo benchmark card.

How to read this — our own headline included. Any memory vendor, us included, can pick the benchmark slice and the judge that flatter it most. The number above is the full corpus, not a flattering slice, and we publish our own floor alongside it rather than only competitor rows. The live dashboard serves this bench-v1 canon as its default scope, with the four-grader "By judge" view; the June 2026 single-conversation grid (conv-26/conv-47) remains selectable there as a historical scope.

The full two-axis matrix, dated vendor rows, and reproducibility live on the LoCoMo benchmark card. The June 2026 conv-26/conv-47 matrix — selectable on the live dashboard as a historical scope — is archived at locomo-archive-2026-06.

Other benchmarks ​

LoCoMo above is our primary, fully-committed matrix. We also run four additional benchmarks, each testing a different memory operation. These have measured engine numbers, but each with a provenance gap relative to the LoCoMo matrix. BEAM-10M is a committed, live matrix cell — but its closest run used a different reader than the canonical protocol. HotpotQA, MuSiQue, and LongMemEval are not yet in the matrix at all — their numbers live only in raw run JSONs, under different readers or partial subsets. Each has its own page with the full protocol, caveats, and the not-comparable competitor rows; read the numbers there with their limits attached.

  • BEAM — long-term agentic memory at conversational scale up to 10M tokens (100 conv / 2,000 Q), the only one of these to test contradiction resolution and event ordering. Closest measured engine run: 0.610 judge pass-rate on BEAM-10M (200 questions), but under a claude-sonnet-4-6 reader, not the canonical gpt-5 protocol — so it is a closest-measured run, not a head-to-head leaderboard row. The headline overstates true retrieval grounding (GT-atom-recall@100 was lower); see the page.
  • HotpotQA — multi-hop QA over multiple supporting documents (bridge + comparison). Closest measured engine run: Answer F1 ≈0.778 (n=500), with support recall ~1.0 because the candidate pool is only 10 paragraphs — but with an unrecorded NVIDIA-hosted small reader and no judge, so it is not on the same axis as the GPT-4o-backbone competitor numbers.
  • MuSiQue — compositional multi-hop reasoning where each answer feeds the next hop (longer chains than HotpotQA). Closest measured engine run: Answer F1 0.457 on the 2-hop subset only (n=200), reader Llama-3.1-8B, no judge; retrieval/chain-coverage ~0.84, so the reader, not retrieval, is the bottleneck.
  • LongMemEval — long-horizon conversational memory across 500 instances, five abilities including knowledge updates and abstention. The two paper-comparable engine runs disagree: a weighted-mean overall of ≈0.62 (run 0327) versus ≈0.79 (run 0323), the gap attributable to a lenient judge pass on the higher run. Without a frozen judge-prompt hash we cannot adjudicate which is canonical, so we present both and headline neither.

Ongoing work ​

  • Bringing research-prototype features into the public API — diversity/temporal-deduplication before the reader, second-pass retrieval, anti-refusal reader prompt
  • Temporal grounding — a principled temporal sub-graph rather than ad-hoc resolvers; the current production path has no dedicated time layer
  • Depth and reader rungs at full corpus — the recipe ladder is measured on one conversation; whether to spend on the full-corpus depth sweep is an open decision
  • Held-out corpora — LoCoMo's full ten conversations are now covered (bench-v1); extending evaluation to held-out conversations from other corpora is still open
  • A canonical cross-provider protocol — re-running every system (ours and competitors') under a single fixed reader + judge so the "other benchmarks" numbers above become directly comparable leaderboard rows, not closest-measured runs
  • Parallel judge fan-out — score-neutral wall-clock reduction (judges currently dominate per-question latency)

Common questions ​

What benchmarks does Mnemoverse run? ​

Five: LoCoMo (the primary, fully-committed cross-system matrix), plus BEAM, HotpotQA, MuSiQue, and LongMemEval — each testing a different memory operation, each with its own page, protocol, and caveats.

What is Mnemoverse's primary benchmark? ​

LoCoMo — a conversational-memory benchmark across multi-session dialogues. It is the fully-committed matrix; the other four have measured engine numbers but each with a provenance gap relative to the LoCoMo matrix.

Why does a benchmark number need its config? ​

Because most of these benchmarks are graded by an LLM-as-judge, and judge model, judge prompt, dataset subset, and reader backbone can each swing a score by double digits. A number is only meaningful next to the exact rig that produced it.

What does "which LoCoMo" mean? ​

The name "LoCoMo" points at two different datasets — the full paper corpus (50 conversations / 7,512 questions) and the full 10-conversation public release (n=1,986, all five categories) that most vendors run, ours included. Our bench-v1 matrix covers that full public release; an earlier single-conversation slice (conv-26) is archived, not current. Read any "LoCoMo 90+%" claim by asking which LoCoMo.

Are Mnemoverse's benchmark numbers comparable to competitors'? ​

Not yet directly. BEAM's closest run used a different reader than the canonical protocol; HotpotQA, MuSiQue, and LongMemEval are not yet in the matrix and use different readers or partial subsets. A canonical cross-provider protocol — one fixed reader + judge — is in progress to make them comparable leaderboard rows.

— Mnemoverse is a persistent-memory API for AI agents. Free key: console.mnemoverse.com · Plans and limits · Docs: Getting Started