Erratum (2026-09-03)
The numbers on this page — J=0.116 to 0.862 across versions v0.1–v0.7, and the leaderboard comparison below — come from a pipeline with an oracle leak discovered on 2026-03-04, and are retracted. Ground-truth evidence structure was inadvertently used to build Hebbian graph connections during evaluation, giving retrieval an unfair structural advantage. At publication (2026-04) we believed the leak was limited to v0.1–v0.2 and removed before v0.3 — that is what the note further down this page still says. The bench-v1 audit (2026-09) established that the same leak we then believed removed before v0.3 in fact reached the whole v0.1–v0.7 line.
Our current registered numbers (bench-v1, the full 10-conversation public LoCoMo release — not the 50-conversation / 7,512-question paper corpus): LLM-judge 0.5468 (stock) / 0.5826 (tuned-B) against a floor of 0.5695, over all 1,986 questions (five categories, adversarial included); recall@k 0.6316 (stock) / 0.6939 (tuned-B) against a naive-cosine floor of 0.7138, over the 1,536 questions that carry an evidence label (the 1,540 non-adversarial questions, categories 1–4, minus four without a label; the adversarial category is excluded from this axis by protocol). On the LLM-judge axis tuned-B sits 1.31pp above the floor — not separable at 95% confidence (paired bootstrap on that grader); on recall@k it remains 1.99pp below the floor, though its delivered-set recall (0.7417) exceeds it. We do not claim beating naive cosine. We publish the floor openly, as a self-critical baseline. Methodology and the full registry: Benchmarks.
The engineering narrative below — the visualization tool, the graph self-organization findings — is kept for its own value. Every J score in it is historical and retracted, not current.
Building Memory That Scales
The Journey (historical numbers, retracted)
An AI that remembers which colleague recommended the Python library — and what other projects they contributed to. That tracks how the team's opinion on microservices changed after the March outage. Not keyword search. Actual memory across hundreds of sessions.
The current version of the memory engine — the one with benchmarks and a leaderboard position — was built over four months in early 2026. But the idea is older. The first prototype dates to early 2023, right after ChatGPT-3 appeared: a knowledge base built on FAQ entries stored in SQLite, tagged with entities extracted by spaCy. No embeddings, no cosine similarity — embedding models at the time were too weak for production semantic search. It was keyword matching with NER on top.
The approach has changed completely since then. The goal has not: give AI agents the ability to remember what worked, forget what didn't, and build an internal map of what matters.
Seven versions of the current engine later, the system scored 0.862 on the LoCoMo benchmark (1,986 questions across 10 multi-session conversations), placed second on the leaderboard, and maintained that quality as memory grew 14× in size (retracted; see erratum above).
What did 0.862 mean in practice? It meant the engine could recall details from arbitrarily long conversation histories — hundreds of sessions, weeks of interaction — and answer questions about them correctly 86% of the time, on that retracted run. Not just simple lookups ("what's Caroline's email?") but queries that require real cognitive work:
- Multi-hop: "Which project did the person who recommended the Python library also contribute to?" — the engine finds the library recommendation, identifies who made it, then finds their other projects. Three separate memories, chained together.
- Temporal: "Did the team's opinion on microservices change after the outage in March?" — the engine finds opinions expressed before and after a specific event, then surfaces both so they can be compared.
These are the kinds of questions a colleague answers effortlessly after months of working with you — and that every AI assistant today fails at, because it cannot remember last Tuesday.
This is the story of how we got there — and the 3D visualization tool that became our most important research instrument along the way.
TL;DR
- Retracted (2026-09-03). The J=0.116 to 0.862 progression and the leaderboard placement below come from the oracle-leak-era pipeline and are withdrawn; see the erratum at the top of this page and the bench-v1 results that replace them.
- The biggest single jump (+11%) came from a better embedding model; a Hebbian knowledge graph later lifted multi-hop reasoning.
- Quality held flat as memory grew 14× (419 → 5,880 atoms), and the graph self-organised into a sparse, power-law structure where 57% of edges are never reused.
- Honest limits remain: multi-hop is the bottleneck, benchmark data caps at ~6K memories, and native hyperbolic embeddings aren't production-ready yet.
Starting from nothing
Numbers in this section are retracted
Every J score and table from here through "Testing at scale (v0.7)" comes from the 2026-03-04 oracle-leak-era pipeline (see the erratum at the top of this page) and is retracted — kept for the engineering narrative, not as a current result.
Version 0.1 had no intelligence. It stored memories as vectors, retrieved by cosine similarity, and returned raw results without any processing. No LLM reader, no graph, no consolidation. J score: 0.116. Barely better than random.
We needed a baseline to measure against, and 0.116 was it. Everything that followed was an attempt to understand why retrieval alone wasn't enough.
(A note on early measurements: versions 0.1-0.2 had a data leak — ground-truth evidence structure was inadvertently used to build Hebbian connections during evaluation. The graph didn't see answers directly, but it learned which memories were relevant to each question, giving retrieval an unfair structural advantage. This was discovered and removed before v0.3. All numbers from v0.3 onward use blind feedback only. 2026-09 update: the later-found oracle leak affected the whole v0.1–v0.7 line — see the erratum above.)
The embedding jump (v0.2 → v0.3)
The first real jump came from a single change: better embeddings.
| Version | J Score | What changed |
|---|---|---|
| v0.2 | 0.776 | Added LLM reader with prompt tuning |
| v0.3 | 0.865 | Better embedding model |
That's +11% from switching the embedding model. The old model (2021 vintage, 384 dimensions) simply couldn't distinguish between "Caroline prefers morning meetings" and "the team discussed scheduling options." The new model could. Multi-hop accuracy — questions requiring evidence from multiple memories — jumped from 0.450 to 0.723. A 61% improvement from one change.
This taught us something we should have known earlier: the quality of the embedding model dominates everything downstream. Better retrieval is worth more than clever post-processing.
Graph-based retrieval (v0.5)
Version 0.5 added the Hebbian knowledge graph — connections between memories that strengthen when memories are retrieved together and lead to good outcomes.
| Version | J Score | Multi-hop |
|---|---|---|
| v0.3 | 0.865 | 0.723 |
| v0.5 | 0.878 | 0.770 |
The overall improvement was modest (+1.5%), but multi-hop went from 0.723 to 0.770. The graph found evidence chains that cosine similarity alone couldn't: memory A links to memory B through a shared concept, even though A and B aren't directly similar.
Testing at scale (v0.7)
The question we kept avoiding: does this hold up as memory grows?
Version 0.7 added per-conversation scaling instrumentation. After processing all 10 conversations, the engine held 5,880 memories — fourteen times more than after the first conversation (419).
The results were better than we expected.
Quality stays flat
| Memory size | J Score | Retrieval Recall |
|---|---|---|
| 419 atoms | 0.845 | 0.885 |
| 1,451 atoms | 0.898 | 0.941 |
| 2,760 atoms | 0.831 | 0.908 |
| 4,123 atoms | 0.877 | 0.938 |
| 5,880 atoms | 0.873 | 0.897 |
J score variance across conversations (std=0.021) is larger than any trend with memory size. The difficulty of individual conversations matters more than how much the engine remembers. At 14× growth, quality is indistinguishable from where it started.
The graph self-organises
As memory grows, the graph develops structure without being told to:
- Density drops from 1.74% to 0.69%. The graph becomes more selective, not more noisy. Each new memory adds connections only where they matter.
- Hub ratio grows from 6× to 32×. A small number of concepts accumulate many connections while most concepts have only one or two. This is the power-law degree distribution seen in real-world knowledge networks — Wikipedia articles, citation graphs, social networks.
- 57% of edges are never accessed after creation. Most connections form once and are never reinforced. Only 4% of edges carry meaningful weight (>0.2). The graph naturally separates signal from noise.
These are emergent properties. We didn't program sparsification or hub formation. They arise from the combination of co-activation strengthening and the structure of real conversation data.
Seeing it happen: the visualization dashboard
Numbers in tables are useful. But the moment we truly understood how memory behaves was when we built a 3D visualization of the graph and watched it grow in real time.
graph.mnemoverse.com renders the memory graph as an interactive 3D force-directed layout:
- Node size shows importance — how novel the memory was when it entered
- Node color shows category — what kind of knowledge it represents
- Node shape shows outcome — did this memory lead to good results (sphere), bad results (octahedron), or unknown (cube)?
- Edge thickness shows connection strength — how often two memories are retrieved together successfully
- Orange rings flag memories whose internal representation has drifted from their original encoding
- Glowing nodes are prototypes — consolidated summaries created during sleep cycles
Walk through the graph after a 10-conversation benchmark run and you see the scaling story play out visually. Hub nodes — the concepts that connect many memories — sit at the center, pulling related clusters toward them. Peripheral nodes with weak connections float at the edges. The 57% of dead edges are visible as thin, barely-there lines connecting nodes that never co-activated.
This is an Euclidean projection of what is conceptually a hyperbolic structure. Like a Mercator map, it distorts — nodes near the "boundary" of the knowledge space appear closer together than they should. But it gives an intuition that no table of numbers can: memory has spatial structure, and that structure tells you something about what the agent knows.
The dashboard became our primary research tool. Before running benchmarks, we check the graph visually. After a benchmark, we look for new clusters, orphaned nodes, suspiciously large hubs. We have caught bugs this way — a memory that shouldn't have connected to anything showing up at the centre of a cluster. No automated test would have flagged it.
Where we stand on the leaderboard
Provenance. The Mnemoverse figures here come from a full 10-conversation LoCoMo run in February 2026 (v0.7) and reflect that snapshot of the engine and the field. Benchmark methodology for agent memory is still unstable across the industry — scores move with run parameters, judge choice, and which dataset subset is used as much as with the systems themselves — so we built a more reproducible in-house protocol. For the registered matrix at the time (run 2026-06-08, with fixed judges and a recorded dataset hash), since archived, see the June 2026 matrix.
2026-09-03 update. That February 2026 run is itself the one found to be oracle-corrupted (leak discovered 2026-03-04) and is retracted — see the erratum at the top of this page. The reproducible protocol mentioned above is now live as bench-v1; see Benchmarks for the current registered numbers.
Numbers in this table are retracted
The Mnemoverse row below comes from the 2026-03-04 oracle-leak-era run and is retracted. The other rows are external, self-reported figures of that era, shown for context only — we neither audited nor endorse them, and none of them reflect current registered numbers. See the erratum at the top of this page.
The LoCoMo leaderboard provides context for these numbers:
| System | J Score |
|---|---|
| MemMachine v0.2 | 0.912 |
| Mnemoverse v0.7 | 0.862 |
| Memobase | 0.758 |
| Zep | 0.751 |
| Mem0 | 0.669 |
On that retracted run we placed second, five points behind, with the gap concentrated in multi-hop reasoning (0.707 against 0.897). This is not a current standing: the whole row is withdrawn, and the bench-v1 card carries what replaced it.
Of the scores above, only the Mnemoverse row is our own measurement. The other four rows are external figures from the LoCoMo leaderboard, produced under their own harnesses — not ours — so the comparison is indicative rather than protocol-controlled. Vendor-reported numbers can differ again: Mem0, for instance, reports a higher LoCoMo figure of its own. When we quote a system's self-reported score, we cite the source; see the 2026 memory landscape.
We also evaluated on two other benchmarks:
| Benchmark | Key result |
|---|---|
| HotpotQA (500 questions) | F1=78%, retrieval recall=100% |
| LongMemEval (500 questions) | Retrieval recall=99.3% |
Retrieval recall was consistently high across all benchmarks in that retracted run (89-100%). The gap between recall and final accuracy came from answer extraction — the reader had the right evidence but didn't always produce concise answers in the expected format, and closing it was our primary area of active work at the time. (Figures from the retracted run; registered recall@10 under bench-v1 is 0.6316 stock / 0.6939 tuned-B — see erratum above.)
What we haven't solved
Honest accounting of limitations:
Multi-hop is the bottleneck. Questions that require chaining evidence across multiple memories are where we lose to the top system. Our graph expansion helps, but not enough.
Benchmark data caps out at 6K memories. The 5,880 atoms represent all 10 LoCoMo conversations combined — the largest publicly available conversational memory dataset we have found. The engine itself can handle more, but we lack a benchmark with 10× more data to validate quality at that scale. Building one is planned.
Automatic consolidation is not on by default yet. Sleep-cycle consolidation — clustering redundant atoms into prototypes — ships as an on-demand operation (POST /api/v1/memory/consolidate) and is available in production. What is not yet enabled by default is the automatic background sleep-cycle that would run it on a schedule (maintenance_hard_consolidation): we are still tuning when and how aggressively to compress — including against the benchmarks on this page — before turning it on automatically. Until then, unless consolidation is triggered explicitly, memory grows without automatic compression.
The visualization is Euclidean. The mathematical foundation uses hyperbolic geometry, but the dashboard renders in 3D Euclidean space. It's useful for intuition and bug-catching, but it doesn't faithfully represent the hyperbolic distances. Building a true hyperbolic visualization is future work.
Hyperbolic embeddings are the missing piece. Better embeddings transformed retrieval quality (+11% from one model swap). We expect the same effect for geometric placement — positioning memories in the Poincaré ball according to their hierarchical relationships. But the usual workaround — projecting Euclidean embeddings onto the hyperbolic manifold — is noisy, so we do not ship it: the engine stays Euclidean end to end and the geometry is deferred. Native hyperbolic embedding models are not yet mature enough for production; closing that gap is the aim of our tensor-hyperbolic graph research.
Common questions
What is Mnemoverse's LoCoMo score?
The 0.862 figure once quoted here was retracted (2026-09-03 erratum) — it came from a pipeline with an oracle leak found on 2026-03-04. Our current registered numbers on the full 10-conversation public LoCoMo release are LLM-judge 0.5468 (stock) / 0.5826 (tuned-B) over all 1,986 questions (five categories), and recall@k 0.6316 (stock) / 0.6939 (tuned-B) over the 1,536 questions that carry an evidence label (the 1,540 non-adversarial questions, categories 1–4, minus four without a label; the adversarial category is excluded from this axis by protocol); see Benchmarks for the full registry and methodology.
What most improved the memory engine's accuracy?
Historical, retracted (oracle-leak era): in the retracted v0.1–v0.7 series, the largest single jump was +11% from switching to a better embedding model (v0.2 to v0.3, 0.776 to 0.865 on the retracted J score); multi-hop accuracy rose from 0.450 to 0.723 on that same run. The qualitative lesson — better retrieval matters more than post-processing — still holds; the specific numbers are not part of bench-v1's registered results.
Does memory quality degrade as it grows?
Historical, retracted (oracle-leak era): across that retracted v0.7 run, as memory grew 14× (419 to 5,880 atoms) the J score stayed flat — variance across conversations (std 0.021) exceeded any trend with size, and retrieval recall held between 88.5% and 94.1% (the run's own table). This scaling behavior has not been re-measured under bench-v1; read it as an unconfirmed historical finding, not a current claim.
What is a Hebbian knowledge graph in memory?
Connections between memories that strengthen when they are retrieved together and lead to good outcomes. Added in v0.5; in the now-retracted v0.1–v0.7 series it appeared to lift multi-hop from 0.723 to 0.770 by finding evidence chains cosine similarity alone missed — a historical finding, not a current registered result.
What are the limits of the current memory engine?
Multi-hop reasoning was the bottleneck in the retracted v0.1–v0.7 series (0.707 versus the leader's 0.897 on that now-invalid leaderboard). Independent of that retraction, benchmark data still caps at about 6K memories, automatic consolidation is still not on by default, and native hyperbolic embeddings are still not production-ready.
Sources
Related
- Detailed benchmark tables — full results across LoCoMo, HotpotQA, LongMemEval
- Design Language — how each visual element maps to an engine concept
- When AI Cites What Doesn't Exist — why persistent memory needs verification
- AI Memory Landscape 2026 — where Mnemoverse fits in the market
- The Judge Says Yes Too Easily — how the LLM judge behind these J scores works, and why leniency inflates them
Edward Izgorodin, April 2026 — LinkedIn
— Mnemoverse is a persistent-memory API for AI agents. Free key: console.mnemoverse.com · Plans and limits · Docs: Getting Started