Benchmark Wars
How AI-memory systems are measured, judged, and gamed — evaluation methods, LLM-as-judge pitfalls, and honest benchmarking.
7 articles
AI Memory Benchmarks: A Field Guide
A map of how AI-agent memory is actually measured — LoCoMo, LongMemEval, BEAM and the long-context tests — what each one checks, what it misses, and which to trust.
LLM-as-Judge Variance in AI Memory Benchmarks
Swap only the grading prompt on the same fixed answers and an AI-memory benchmark score moves ~40 points. Why you can't trust a memory leaderboard without the judge's recipe.
DeepEval: Pytest for LLM Evaluation
DeepEval is a pytest-style LLM evaluation framework that turns evals into CI tests; most metrics are LLM-as-a-judge, so G-Eval has variance and DAG adds rigor.
Hugging Face Evaluate Library: Metrics & compute() Guide
Hugging Face evaluate library: evaluate.load and compute(predictions, references), the list-of-lists shape people get wrong for BLEU, and where it fits in 2026.
How to Evaluate AI Agent Memory
How to evaluate AI agent memory: the framework, the dimensions, a benchmark map (LoCoMo, LongMemEval, BEAM), and why latency and cost are co-equal axes.
LLM-as-a-Judge: Bias, Leniency & the LoCoMo Number
LLM-as-a-judge leniency and bias explained: how a "be generous" grader, MT-Bench failures, and the LoCoMo judge shape the memory-benchmark numbers vendors cite
LangChain & LangSmith Evaluation: The Memory Blind Spot
LangChain/LangSmith evaluation explained: datasets, LLM-as-judge biases, the eval tool landscape, and the blind spot none cover — whether your agent remembers.