Benchmark Wars
How AI-memory systems are measured, judged, and gamed — evaluation methods, LLM-as-judge pitfalls, and honest benchmarking.
9 articles
BLEU vs ROUGE vs F1 vs SARI: Pick the Right Metric
BLEU, ROUGE, F1, accuracy, and SARI explained by metric question, Evaluate I/O shape, failure mode, and routing.
Can You Trust an LLM Judge? A Field Manual
LLM-as-judge reliability explained: assess position bias, verbosity, self-preference, rubrics, retrieval recall, and benchmark comparability.
AI Memory Benchmarks: A Field Guide
A map of how AI-agent memory is actually measured — LoCoMo, LongMemEval, BEAM, long-context tests — what each checks, what it misses, and which to trust.
LLM-as-Judge Variance in AI Memory Benchmarks
Swap only the grading prompt on the same answers and an AI-memory benchmark score moves ~40 points — you can't trust a leaderboard without the recipe.
DeepEval: Pytest for LLMs — G-Eval, DAG & RAG Triad
DeepEval turns LLM evals into pytest tests that fail CI builds. 50+ metrics, but most are LLM-as-a-judge (G-Eval: 0.514 vs humans) — DAG adds determinism.
Hugging Face Evaluate Library: load(), compute() Guide
Practical Hugging Face evaluate docs: evaluate.load() + compute(predictions, references), the BLEU list-of-lists trap, and no information_retrieval metric.
How to Evaluate AI Agent Memory
How to evaluate AI agent memory: the framework, the dimensions, a benchmark map (LoCoMo, LongMemEval, BEAM), and why latency and cost are co-equal axes.
LLM-as-a-Judge: Bias, Leniency & the LoCoMo Number
LLM-as-a-judge leniency and bias explained: how a "be generous" grader, MT-Bench failures, and the LoCoMo judge shape the memory-benchmark numbers vendors cite
LangChain & LangSmith Evaluation: The Memory Blind Spot
LangChain/LangSmith evaluation explained: datasets, LLM-as-judge biases, the eval tool landscape, and the blind spot none cover — whether your agent remembers.