Skip to content

Benchmark Wars

How AI-memory systems are measured, judged, and gamed — evaluation methods, LLM-as-judge pitfalls, and honest benchmarking.

11 articles

Oct 4, 2026·26 min read·Updated Nov 5, 2026

How Are AI Code Review Tools Benchmarked, and Can You Trust the Results?

AI code review benchmarks explained: answer keys, LLM judges, counting rules and live leaderboards, and what to ask before trusting a #1 claim.

Sep 20, 2026·14 min read

RAGAS Metrics Explained: Reference-Free RAG Evaluation

RAGAS metrics explained: faithfulness, response relevancy, context precision and recall. Two of the six text RAG metrics need no reference; none spans sessions.

Aug 17, 2026·10 min read

BLEU vs ROUGE vs F1 vs SARI: Pick the Right Metric

BLEU, ROUGE, F1, accuracy, and SARI explained by metric question, Evaluate I/O shape, failure mode, and routing.

Jul 12, 2026·10 min read

Can You Trust an LLM Judge? A Field Manual

LLM-as-judge reliability explained: assess position bias, verbosity, self-preference, rubrics, retrieval recall, and benchmark comparability.

Jun 23, 2026·18 min read

AI Memory Benchmarks: A Field Guide

A map of how AI-agent memory is actually measured — LoCoMo, LongMemEval, BEAM, long-context tests — what each checks, what it misses, and which to trust.

Jun 23, 2026·13 min read·Updated Sep 5, 2026

LLM-as-Judge Variance in AI Memory Benchmarks

Swap only the grading prompt on the same answers and an AI-memory benchmark score moves ~40 points — you can't trust a leaderboard without the recipe.

Jun 7, 2026·9 min read

DeepEval: Pytest for LLMs — G-Eval, DAG & RAG Triad

DeepEval turns LLM evals into pytest tests that fail CI builds. 50+ metrics, but most are LLM-as-a-judge (G-Eval: 0.514 vs humans) — DAG adds determinism.

Jun 7, 2026·6 min read·Updated Aug 17, 2026

Hugging Face Evaluate Library: load(), compute() Guide

Practical Hugging Face evaluate docs: evaluate.load() + compute(predictions, references), the BLEU list-of-lists trap, and no information_retrieval metric.

Jun 6, 2026·15 min read

How to Evaluate AI Agent Memory

How to evaluate AI agent memory: the framework, the dimensions, a benchmark map (LoCoMo, LongMemEval, BEAM), and why latency and cost are co-equal axes.

Jun 5, 2026·13 min read

LLM-as-a-Judge: Bias, Leniency & the LoCoMo Number

LLM-as-a-judge leniency and bias explained: how a "be generous" grader, MT-Bench failures, and the LoCoMo judge shape the memory-benchmark numbers vendors cite

Jun 4, 2026·9 min read

LangChain & LangSmith Evaluation: The Memory Blind Spot

LangChain/LangSmith evaluation explained: datasets, LLM-as-judge biases, the eval tool landscape, and the blind spot none cover — whether your agent remembers.

← Back to the Library