MnemoverseDocs

Appearance

Edward Izgorodin·Published Jun 7, 2026Updated Aug 17, 2026

Hugging Face Evaluate Library: load(), compute(), and the One-Liner Everyone Gets Wrong ​

Hugging Face's evaluate library promises a BLEU score in one line of Python — and it delivers. The catch is that it is easy to compute a BLEU number that is quietly wrong, because the references argument has a shape that isn't the obvious one. Get the shape right and evaluate is the fastest path to a trustworthy metric. Get it wrong and you ship a number that means nothing. Here is the correct usage, task by task, plus an honest take on where the library belongs in 2026.

TL;DR

  • evaluate loads dozens of evaluation modules with one line and scores them with compute().
  • Three kinds of module: metrics (predictions vs references), comparisons (two models), measurements (dataset properties).
  • The shape: for multiple references per prediction, BLEU/sacreBLEU references is a list of lists of strings; a flat list is read as exactly one reference per prediction (verified on v0.4.6).
  • combine(), the evaluator, and EvaluationSuite cover everything past a single metric.
  • There is no information_retrieval module — evaluate.load("information_retrieval") isn't built in; recall@k, MRR and nDCG live in other tools.
  • For LLM evaluation, Hugging Face now points to LightEval; evaluate is the classic-metrics layer, not deprecated.

The trap, first ​

Start with the mistake, because it is the reason this library so often gets looked up twice.

python
import evaluate
bleu = evaluate.load("bleu")

# ERRORS — two references for ONE prediction, passed flat:
# ValueError: Mismatch in the number of predictions (1) and references (2)
bleu.compute(predictions=["hello there general kenobi"],
             references=["hello there general kenobi", "hello there!"])

# SILENT — the counts happen to line up, so nothing complains:
# each reference is paired with a DIFFERENT prediction, and it still scores:
# bleu 0.691441569283882 on evaluate 0.4.6 with the default tokenizer
# (the exact value moves with version/tokenizer; the silent mis-pairing does not)
bleu.compute(predictions=["hello there general kenobi", "hello there!"],
             references=["hello there general kenobi", "foo bar foobar"])

# RIGHT — references as a list of *lists* of strings
bleu.compute(predictions=["hello there general kenobi"],
             references=[["hello there general kenobi", "hello there!"]])

The reason is linguistic, not arbitrary: a translation can have several equally valid references, so BLEU is defined against a set of references per prediction. predictions is a flat list of candidate strings; references is a list where each entry is itself a list of acceptable strings. A flat list is also accepted, and it means exactly one reference per prediction (verified on evaluate v0.4.6) — which is precisely why the multi-reference case bites. Pass several references flat and one of two things happens: the counts stop matching and you get ValueError: Mismatch in the number of predictions (1) and references (2), or they happen to match and each reference is silently paired with a different prediction. The second is the dangerous one — it scores, and the number is wrong (verified on evaluate 0.4.6, sacrebleu raises the same way). Once that clicks, the rest of the library is genuinely one line at a time.

What the evaluate library is ​

evaluate is a library for evaluating ML models and datasets, exposing "dozens of popular metrics" through one uniform API. It operates on plain Python lists, so it is framework-agnostic — it works with PyTorch, TensorFlow and scikit-learn outputs alike. Its modules come in three kinds:

  • Metrics — predictions vs ground truth (accuracy, F1, BLEU, ROUGE, METEOR, exact_match).
  • Comparisons — two models against each other.
  • Measurements — properties of a dataset (text complexity, label distribution).

Most day-to-day use is metrics, and the pattern is always the same: evaluate.load("name"), then .compute(...), which returns a dictionary. For what each metric actually measures and when it lies, see the concept companion: BLEU vs ROUGE vs F1 vs SARI.

Recipe 1 — Score a classifier (accuracy, F1) ​

Classification metrics take flat lists of labels:

python
accuracy = evaluate.load("accuracy")
accuracy.compute(predictions=[0, 1, 1, 0], references=[0, 1, 0, 0])
# {'accuracy': 0.75}

f1 = evaluate.load("f1")
f1.compute(predictions=[0, 1, 1], references=[0, 1, 0], average="macro")

Recipe 2 — Score a translation (BLEU / sacreBLEU) ​

The one to memorize — predictions is a list of strings; references is a list of lists of strings:

python
bleu = evaluate.load("bleu")
predictions = ["hello there general kenobi", "foo bar foobar"]
references  = [["hello there general kenobi", "hello there!"], ["foo bar foobar"]]
bleu.compute(predictions=predictions, references=references)

# sacreBLEU uses the same list-of-lists shape, but — unlike bleu — it requires the
# SAME number of references for every prediction, so pad to equal length:
sacrebleu = evaluate.load("sacrebleu")
sacrebleu.compute(
    predictions=predictions,
    references=[["hello there general kenobi", "hello there!"],
                ["foo bar foobar", "foo bar foobar"]],
)
# {'score': ..., 'counts': [...], 'totals': [...], 'precisions': [...],
#  'bp': ..., 'sys_len': ..., 'ref_len': ...}

A flat list of reference strings is read as one reference per prediction (verified on v0.4.6 for both bleu and sacrebleu). The dangerous case is having multiple references and passing them flat: the counts stop lining up with predictions, and you get a shape error or, worse, silently mis-paired references. (And note the asymmetry above: bleu accepts a different number of references per prediction; sacrebleu does not.)

Recipe 3 — Score a summary (ROUGE) and exact strings ​

python
rouge = evaluate.load("rouge")
rouge.compute(predictions=["the cat sat on the mat"],
              references=["the cat sat on the mat"])
# {'rouge1': ..., 'rouge2': ..., 'rougeL': ..., 'rougeLsum': ...}

em = evaluate.load("exact_match")
em.compute(predictions=["Paris"], references=["Paris"])   # {'exact_match': 1.0}

ROUGE is recall-oriented and intended for summarization; predictions and references are lists of strings.

Recipe 4 — Several metrics at once (combine) ​

python
preds, refs = [0, 1, 1, 0], [0, 1, 0, 0]   # same flat label lists as Recipe 1
clf = evaluate.combine(["accuracy", "f1", "precision", "recall"])
clf.compute(predictions=preds, references=refs)   # one dict with all four

Recipe 5 — Evaluate a whole model (evaluator) ​

The evaluator runs a model or pipeline over a dataset and metric end-to-end, so you skip the manual loop:

python
task = evaluate.evaluator("text-classification")
task.compute(model_or_pipeline="distilbert-base-uncased-finetuned-sst-2-english",
             data="imdb", metric="accuracy")

For many tasks at once, EvaluationSuite bundles (evaluator, dataset, metric) tuples so one model is scored across them, with the suite stored on the Hub.

What it does NOT have ​

A frequent search is evaluate.load("information_retrieval") — there is no such built-in module. The core library is classic-metric territory; IR measures like recall@k, MRR and nDCG come from other libraries or community modules on the Hub, not from evaluate itself. Knowing the boundary saves an afternoon.

Where it fits in 2026 ​

evaluate is excellent at standardized, reproducible metric computation, and it is not going away. But for LLM evaluation, Hugging Face's own focus has shifted to LightEval — a more actively maintained toolkit with 1000+ tasks, from Hugging Face's Leaderboard & Evals team; the evaluate docs themselves now point there for recent LLM work. Read it as a division of labor: evaluate is the classic-metrics layer; LightEval is the LLM-eval layer. Neither is deprecated.

The deeper point is what these metrics can't see. BLEU and ROUGE score string overlap; accuracy and F1 score labels. None of them can tell you whether a model reasoned correctly, used a tool properly, or remembered a fact from three sessions ago — the same blind spot the broader agent-evaluation tooling leaves open, and the reason evaluating an agent's memory needs a different instrument.

Common questions ​

What is the Hugging Face evaluate library? ​

A Python library for one-line evaluation of ML models and datasets — evaluate.load() a module, compute() a score. Modules are metrics, comparisons, or measurements.

What is the predictions/references format for BLEU and sacreBLEU? ​

predictions is a list of strings; references is a list of lists of strings (≥1 reference per prediction). sacreBLEU returns score, counts, totals, precisions, bp, sys_len, ref_len.

How do you use evaluate.load() and compute()? ​

m = evaluate.load("bleu"); m.compute(predictions=preds, references=refs) → a dict of scores.

Does evaluate.load("information_retrieval") exist as a metric? ​

No — evaluate.load("information_retrieval") is not a built-in module. Use other tools or community Hub modules for recall@k, MRR and nDCG.

Should I use evaluate for LLM evaluation in 2026? ​

For classic metrics, yes; for LLMs, Hugging Face now points to LightEval (1000+ tasks). evaluate is not deprecated, just no longer the LLM-eval focus.

Sources ​

— Mnemoverse is a persistent-memory API for AI agents. Free key: console.mnemoverse.com · Docs: Getting Started