MnemoverseDocs

Appearance

Edward Izgorodin·Published Aug 17, 2026

BLEU, ROUGE, F1, SARI: What Each Metric Measures, and When It Lies ​

TL;DR

  • BLEU asks whether generated translations reuse reference n-grams with a brevity penalty. It does not compare meaning.
  • ROUGE asks how much reference wording appears in a generated summary. Its output type changes when use_aggregator=False.
  • F1 asks how precision and recall trade off for labels. Accuracy can be misleading on unbalanced classes.
  • SARI is the text-simplification metric that needs sources, because it scores add, delete, and keep operations against both the input and references.
  • Score mismatches with published papers usually trace to scale, tokenization, or reference shape differences.

BLEU is a machine-translation metric based on modified n-gram precision with a brevity penalty. ROUGE is a summarization metric based on recall-oriented n-gram or longest-common-subsequence overlap. F1 is the harmonic mean of precision and recall for label predictions. SARI is a text-simplification metric that compares a system output against references and the original input sentence.

These metrics are useful because they answer narrow questions. They fail when readers treat those questions as broader quality judgments. And score mismatches with published papers usually trace to scale, tokenization, or reference shape differences.

This article is the metric-concept layer: what each metric measures, its minimal Hugging Face Evaluate input shape, its documented sharp edges, and when to use another tool. For load() and compute() mechanics, see the companion Hugging Face Evaluate deepdive.

Hugging Face Evaluate status in 2026 ​

Hugging Face Evaluate is alive, in low-velocity maintenance. The repository was not archived at the 2026-08-17 review, and the latest release was v0.4.6 on 2025-09-18 for huggingface_hub>=1.0 support, according to the GitHub releases API and PyPI (GitHub API, PyPI). The 2026 commits on the main branch at review time were CI and documentation changes.

The official README says: "For more recent evaluation approaches, for example for evaluating LLMs, we recommend our newer and more actively maintained library LightEval." The docs index carries the same steer (README, docs). That is a division of labor, not a deprecation notice. Evaluate remains a uniform load() and compute() interface for metrics over predictions you already have. LightEval is a task-suite runner for LLM evaluation, not a drop-in replacement for computing BLEU or SARI over local lists of strings.

As of the 2026-08-17 review, the interactive Space widgets for f1, accuracy, and sari showed ModuleNotFoundError: No module named 'distutils'. The metric-card text remained readable in the Space README and GitHub mirror.

BLEU: n-gram precision for machine translation ​

BLEU measures modified n-gram precision against one or more reference translations, then applies a brevity penalty. The Evaluate BLEU card describes predictions as a list of strings and references as a list of strings or a list of lists of strings; the card example shows unequal reference counts per prediction (BLEU card). The nested shape is the one you want for multiple references. A flat list is read as exactly one reference per prediction (verified on evaluate 0.4.6); pass several references flat and you get either a count-mismatch ValueError (for one prediction and two references: Mismatch in the number of predictions (1) and references (2)), or — when the counts happen to match — each reference silently paired with a different prediction. That second shape is the trap the companion deepdive opens with.

Minimal card example:

python
predictions = ["hello there general kenobi", "foo bar foobar"]
references = [["hello there general kenobi", "hello there!"], ["foo bar foobar"]]

Output:

python
{'bleu': 1.0, 'precisions': [1.0, 1.0, 1.0, 1.0], 'brevity_penalty': 1.0, 'length_ratio': 1.1666666666666667, 'translation_length': 7, 'reference_length': 6}

Evaluate BLEU wraps Tensorflow NMT’s bleu.py, not NLTK or sacreBLEU, according to the metric card. Its default tokenizer is tokenizer_13a, described on the card as equivalent to mteval-v13a and used by WMT. The output key is bleu, and the score is on a 0 to 1 scale.

The warranty is narrow. The card states that BLEU compares token overlap instead of meaning, that scores are not comparable across datasets or languages, and that scores can vary greatly with parameters, especially tokenization and normalization (BLEU card). If a paper reports sacreBLEU, do not expect Evaluate BLEU to match by default.

ROUGE: recall overlap for summarization ​

ROUGE measures recall-oriented word overlap, including n-gram overlap and longest-common-subsequence variants, for generated summaries. Evaluate ROUGE accepts predictions as a list of strings and references as a list of strings or a list of lists of strings (ROUGE card).

Minimal card example:

python
predictions = ["hello there", "general kenobi"]
references = ["hello there", "general kenobi"]

Output:

python
{'rouge1': 1.0, 'rouge2': 1.0, 'rougeL': 1.0, 'rougeLsum': 1.0}

The default rouge_types are ['rouge1','rouge2','rougeL','rougeLsum']. With use_aggregator=True, the output is a dictionary of floats on a 0 to 1 scale. With use_aggregator=False, the output type changes to lists of per-sentence scores. The card example says results["rouge1"] == [0.5, 0.0] in that mode.

Two details matter for paper comparisons. The card says ROUGE is case insensitive. It also says rougeL computes longest common subsequence per sentence, while rougeLsum splits text on "\n" first. For multi-sentence summaries, newline-joined sentences are the relevant input shape for paper-comparable rougeLsum (ROUGE card).

F1 and accuracy: classification metrics with class-balance traps ​

Accuracy measures the fraction of predictions equal to references, while F1 measures the harmonic mean of precision and recall. Evaluate accuracy takes integer predictions and integer references, plus normalize and sample_weight (accuracy card).

Minimal accuracy card example:

python
evaluate.load("accuracy").compute(references=[0, 1], predictions=[0, 1])

Output:

python
{'accuracy': 1.0}

The accuracy card’s caveat is direct: "This metric can be easily misleading, especially in the case of unbalanced classes." If normalize=False, accuracy returns the count of correct predictions rather than the fraction.

Evaluate F1 takes integer predictions and references, plus labels, pos_label, average, and sample_weight. The default average is 'binary'; multiclass use requires settings such as 'micro', 'macro', 'weighted', 'samples', or None (F1 card).

Minimal F1 card example:

python
f1_metric = evaluate.load("f1")
f1_metric.compute(references=[0, 1, 0, 1, 0], predictions=[0, 0, 1, 1, 0])

Output:

python
{'f1': 0.5}

The card’s multiclass example uses predictions=[0,2,1,0,0,1] and references=[0,1,2,0,1,2]. It reports macro as 0.27, micro as 0.33, weighted as 0.27, and average=None as:

python
{'f1': array([0.8, 0. , 0. ])}

Under the hood, Evaluate’s F1 module imports f1_score from sklearn.metrics and forwards to it (source). If you need classification reports or confusion matrices, scikit-learn is usually the more direct tool (scikit-learn model evaluation).

SARI: the text-simplification metric that needs sources ​

SARI measures text simplification by scoring words added, deleted, and kept against both the source sentence and reference simplifications. The SARI card defines it as: "SARI (system output against references and against the input sentence)… compares the predicted simplified sentences against the reference and the source sentences. It explicitly measures the goodness of words that are added, deleted and kept by the system." (SARI card).

That is why SARI’s Evaluate shape is different:

python
sources      # list of str
predictions  # list of str
references   # list of lists of str

The formula on the card is:

python
sari = ( F1_add + F1_keep + P_del) / 3

Note the asymmetry: additions and keeps are scored with F1, deletions with precision only. The design reason is stated in the original paper: "For deletion, we only use precision because over-deleting hurts readability much more significantly than not deleting" (Xu et al. 2016).

The output is {'sari': float} on a 0 to 100 scale, where higher is better. The card’s identical-sentence example returns:

python
{'sari': 100.0}

The card’s three-reference example returns:

python
{'sari': 26.953601953601954}

SARI has an implementation gotcha that matters for reproducibility. Evaluate’s implementation is adapted from Tensorflow tensor2tensor and differs from the original Xu et al. codebase in two documented ways: 0/0 is defined as 1, which rewards exact matches, and the implementation includes a fix to an alleged bug in the keep score (SARI card, Xu et al. 2016). Those choices can make SARI numbers differ from the original implementation.

Gotcha catalog: BLEU, ROUGE, F1, accuracy, and SARI ​

MetricGotchaWhy it matters
BLEU vs sacreBLEUEvaluate BLEU returns bleu on a 0 to 1 scale. Evaluate sacreBLEU returns score on a 0 to 100 scale.A score can look off by a factor of 100 if you compare keys and scales carelessly (BLEU card, sacreBLEU card).
Reference shapeBLEU accepts unequal numbers of references per prediction. sacreBLEU requires the same number of references for each prediction.Shape errors can produce different results or invalid calls (sacreBLEU card).
ROUGE aggregationuse_aggregator=True returns floats. use_aggregator=False returns lists.Downstream code can break if it expects one output type (ROUGE card).
rougeL vs rougeLsumrougeLsum splits on newline characters before scoring.Multi-sentence summaries should use newline-joined sentences for comparable rougeLsum results (ROUGE card).
AccuracyThe card warns that accuracy can be misleading on unbalanced classes.A high accuracy can hide poor minority-class behavior (accuracy card).
F1 implementationEvaluate F1 forwards to sklearn.metrics.f1_score.For broader classification diagnostics, use scikit-learn directly (F1 source).
SARI implementationEvaluate SARI differs from the original implementation on 0/0 handling and a keep-score fix.Published SARI scores can differ across implementations (SARI card).

The companion deepdive also notes two boundaries that belong in this catalog: Evaluate has no built-in information_retrieval module, and Evaluate plus LightEval is a division of labor rather than a replacement story (Evaluate deepdive).

When Evaluate is the wrong tool ​

The decision cue is simple: if you have predictions and references already materialized and want overlap scores, stay in Evaluate. Otherwise route below.

JobBetter routeReason
LLM benchmark suitesLightEvalLightEval calls itself an "all-in-one toolkit" for evaluating LLMs across multiple backends, with 1000+ tasks per its README at review time. It is not a drop-in compute() replacement.
PyTorch training-loop metricsTorchMetricsTorchMetrics documents PyTorch metric implementations, distributed-training compatibility, batch accumulation, and multi-device synchronization.
Publishable machine-translation scoressacreBLEU standalonesacreBLEU focuses on shareable, comparable, reproducible BLEU scores and emits version signatures. Evaluate’s sacreBLEU module wraps it.
Broad classification diagnosticsscikit-learnEvaluate F1 forwards to scikit-learn, while scikit-learn also provides classification reports and confusion matrices.
Open-ended generation qualityDeepEval or the LLM judge field manualReference-free rubric judging is a different task from string overlap.

Evaluate still earns its keep when you have predictions and references already materialized and want one uniform interface across corpus-level NLG metrics such as BLEU, ROUGE, SARI, and related modules.

The larger blind spot remains: overlap metrics cannot tell whether a model reasoned, used a tool, or remembered a fact from three sessions ago. For memory-specific evaluation context, see evaluating agent memory.

Mnemoverse sits on that side of the boundary: persistent memory for agents that need to store, retrieve, and verify knowledge across sessions.

Common questions ​

What is the difference between BLEU and ROUGE? ​

BLEU is modified n-gram precision with a brevity penalty, mainly for translation. ROUGE is recall-oriented n-gram or LCS overlap, mainly for summarization.

What is the SARI metric and why does it need sources? ​

SARI is a text-simplification metric that compares a system output against both references and the original source sentence, so its Evaluate input requires sources, predictions, and references.

What does F1 macro vs micro vs weighted mean? ​

Macro averages per-class F1 scores equally, micro pools decisions before scoring, and weighted averages per-class F1 scores by class support.

Why doesn't my BLEU score match the paper? ​

BLEU scores can change with tokenization, normalization, parameters, dataset, language, implementation, and scale. Evaluate BLEU returns bleu on a 0 to 1 scale, while sacreBLEU returns score on a 0 to 100 scale.

Is the Hugging Face Evaluate library deprecated? ​

No. The repository was not archived at the 2026-08-17 review, and the latest release was v0.4.6 on 2025-09-18. Hugging Face recommends LightEval for newer LLM evaluation approaches.

Sources ​

Hugging Face Evaluate status ​

Metric cards and implementations ​

Alternative tools ​

— Edward Izgorodin · Last updated 2026-08-17

— Mnemoverse is a persistent-memory API for AI agents. Free key: console.mnemoverse.com · Docs: Getting Started