RAGAS Metrics Explained: What Each One Needs Before It Can Score Anything
TL;DR
- RAGAS is an open-source Python framework for scoring retrieval-augmented generation pipelines, named in its founding paper "a framework for reference-free evaluation of Retrieval Augmented Generation (RAG) pipelines". Founding paper
- Of the six text metrics in today's RAG category, Faithfulness and Response Relevancy need no reference; Context Precision has a reference-free path (
ContextUtilization) and a reference-requiring one (ContextPrecision); Context Recall, Context Entities Recall and Noise Sensitivity need a reference in every documented mode. Metric index- Every judged metric takes an evaluator model as
llm=, and tokens go uncounted unless you attach aTokenUsageParser, shipped only for OpenAI. Cost howto- The widest unit RAGAS models is one conversation: one
SingleTurnSampleinteraction or oneMultiTurnSampleexchange. No built-in metric is a cross-session measure. Sample schema · Where evaluation tools stop
Start with the dataset you can supply. Do you have a question and a response? Retrieved passages? A reference answer? Those columns decide which RAGAS measurement you can make.
In production you cannot choose an evaluation metric only by what it measures. You choose it by what data you can feed it. So read RAGAS by what each metric demands as input, not by its name or its reputation.
RAGAS is an open-source Python framework for scoring retrieval-augmented generation pipelines. Shahul Es, Jithin James, Luis Espinosa Anke and Steven Schockaert published it in the system-demonstrations track of EACL 2024, pages 150 to 158. Its abstract promises metrics usable "without having to rely on ground truth human annotations". That described the paper's three metrics, not every metric shipping today. ACL Anthology · arXiv
Version discipline, stated once. Every statement about library behavior below comes from docs.ragas.io read on 2026-09-19 on the en/stable channel: a moving target, not a version pin, with no version selector and one footer date on every page cited here, December 9, 2025. The shipping release is ragas 0.4.3, on PyPI since 2026-01-13, from vibrantlabsai/ragas, which the old explodinggradients address returns an HTTP 301 to. The paper keeps its own clock: arXiv v1 September 2023, v2 April 2025, about nineteen months apart, and the paper's three metric names do not map one to one onto today's: Answer Relevance is now Response Relevancy, and Context Relevance now names a different, Nvidia-family metric. A sentence crossing from 2023 to 2026 has to say which era it means. PyPI
RAGAS metrics: what each one needs as input
The metric index lists seven category headings. This article covers one, Retrieval Augmented Generation, and inside it the six text metrics; the category also holds Multimodal Faithfulness and Multimodal Relevance, not assessed here. Context Precision takes two rows, its LLM-judged variants falling on opposite sides of the reference column.
| Metric | Needs a reference? | Inputs | Judge |
|---|---|---|---|
| Faithfulness | No | user_input, response, retrieved_contexts | LLM |
| Response Relevancy | No | user_input, response | LLM and an embedding model |
Context Precision (ContextUtilization) | No | user_input, response, retrieved_contexts | LLM |
Context Precision (ContextPrecision) | Yes | user_input, reference, retrieved_contexts | LLM |
| Context Recall | Yes, every documented mode | LLM mode: user_input, retrieved_contexts, reference. Non-LLM mode: retrieved_contexts, reference_contexts. ID mode: retrieved_context_ids, reference_context_ids | LLM, string or ID comparison |
| Context Entities Recall | Yes | reference, retrieved_contexts | LLM entity extraction |
| Noise Sensitivity | Yes | user_input, reference, response, retrieved_contexts | LLM |
Two of the six are strictly reference-free. One splits into two named paths. Three need a reference in every documented mode. Outside this category the Nvidia family adds two more reference-free metrics, Context Relevance and Response Groundedness, plus Answer Accuracy, which needs one. So "only two metrics in RAGAS are reference-free" is false without its category qualifier, and the Nvidia Context Relevance is not the founding paper's metric of that name. Nvidia metrics
Needing a reference is not the same as needing a human annotator: a reference can be generated.
Pick the row from the failure you suspect. Retriever ranking is Context Precision, retriever coverage is Context Recall, generator invention is Faithfulness, generator drift off the question is Response Relevancy. Predict the reference column before you build the dataset.
RAGAS faithfulness and response relevancy: the two reference-free metrics
Faithfulness is the RAGAS metric that measures how factually consistent a response is with the retrieved context, scored 0 to 1, higher being better: the claims in the response supported by the retrieved context over the total claims in the response. Inputs are user_input, response and retrieved_contexts; no reference appears anywhere on the page.
from ragas.metrics.collections import Faithfulness
scorer = Faithfulness(llm=evaluator_llm)
result = await scorer.ascore(
user_input=question,
response=answer,
retrieved_contexts=chunks,
)
# result.value holds the 0 to 1 scoreNot executed for this article. It is the ragas.metrics.collections form the page documents, awaited because ascore is asynchronous and constructed with the evaluator model the metric requires. The SingleTurnSample form sits under the page's "Legacy Metrics API" heading, with the docs' own note that the API "will be deprecated in version 0.4 and removed in version 1.0", so a snippet copied from an older write-up is a snippet on the deprecated path.
A high faithfulness score indicates the generator did not invent claims relative to what the retriever returned. It does not indicate the retriever returned the right chunks, or that the answer addresses the question.
A legacy variant, FaithfulnesswithHHEM(llm=evaluator_llm), swaps in Vectara's HHEM-2.1-Open. It replaces only the verification step; an LLM still extracts the claims.
Response Relevancy is the RAGAS metric that measures how relevant a response is to the user input: it generates artificial questions back from the response, embeds each alongside the user input, and averages the cosine similarities. It is calculated from user_input and response, needs an LLM and an embedding model, and needs no reference.
Two details change how to read the number. The docs warn that the score "usually falls between 0 and 1" but that this "is not guaranteed due to cosine similarity's mathematical range of -1 to 1". And the paper is explicit about what the measurement leaves out: "our assessment of answer relevance does not take into account factuality, but penalises cases where the answer is incomplete or where it contains redundant information." A high relevance score says nothing about whether the answer is true. Paper, Section 3
Names depend on the API: the page title reads Response Relevancy, the first in-body heading Answer Relevancy, the collections class is AnswerRelevancy, the legacy class ResponseRelevancy. All are live. The legacy example there also fills in retrieved_contexts the formula never uses.
RAGAS context precision vs context recall
Context Precision is the RAGAS metric that evaluates the retriever's ability to rank relevant chunks higher than irrelevant ones for a given query. It comes in four shapes across six class names, an easy place to ship code that does not do what you think. ContextPrecision, legacy LLMContextPrecisionWithReference, judges chunks against a reference answer. ContextUtilization, legacy LLMContextPrecisionWithoutReference, judges them against the system's own response and is the documented choice "when you don't have a reference answer". NonLLMContextPrecisionWithReference and IDBasedContextPrecision drop the LLM, but not the reference data: in place of a reference answer they take reference_contexts or reference_context_ids.
Context Recall is the RAGAS metric that measures how much of a reference answer the retrieved context covers, and the docs state the constraint outright: "calculating context recall always requires a reference to compare against." Its LLM mode decomposes the reference into claims and divides those supported by the retrieved context by the total; it "uses reference as a proxy to reference_contexts, which makes it easier to use as annotating reference contexts can be very time-consuming". The non-LLM modes ask for exactly that annotation: NonLLMContextRecall compares retrieved_contexts with reference_contexts, and IDBasedContextRecall compares retrieved_context_ids with reference_context_ids.
Notice the denominator. Faithfulness starts from what the system said. Context Recall starts from what the reference says it needed to retrieve. Precision and recall then approach retrieval from opposite angles: ranking accuracy against coverage.
Context Entities Recall divides the entities shared between the retrieved contexts and the reference by the entities in the reference. Noise Sensitivity "measures how often a system makes errors by providing incorrect responses when utilizing either relevant or irrelevant retrieved documents", takes user_input, reference, response and retrieved_contexts, and is the one row where lower is better. Do not apply a higher-is-better convention across the entire table.
The judge underneath every RAGAS score, and what it costs
Every LLM-judged metric in that table is constructed with an evaluator model passed as llm=, and Response Relevancy adds an embedding model. The string and ID comparisons are exceptions to the judge dependency, not extra reference-free paths. The number you read is a judge's estimate. Liu et al., in a February 2025 paper on RAG-judge reliability, state it plainly:
"LLM-based judgment models provide the potential to produce high-quality judgments, but they are highly sensitive to evaluation prompts, leading to inconsistencies when judging the output of RAG models."
That is a finding about the mechanism every judged RAGAS metric depends on, not a RAGAS benchmark result: the paper does not name RAGAS. The bias and leniency patterns live in LLM-as-a-Judge: Bias, Leniency & the LoCoMo Number. Study
A judge model also means a token bill, and RAGAS does not count it for you: "By default, Ragas does not calculate the usage of tokens for evaluate()." The reason given is that langchain's LLMs do not report usage uniformly; the fix is a TokenUsageParser. The library ships one, get_token_usage_for_openai from ragas.cost. Every other provider is bring-your-own.
The documentation's own worked example is worth reading closely, because it does not reconcile. Scoring a twenty-row dataset with context recall, the page prints TokenUsage(input_tokens=25097, output_tokens=3757, model=''). It quotes GPT-4o list prices of $5 per million input tokens and $15 per million output tokens, calls result.total_cost(cost_per_input_token=5 / 1e6, cost_per_output_token=15 / 1e6), and prints 1.1692900000000002. Those token counts at those prices give 25,097 × 5/1e6 + 3,757 × 15/1e6 = 0.18184, about $0.18. The printed total and the printed inputs do not agree on the page as read on 2026-09-19. The page offers no reconciliation, and this article does not guess at one.
With the page's printed total and its own token counts more than six times apart, neither belongs in a budget. The only figure worth a budget line is the one you measured yourself, with a TokenUsageParser on your own rows. Cost howto
What the RAGAS WikiEval numbers establish
The founding paper validated its three metrics on WikiEval, built from 50 Wikipedia pages covering "events that have happened since the start of 2022". Two annotators judged every item, and the judgment was not a score. It was a pairwise choice between the normal output and a deliberately degraded counterpart.
| Paper-era metric | Degraded counterpart | Ragas | GPT Score | GPT Ranking |
|---|---|---|---|---|
| Faithfulness | Answer generated with no context | 0.95 | 0.72 | 0.54 |
| Answer Relevance | Answer prompted to be incomplete | 0.78 | 0.52 | 0.40 |
| Context Relevance | Context padded with back-link sentences | 0.70 | 0.63 | 0.52 |
Every figure there is a pairwise-preference agreement rate: the fraction of comparisons where the metric picked the same item the annotators did. None is absolute accuracy, and none validates the library shipping today. The paper never says how many comparisons produced those percentages, and it carries no limitations section and no claim that the result extends beyond Wikipedia.
The authors name their weakest dimension themselves: "We found context relevance to be the hardest quality dimension to evaluate. In particular, we observed that ChatGPT often struggles with the task of selecting the sentences from the context that are crucial, especially for longer contexts." That limit belongs to the paper's sentence-selection Context Relevance, not to the Nvidia-family metric of the same name today. Paper, Sections 4 and 5
RAGAS test set generation
Scoring is one half of RAGAS. The other half builds the dataset you score, and it is the more involved half. It is knowledge-graph based: documents are split into hierarchical nodes, LLM-based or rule-based extractors pull information from each node, a relationship builder links nodes on that information, and apply_transforms runs the chain "in a sequence". Personas come from generate_personas_from_kg(kg=kg, llm=llm, num_personas=5), where the keyword is num_personas and a wrong guess ships code that does not run. Generation concepts · Persona howto
Each generated row carries a query, a context and a reference. Generated references address dataset construction. They do not turn a reference-dependent metric into a reference-free one.
What RAGAS does not measure: agent memory
An evaluation dataset in RAGAS is a list of SingleTurnSample or MultiTurnSample instances. MultiTurnSample is "a multi-turn interaction between Human, AI and optionally a Tool", so this is not a single-turn-only framework. The widest unit it models is one conversation. Dataset · Samples
Across sixteen documentation pages fetched on 2026-09-19, the strings memory, session, recency, consolidat and contradict return zero case-insensitive matches, with one irrelevant exception on the Context Entities Recall page. The agentic metrics score one trajectory's tool use and goal completion inside a single run or conversation. No built-in RAGAS metric is a cross-session or longitudinal measure. Context Precision evaluates the passage in front of the model at rank k, not the knowledge an agent retained from last Tuesday. Agentic metrics
This site placed that boundary already. LangChain & LangSmith Evaluation: The Memory Blind Spot lists RAGAS with DeepEval, TruLens, Braintrust and OpenAI Evals, and closes the list with "Useful tools, all of them. All scoped to outputs."
Use RAGAS to inspect the retrieved evidence and the generated response in a memory-backed application. Do not treat that score as proof that the memory works across time: per-response scores alone do not establish persistence, consolidation, contradiction handling or recency across sessions. That takes a longitudinal protocol on top of per-response metrics. How to Evaluate AI Agent Memory defines the problem as measuring "whether an agent's memory behaves correctly across many sessions", naming recall, consolidation, contradiction handling and recency, and explicitly "not whether any single response is good", and the benchmark map lives there and in the field guide.
Use the input column to pick the metric, and budget a judge model. When the question is whether a fact survived from three sessions ago, that is a different instrument.
Common questions
What is RAGAS?
RAGAS is an open-source Python framework for scoring RAG pipelines; its own paper expands the name as Retrieval Augmented Generation Assessment. It was published as a system-demonstration paper at EACL 2024, and ships today as the Apache 2.0 package ragas, version 0.4.3, from github.com/vibrantlabsai/ragas.
Is RAGAS reference-free?
Partly. Of the six text metrics in the retrieval-augmented-generation category, Faithfulness and Response Relevancy need no reference. Context Precision splits: ContextUtilization scores chunks against the system's own response, ContextPrecision needs a reference answer. Context Recall, Context Entities Recall and Noise Sensitivity require a reference in every documented mode. Needing a reference is not the same as needing a human annotator: a reference can be generated.
What does the RAGAS faithfulness metric measure?
Faithfulness measures how factually consistent a response is with the retrieved context: the number of claims in the response supported by the retrieved context, divided by the total number of claims in the response, scored 0 to 1, where higher is better. It needs user_input, response and retrieved_contexts, and no reference.
What is the difference between context precision and context recall in RAGAS?
Context Precision asks whether the relevant chunks were ranked near the top of what the retriever returned; its ContextUtilization path compares them against the system's own response instead of a gold answer. Context Recall asks whether the retrieved context covers the claims in a reference answer, and the docs say it always requires a reference.
Does RAGAS evaluate agent memory?
No built-in RAGAS metric is a cross-session or longitudinal measure. The widest unit RAGAS models is one conversation: a SingleTurnSample is one interaction, a MultiTurnSample one multi-turn exchange. Across sixteen documentation pages checked on 2026-09-19, nothing defines persistence across sessions, consolidation, contradiction handling or recency. You can score a memory-backed agent's responses with RAGAS; those per-response scores alone do not establish that the memory works across time.
How much does it cost to run RAGAS?
RAGAS does not count tokens by default: it needs a TokenUsageParser and ships one only for OpenAI. Its own twenty-row context-recall example reports 25,097 input and 3,757 output tokens and prints a total of 1.1692900000000002, while those counts at the GPT-4o rates the page quotes come to about $0.18. The page does not reconcile as read on 2026-09-19, so treat neither figure as a budget: attach a TokenUsageParser and measure your own rows, remembering the bill scales with row count and with how many judged metrics you run.
Sources
All read 2026-09-19.
- Paper: abstract and versions, full text, EACL 2024 record, WikiEval dataset.
- Project: GitHub, PyPI.
- Metrics, stable channel, pages last updated December 9, 2025: index, Faithfulness, Response Relevancy, Context Precision, Context Recall, Context Entities Recall, Noise Sensitivity, Nvidia family, agentic metrics.
- Mechanics: cost howto, evaluation dataset, evaluation sample.
- Test set generation: RAG concepts, persona howto.
- Judge dependency: Liu et al., Judge as A Judge, arXiv:2502.18817.
Related
- DeepEval: Pytest for LLMs
- LangChain & LangSmith Evaluation: The Memory Blind Spot
- How to Evaluate AI Agent Memory
- LLM-as-a-Judge: Bias, Leniency & the LoCoMo Number
- Hugging Face Evaluate Library: load(), compute() Guide
Edward Izgorodin, September 2026 · LinkedIn
— Mnemoverse is a persistent-memory API for AI agents. Free key: console.mnemoverse.com · Plans and limits · Docs: Getting Started