Evaluating a RAG Pipeline with Ragas: A Minimal Working Example

Part 3 of a series on evaluating LLM apps with Ragas.

Evaulate RAG - RAGAs Framework - Infographic - By Vinay C

The last two articles were conceptual. This one is hands-on. We'll run a real Ragas evaluation on our policy assistant and, more importantly, read the output like an engineer instead of staring at numbers.

I'm going to keep the setup deliberately short. Most Ragas tutorials spend three quarters of their length on installation and API keys, and then two sentences on the results, which is backwards. The results are the whole job.

A note on naming and versions

pip install ragas. That's the package. As I mentioned in article 1, don't confuse it with RagaAI LLM Hub, which is unrelated.

One thing that trips people up: Ragas is moving APIs. There's a newer per-metric "collections" API (from ragas.metrics.collections import ..., called with await scorer.ascore(...)) and an older batch API (from ragas.metrics import ... used with evaluate(...)). The batch evaluate() flow is still what the official quickstart uses, so that's what I'll show here. Just know the legacy per-metric imports are slated for deprecation, so check the version you're on. I'll flag both.

To follow along, the whole setup is pip install ragas langchain-openai and setting your OPENAI_API_KEY in the environment. That's it. Everything else in this article is about what to do with the output, not how to install things.

The four inputs every Ragas sample needs

Whatever else changes, a single evaluation sample comes down to four fields:

• user_input — the question the user asked.
• retrieved_contexts — the chunks your retriever returned, as a list of strings.
• response — what your system actually answered.
• reference — the correct answer, where you have one (optional, but several metrics need it).

Your job before touching Ragas is to run your real pipeline over a set of questions and record those four things per question. That's it.

Building the evaluation data

Assume we already have a policy_assistant object with a .retrieve() and .answer() method. We loop our test questions through it and collect the four fields.

test_questions = [
    "How many weeks of paternity leave am I entitled to?",
    "Can I carry over unused vacation days to next year?",
    "What is the notice period for resignation?",
    # ... 15-20 of these
]

reference_answers = [
    "Employees are entitled to 4 weeks of paid paternity leave, which may be split.",
    "Up to 5 unused vacation days may be carried over into the next calendar year.",
    "The standard notice period for resignation is 30 days.",
    # ...
]

dataset = []
for question, reference in zip(test_questions, reference_answers):
    contexts = policy_assistant.retrieve(question)   # list[str]
    answer = policy_assistant.answer(question, contexts)
    dataset.append({
        "user_input": question,
        "retrieved_contexts": contexts,
        "response": answer,
        "reference": reference,
    }) 

Notice the reference answers are short and written by a human who knows the policy. Fifteen to twenty of these is enough to start. You do not need thousands to learn something useful.

Running the evaluation

from ragas import EvaluationDataset, evaluate
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from ragas.metrics import (
    LLMContextPrecisionWithReference,
    LLMContextRecall,
    Faithfulness,
    ResponseRelevancy,
)

evaluation_dataset = EvaluationDataset.from_list(dataset)
evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))
evaluator_embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings())

result = evaluate(
    dataset=evaluation_dataset,
    metrics=[
        LLMContextPrecisionWithReference(),
        LLMContextRecall(),
        Faithfulness(),
        ResponseRelevancy(),
    ],
    llm=evaluator_llm,
    embeddings=evaluator_embeddings,   # response relevancy needs an embedding model
)
print(result) 

An evaluator LLM does the judging for most of these metrics (and response relevancy also uses the embedding model to compare questions). That's a whole topic on its own, and article 6 is dedicated to whether you can trust it. For now, take the scores as informative but not gospel.

One practical note before you run it: this isn't free or instant. Twenty questions across four LLM-judged metrics is a few hundred model calls, so expect some token cost and a minute or two of waiting. That's another reason to start with 15–20 questions rather than thousands, until the loop is working and you trust it.

Reading the output

You get back something like:

{'llm_context_precision_with_reference': 0.78,
 'context_recall': 0.61,
 'faithfulness': 0.83,
 'answer_relevancy': 0.90} 

Here's where most people stop and where the actual work begins. Aggregate scores are a starting point, not a conclusion. Walk through them the way we did in article 2.

Recall is the lowest at 0.61. That says roughly four out of ten times, the evidence needed to answer correctly wasn't even retrieved. That's my first suspect. Precision at 0.78 is okay, so when the retriever does find things, they're mostly ranked sensibly, but recall is leaking. Faithfulness at 0.83 means the model occasionally states things the context doesn't support, worth watching but secondary. Relevancy at 0.90 tells me the model understands the questions fine.

The story the numbers tell: retrieval is the weak link, not generation. If I only looked at the fluent, on-topic answers, I'd never have guessed that.

Don't stop at the average, read the rows

The single most useful thing you can do is export per-row scores and sort by the worst.

df = result.to_pandas()
print(df.sort_values("context_recall").head()) 

The average hides everything interesting. When I did this on a real policy assistant, the low-recall rows clustered around one theme: questions whose answers lived in tables (leave balances, notice periods by tenure), and our chunker was splitting tables mid-row so the relevant numbers got scattered. The average score would never have told me "your table chunking is broken." The sorted rows did in about five minutes.

Interview angle

A practical question you'll hit, sometimes as a take-home: "You've built a RAG chatbot. How do you know it's hallucinating, and how would you catch it?"

The weak answer is "I'd read through some responses and check." The strong answer is that you don't eyeball it, you measure it, three moves:

• Score faithfulness on every response. It breaks each answer into claims and checks them against the retrieved context, so a low faithfulness score is your hallucination signal, at scale, not just on the handful you happened to read.
• Test the no-answer cases on purpose. Add questions whose answer isn't in the knowledge base and check the bot abstains instead of inventing, because hallucination shows up most on questions it should have refused.
• Read per-row, not the average. A healthy-looking 0.83 mean can hide a cluster of 0.2s on one topic.

The detail that marks you as someone who's actually done this: be precise about what faithfulness proves. It tells you the answer is grounded in the retrieved text, not that the text is correct. To check real correctness you'd add factual correctness against reference answers. Naming that limit, rather than claiming faithfulness "catches hallucination," is the senior signal.

The bigger version of this question, "design a chatbot that doesn't hallucinate," is a full system-design problem, not a measurement one. That's the capstone at the end of this series (article 10), because a good answer needs every piece we haven't covered yet.

The conclusion that matters

The score is not the result. The diagnosis and the change you make next are the result.

A run that produces context_recall: 0.61 and nothing else is almost useless. The same run, plus "the failures are all table-based questions because chunking splits tables," is a work item you can actually pick up.

Which raises the obvious question: those fifteen questions I hand-wrote, are they even the right questions? If my test set only contains easy queries, good scores mean nothing. That's the subject of the next article: building a test dataset that represents reality, and using Ragas to generate test questions from your own documents.