RAGA Framework, Part 4: Your Evaluation Is Only as Good as Your Test Data

Part 4 of a series on evaluating LLM apps with Ragas.

Article content

LLM Evals - Generate Datset using RAGA Framework - by Vinay - infographic

In the last article we ran an evaluation on fifteen hand-written questions and learned that retrieval was our weak spot. But there's a problem I glossed over: I wrote those fifteen questions myself, and I know how the system works. That makes them too kind. Left to my own devices, I test the paths I already expect to work.

Your metrics are computed over your test set. If the test set doesn't look like reality, your scores are measuring the wrong thing, and a green dashboard means nothing. This article is about building a test set that actually stresses the system.

What real test data has to include

When I sat down and listed the kinds of questions employees actually send our policy assistant, versus the kinds I'd been testing, the gap was embarrassing. A test set that represents reality needs, at minimum:

Real user questions. Pulled from logs, phrased the way people actually type, typos and all.
Expert-written cases. Questions an HR expert knows are commonly misunderstood.
Hard and ambiguous questions. "How much leave do I get?" without saying which kind.
Questions with no answer in the documents. The correct response is "I don't know" or "that isn't covered." Systems love to hallucinate here.
Multi-document questions. Answers requiring you to combine two policies.
Temporal and policy-sensitive questions. "What's the 2026 contribution limit?" where an old document would mislead.
Adversarial phrasing. Leading questions, false premises, prompt-injection attempts.
Different personas. A new hire, a manager, and a contractor ask the same underlying thing very differently.

The "no answer in the documents" category is the one I'd most encourage you to add first. It's where confident hallucination does the most damage, and almost nobody tests it.

Generating test data with Ragas

Hand-writing all of that is slow, and you'll still miss cases. Ragas can generate a synthetic test set directly from your documents, which is a fast way to get broad coverage.

The interesting part is how it does it, because that's what makes the questions varied rather than trivial. Ragas builds a knowledge graph from your documents, enriches it with transformations that pull out entities, summaries, and links between chunks, and then generates questions from that graph using a mix of query types.

from ragas.testset import TestsetGenerator
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.document_loaders import DirectoryLoader

# Load your actual policy documents
docs = DirectoryLoader("policy_docs/", glob="**/*.md").load()

generator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))
generator_embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings())

generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)
testset = generator.generate_with_langchain_docs(docs, testset_size=30)

testset.to_pandas()   # inspect and curate before you trust it

Here's the payoff that's easy to miss: each generated row comes with more than a question. You get:

user_input — the question itself,
reference — a golden answer,
and the reference contexts it was built from.

Those are the same fields you hand-wrote in article 3. In other words, synthetic generation produces the golden answers for you, which is normally the slowest, most tedious part of building a test set. That alone is often worth the setup.

Start small: testset_size=30 to 50 is plenty to begin with. Grow it as you discover real failure patterns worth pinning down, not before.

Single-hop, multi-hop, and why the mix matters

Under the default settings, Ragas splits generation across query types, roughly: half single-hop specific questions, a quarter multi-hop abstract, a quarter multi-hop specific.

A single-hop question is answerable from one chunk: "How many weeks of paternity leave are there?" A multi-hop question needs information stitched from more than one place: "If I take paternity leave and then resign, how much notice do I owe?" That pulls from the leave policy and the resignation policy at once.

This split matters because single-hop and multi-hop questions fail for different reasons. Single-hop failures usually mean bad chunking or retrieval. Multi-hop failures often mean your retriever can find one relevant chunk but not assemble several. If your test set is all single-hop (which hand-written sets tend to be), you'll never see the multi-hop weakness, and multi-hop is where real users get let down.

Personas make questions sound like real people

Ragas also supports personas: you describe the kinds of users you have, and it generates questions in their voice. A "confused new hire" asks differently from a "detail-oriented manager checking compliance." This matters because retrieval quality is sensitive to phrasing. A system that nails formal questions can fall apart on the vague, casual way a stressed new employee actually types. Testing across personas surfaces that.

The warning I'd tattoo on this

Synthetic test data expands coverage. It does not replace real production queries and human-reviewed examples.

I've seen teams generate a thousand synthetic questions, score well, and ship something that fell over on day one, because the synthetic questions were all clean, well-formed, and answerable, while real users asked messy, half-formed, out-of-scope things. Synthetic generation draws its questions from your documents, so it tends to ask what your documents can answer. That's the opposite of the adversarial and no-answer cases you most need.

Use synthetic data for breadth. Then always fold in:

A sample of real queries from logs.
Human-reviewed hard cases.
Deliberate no-answer and adversarial questions.

Those no-answer cases you have to write by hand, since generation won't make them for you. Two easy sources: pull real out-of-scope questions from your logs, or ask about a topic you've confirmed isn't in the documents (last year's policy, a benefit you don't offer). The correct answer for each is "I don't know" or "that isn't covered," and that's exactly what you're checking the system does.

And curate. Generated questions include duds. Read them, drop the nonsensical ones, fix the references. Ten minutes of curation is the difference between a test set you trust and a number that lies to you.

Interview angle

"How would you build an evaluation set for a RAG system you're deploying at a client?" comes up in almost every forward-deployed interview, and for good reason: it's usually the first real thing you do once you're on the job. A strong answer covers three things:

The categories that must be represented — especially the three people forget: no-answer, multi-hop, and adversarial questions.
Synthetic generation for breadth — use it to cover a lot of ground fast, but say plainly what it misses: it only produces clean, answerable questions, so the adversarial and no-answer cases still have to come from you.
Real logged queries and human-curated hard cases folded in — this is where the adversarial and no-answer cases from the previous point come from, and it's what stops you from only testing your own assumptions.

The detail that sets a forward-deployed answer apart is the cold start. At a client you often land with zero labeled test data, and interviewers want to know how you'd start from nothing. The move:

Bootstrap from the client's own documents with synthetic generation, which gives you coverage and golden answers on day one.
Swap synthetic cases for real logged queries progressively, as traffic arrives.
Add personas and the single-hop/multi-hop distinction for bonus points.

The whole thing signals that you understand eval quality is upstream of everything else.

The takeaway

Before you trust a single score, ask: does my test set contain the questions that would actually break this system? Generate for breadth with Ragas, but seed it with real, hard, and unanswerable questions, and curate what comes out.

Now that we have honest scores on a representative set, we can start changing the system on purpose. The next article is about turning scores into engineering decisions: running controlled experiments, changing one variable at a time, and weighing quality against cost and latency.