RAGA Framework, Part 1: Why and how can you evaluate your LLM App?

Part 1 of a series on evaluating LLM apps with Ragas.

RAGA Framework explained - part 1 - by Vinay C

The first RAG chatbot I built demoed beautifully. I asked it five or six questions, the answers came back fluent and confident, and everyone in the room nodded. We shipped it to a small pilot group. Within two days I had a list of answers that were wrong, half-right, or confidently made up. None of them were questions I had tried.

That gap between "looks good in the demo" and "actually works" is what this series is about. I want to write down what I learned the hard way, so you can skip the part where you find out in production.

Before anything else, one naming thing, because it tripped me up early. This series is about Ragas (sometimes written RAGAS), the open-source framework for evaluating RAG and LLM applications. There is a separate, unrelated project called RagaAI LLM Hub. Different tool, different company. When you search, you will hit both. We are talking about Ragas: the library you pip install ragas.

Why "it seems fine" is not a measurement

The instinct with an LLM app is to test it the way you'd test a person: ask a few questions, read the answers, form an impression. This feels reasonable and it is completely unreliable.

Here is the problem. You read ten answers, they look fine, you conclude the system is fine. But ten answers tell you almost nothing about the thousands of questions real users will ask. You also read those ten answers while knowing what the right answer is, which makes a plausible-sounding wrong answer very easy to wave through. And "fine" is not a number. You cannot compare today's "fine" against last week's "fine" or against the version you tried with a different chunk size. You are running on vibes.

Vibes work right up until they don't, and when they stop working you have no idea which change caused it.

A base model is not an LLM application

Early on I conflated two different things: how good the model is, and how good my app is.

The model, say GPT-4 or Claude, has already been benchmarked to death. Its raw capability is not your problem. Your problem is everything you wrapped around it: how you split your documents into chunks, how you retrieve the relevant ones, how you stuff them into the prompt, what instructions you give, how you handle a question with no answer in your data.

You can pair an excellent model with a bad retrieval setup and get garbage. The model was never the weak link. This matters because it tells you what to measure. Benchmarking the base model tells you nothing useful. You need to measure your pipeline, on your data, answering your users' questions.

Traditional accuracy doesn't fit

If you come from classic ML, your reflex is accuracy, precision, recall against a labeled test set. There is one right answer, you check whether the model got it.

LLM output breaks this. There are many correct ways to phrase an answer. An answer can be factually right but not address the question. It can be well written and quietly hallucinated. It can be grounded in your documents but useless to the person asking. A single accuracy score cannot see any of this, because "correct" for generated text is not one thing. It is several things at once: is it grounded, is it relevant, did retrieval even find the right information.

That is the gap Ragas fills. Instead of one blunt score, it breaks quality into dimensions you can measure separately and act on. We'll go deep on the specific metrics in the next article.

Offline evaluation vs. watching production

Two different jobs that people mix up.

Offline evaluation is what you do before shipping. You keep a fixed set of test questions, run your pipeline over them, and score the results. Because the test set is fixed, you can change one thing, rerun, and see whether your numbers moved. This is where you catch regressions before users do.

Production monitoring is what you do after shipping. Real users ask things you never imagined. You sample real traffic, watch for failures, and feed the interesting cases back into your test set. This is where you learn what you didn't know to test.

You need both. Offline eval without production monitoring means you only ever test your own assumptions. Production monitoring without offline eval means you find out about problems only after they've reached users. Ragas is built mostly for the offline side, which is where I'd tell anyone to start, because a solid offline loop pays for itself immediately.

Where Ragas actually sits

Think of the development cycle as a loop:

Build → Test → Diagnose → Improve → Retest

You build a version of the pipeline. You test it against your evaluation set. The scores tell you where it's failing, not just that it's failing. You form a hypothesis, change one thing, and retest to see if the change helped or hurt.

Ragas lives in the Test and Diagnose steps. It turns "the chatbot feels off" into "retrieval is finding the right documents but the model is ignoring them," which is something you can actually fix. The loop is the whole point. Without it, every improvement is a guess and every guess is unverifiable.

The one thing to take away

If you do nothing else after reading this: stop evaluating your LLM app by reading a handful of answers and deciding it seems fine.

Build a small fixed set of test questions, even fifteen or twenty to start, and commit to running your system against the same set every time you change something. That single habit is the difference between engineering and hoping.

In the next article I'll get concrete about the four core RAG metrics: context precision, context recall, faithfulness, and response relevancy. And more useful than the definitions, how to read them together to figure out why a system is failing, not just that it is.