RAGA Framework, Part 5: How to Fix a RAG Pipeline Using Evaluation Results

Part 5 of a series on evaluating LLM apps with Ragas.

How to Fix a RAG Pipeline Using Evaluation Results - Infographic - vinay c

By now we have a representative test set and honest scores. In article 3 the numbers pointed at retrieval, specifically low recall on table-based questions. This article is about what you do next: change the system on purpose, measure whether the change helped, and know what it cost you.

This is the step where most people abandon rigor. They read a low score, change four things at once, see the number go up, and declare victory, with no idea which change did it or what it broke. Then next month something regresses and they can't reason about why. The fix is boring and it works: change one variable at a time.

The knobs you can turn

A RAG pipeline has a lot of adjustable parts. The ones I actually end up touching:

• Chunk size and overlap
• Number of chunks retrieved (top-k)
• Embedding model
• Search strategy (dense, keyword, hybrid)
• A reranker on top of retrieval
• Prompt instructions
• Generation model
• Temperature
• How context is formatted into the prompt

Faced with a low score, the temptation is to reach for the generation model, because that feels like the powerful lever. But article 3 told us recall was the problem, and no generation change improves recall. So we start upstream, at chunking and retrieval.

One change at a time

Here's the loop, stated plainly:

Baseline → hypothesis → one change → re-evaluate → compare → decide

Applied to our case:

• Baseline: recall was 0.61, failing mostly on table questions.
• Hypothesis: the chunker splits tables mid-row, so leave-balance and notice-period numbers get scattered across chunks and retrieval can't find a complete answer.
• Change: switch to a table-aware chunker that keeps each table intact, and nothing else.

Running it as a Ragas experiment

Start with the purpose. An "experiment" in Ragas is one saved run of your system over the whole test set under a single configuration: your baseline, the table-aware chunker, a new model.

Each run is written to disk as a labeled record. That's what makes it useful: you can pull up "baseline" and "table-aware chunker" side by side later and compare like-for-like, instead of trusting a scatter of notebook runs you can't reconstruct.

The @experiment decorator gives you this with almost no plumbing. The work splits cleanly:

• You write: one async function for a single row — run your system on that question and return its outputs plus some metadata (which experiment, which chunker, and so on).
• Ragas does the rest: applies your function to every row, collects the results, and saves them to disk.

You describe how to handle one question; the framework runs the whole set and records it.

from ragas import experiment, Dataset

@experiment()
async def table_aware_chunking(row):
    # `row` is a SINGLE test case (one dict). Ragas calls this function
    # once per row for you — you never loop over the dataset yourself.
    contexts = policy_assistant_v2.retrieve(row["user_input"])   # v2 = table-aware chunker
    answer = policy_assistant_v2.answer(row["user_input"], contexts)
    return {
        **row,
        "retrieved_contexts": contexts,
        "response": answer,
        "experiment_name": "table_aware_chunking_v1",
        "chunker": "table_aware",
    }

dataset = Dataset.load(name="policy_eval", backend="local/csv", root_dir="./data")

# .arun() passes the WHOLE dataset; Ragas runs table_aware_chunking on each
# row and collects every returned dict into `results`.
results = await table_aware_chunking.arun(dataset) 

Results get written to a timestamped CSV under experiments/, so every run is a record you can go back to. That trail matters more than it sounds: three weeks later, "why did we pick the table-aware chunker?" has an answer sitting in a file instead of in someone's memory.

Compare on more than the headline metric

When you compare the new run to the baseline, don't look only at the score you were trying to move. Look at four things:

• Evaluation scores — did recall go up, and did anything else go down?
• Cost — more chunks or a bigger model means more tokens per query.
• Latency — a reranker adds a network hop; users feel it.
• Error categories — did the type of failure change?

For our chunking change, suppose recall jumps from 0.61 to 0.79 and faithfulness holds steady. That's a clear win, and cost and latency barely move because we changed how documents are split, not how many we retrieve or which model runs. Easy decision: keep it.

The interesting decisions are the ones with tradeoffs. Say you then add a reranker and recall climbs another few points, but p95 latency goes from 1.2s to 2.1s and cost per query doubles. Now it's a judgment call, and the right answer depends on the product, not the metric. A compliance tool might happily pay 2 seconds for accuracy. A chat widget might not. The point is that Ragas gives you the accuracy half of that tradeoff in numbers instead of vibes, so the conversation is grounded.

Write down what you were testing

Every experiment should record what changed and why. The metadata fields in the example above (experiment_name, chunker) aren't decoration. Six weeks from now, a table full of experiment_v1, experiment_v2, experiment_final_FINAL tells you nothing. table_aware_chunking_v1 tells you what the run was for. Cheap habit, huge payoff.

Failed experiments are the useful ones

Here's the thing nobody puts in blog posts: most of your experiments won't work, and those are the valuable ones. When I swapped in a "better" embedding model expecting recall to jump, it moved by 0.01 and latency got worse. That non-result was genuinely useful, because it killed a plausible-sounding idea in an afternoon and stopped me from carrying "we should upgrade embeddings" around as a someday-task. A polished demo where everything works teaches you nothing. A failed experiment tells you where the real constraints are.

Report them to yourself honestly. "I tried X, expected Y, got nothing, moved on" is a complete and respectable result.

Interview angle

The question: "You've evaluated your RAG system and recall is low. Walk me through improving it."

The answer they're listening for is a disciplined loop, not a list of tricks:

• Form a hypothesis from the failure pattern, don't guess randomly.
• Change one variable at a time.
• Re-evaluate on the same fixed test set, so the comparison is fair.
• Compare across quality, cost, and latency — not just the metric you were trying to move.
• Start upstream (chunking, retrieval), because a generation fix can't recover missing evidence.

One line takes it from textbook to lived-in: "and I'd keep the failed experiments as evidence." Saying that signals you've actually done this rather than just read about it.

The takeaway

Improving a RAG system is a controlled loop, not a burst of simultaneous edits. Change one variable, measure against a fixed test set, weigh quality against cost and latency, and keep the record, especially of what didn't work.

There's a large assumption underneath everything so far: that the LLM doing the scoring can be trusted. It's an LLM judging an LLM. The next article takes that apart, because if the judge is biased or noisy, every decision in this article rests on sand.