RAG systems need different QA because they are probabilistic: the same query can return different answers, and the typical failure is a fluent, confident, wrong answer that no assertion catches. Keep unit and integration tests for the plumbing, then add an evaluation layer: a versioned dataset of real queries with ideal context and answers, scored for faithfulness, answer relevance and groundedness, with retrieval measured separately from generation. Track hallucination rates across every change and block releases that regress, test prompt injection and permissions explicitly, run evals in CI on every prompt or model change, and keep sampling live answers after launch.
Why does conventional software testing break down for AI systems?
Quality assurance for AI systems is not an extension of conventional software testing, and the teams that treat it as one tend to find out the hard way. The test suite passes, the build goes green, the product ships, and then it hallucinates in front of a client. That sequence is almost always caused by the same misunderstanding: QA for RAG systems and AI agents needs a different mental model, different tooling and different metrics.
This is a practical framework for product teams and founders preparing to launch an AI product: where traditional testing fails, what to replace it with, and how to wire quality checks into your pipeline before you go live, not after the first incident report lands in your inbox.
The determinism problem
Conventional software testing rests on one load-bearing assumption: the same input always produces the same output. That is what makes assertions useful, regression suites trustworthy and green builds meaningful. RAG pipelines and LLM-powered agents are probabilistic by design. At standard temperature settings the same query can return a subtly different answer on consecutive calls, so a passing test today tells you very little about tomorrow. Even with deterministic decoding or fixed seeds, variability re-enters through retrieval results, context ordering and upstream model changes.
This is an architectural property of generative systems, not a defect waiting to be patched, and it means the test-assert-pass model has to be rethought.
What conventional QA misses in practice
A traditional bug is a deviation from a specified function: the button does not save, the calculation is wrong, the API call fails. An AI failure is often a plausible answer that is factually wrong, contextually irrelevant or dangerously confident. No assertion catches that, and unit tests alone will not surface it before production.
Coverage metrics, unit tests and integration checks are still necessary, but they validate plumbing, not reasoning. The gap they leave is where an AI-specific evaluation layer has to sit, on top of your existing test infrastructure.
| What changes | Traditional software | RAG systems and AI agents |
|---|---|---|
| Expected output | One exact value | A range of acceptable answers |
| Test unit | A function or code path | A behaviour across a class of inputs |
| Pass condition | Assertion passes | Score above an agreed threshold |
| Typical failure | Crash, wrong value, broken call | Fluent, confident, wrong answer |
| Test data | Fixtures | A versioned, labelled evaluation dataset |
| Regression check | Same tests, same result | Re-scored eval set compared with a baseline |
| After launch | Error and uptime monitoring | Plus sampling and scoring of live answers for drift |
How do you test the behaviour of a RAG system?
Design test cases around behaviour, not code paths
Prompt testing is the closest AI equivalent of unit testing, but instead of a function you are testing a behaviour. A good prompt test checks whether the system stays within expected boundaries across direct queries, edge cases, adversarial phrasings and ambiguous questions. The goal is not a pass or fail on a single response; it is a distribution of behaviours you can reason about and compare across versions.
For example, a prompt test for a customer-support RAG system might run fifty variants of "What is your refund policy?" across paraphrases, typos and changes of tone, then score each response for faithfulness and relevance. The question is not "did it return the right string?" but whether it stays within acceptable bounds across the whole class of input. The threshold you set - 90%, 95% or higher - should reflect your risk tolerance and use case.
Build an evaluation dataset before you build anything else
An evaluation dataset is a curated set of inputs with expected outputs, quality labels or reference answers. Without one you have no baseline when prompts, models or retrieval logic change, and you end up testing by intuition and shipping by optimism.
A minimum viable eval set for a RAG product holds, for each item:
- A real or realistic query- "What documents do I need to open a business account?"
- The ideal retrieved context - the relevant policy paragraph.
- The ideal answer - a concise, accurate summary drawn only from that paragraph.
Multiplied across dozens of query types, that gives you a baseline no amount of manual spot-checking can replicate. Version it alongside the codebase from the first commit. It is not a testing artefact but a product artefact: it defines what quality means for your system, and it becomes more valuable over time. We cover how to build one, and why it is the expensive part, in evaluating LLM output in production.
Which metrics should you use to evaluate a RAG system?
What faithfulness, relevance and groundedness measure
- Faithfulness measures whether the answer stays within the retrieved context. It catches the most common RAG failure: a model that ignores its sources and invents an answer.
- Answer relevancemeasures whether the response addresses the user's actual question, not just its topic.
- Groundedness, sometimes called attribution, checks whether every claim in the response can be traced back to a retrieved document.
Together they turn evaluation from a binary check into a comparable score you can track across releases. Depending on your use case and risk profile, fluency, toxicity and factual accuracy against external sources may need metrics of their own.
Measure retrieval separately from generation
One of the most common QA mistakes in RAG systems is evaluating retrieval and generation as a single unit. They fail for different reasons. Poor retrieval produces right-sounding answers to the wrong question; poor generation produces wrong answers from the right context.
Measure retrieval with context recall and context precision, and generation with faithfulness. That localises failures to the component that caused them. Treat them as one unit and you fix the wrong thing half the time, then spend the next sprint wondering why the scores did not move. The retrieval side is covered in more depth in how to implement retrieval augmented generation.
How do you test for hallucination regressions?
Track hallucination rates across versions
Most hallucinations are not random. They tend to be repeatable failure patterns under specific conditions: particular query types, ambiguous phrasing, or topics at the edge of the retrieval corpus. Once you have found the hallucination-prone query types in your eval set, track them as a regression metric. Every time the model, prompt template, retrieval strategy or document corpus changes, re-run the eval set and compare hallucination rates with the previous version.
A meaningful increase - whether you define that as an absolute delta, a relative change or a move outside your established confidence interval - blocks the release. Not a note for the post-mortem, not a ticket for the next sprint: a build-blocker.
Security and integration tests specific to AI workflows
Prompt injection is a risk many teams underestimate until it surfaces in production. It happens when user input, or content inside a retrieved document, manipulates the system prompt or causes an agent to take unintended actions. For agents with tool access, test it explicitly: can a crafted input make the agent call a tool it should not, leak data from one user session into another, or bypass permission checks? In a RAG system, also test that retrieval respects document permissions, so one user's query can never surface another customer's data.
Integration tests for multi-step agent workflows should verify that tool calls happen in the right order, that error states are handled, and that the system degrades predictably when an external API is down. These are part of the baseline for any agent system that touches real user data or external services, not checks to add later.
How do you validate an AI system continuously, before and after launch?
Wire eval runs into your deployment pipeline
Every commit that touches a prompt template, retrieval logic, model version or system configuration should trigger an automated eval run. It works like a regression suite in a conventional CI pipeline: current scores are compared with a baseline, and a significant drop in faithfulness, relevance or groundedness fails the build. Quality gates stay active without a manual review of every change, and subtle regressions stop accumulating silently until they surface in production.
Treat post-launch monitoring as part of QA
Quality assurance for AI systems does not end at launch, because production inputs differ from your eval set in ways you cannot fully predict. Log real queries and responses, sample them for review, track metric distributions over time and alert on drift.
A system that was accurate at launch can degrade when the document corpus changes, when the model provider updates the base model, or when users start asking things your eval set did not anticipate. Monitoring catches that before users do. When it finds a failure, the case goes into the eval set, so the same failure cannot ship twice.
How does MacroCoderz build QA into an AI delivery?
We treat quality assurance as a structural concern from the first day of an engagement, not a final-phase activity. Every fixed-price RAG build and embedded engineering pod starts with evaluation dataset design, before the prototype exists. The eval set is committed to your repository alongside the code, so you own your quality baseline with no dependency on us. That matters because it is the most valuable artefact for maintaining an AI system long after the initial build: it lets your team run its own checks, onboard new engineers and make informed decisions when you swap models or change retrieval strategy.
Every build ships with the harness, not just the code that passes it: the labelled dataset, the scoring functions for retrieval and generation, hallucination regression checks, the thresholds file and the CI configuration that runs them on every change in your own infrastructure, with a written note on how to add a case. If you are preparing to launch an AI product and want to talk through how this applies to your system, get in touch.
What is the short version?
QA for RAG systems and AI agents is a discipline in its own right. Teams that treat it as an extension of conventional testing get surprised in production. Teams that invest in eval datasets, behavioural metrics, hallucination regression checks and CI-integrated validation are far better placed to ship products that hold up under real use.
Start with a versioned eval set. Measure retrieval and generation separately. Put hallucination regression into your deployment pipeline. Build monitoring in from the first release. Everything else is refinement on top of that foundation - without it you are shipping on optimism, and optimism is not a quality management system.