A RAG chatbot is probably giving wrong answers if it never says it doesn't know, its citations don't contain the fact it states, it quotes outdated documents, it misreads tables or scanned PDFs, it misses exact product codes, its answer changes when you rephrase, or users ask again or ask for a human. To check, take 30 real questions with known answers, look at what was retrieved before judging the answer, and score retrieval and answers separately. Most fixes are in parsing, indexing and search, not the prompt.
Why does a RAG chatbot give wrong answers without anyone noticing?
A retrieval-augmented (RAG) chatbot answers by finding passages in your documents and asking a model to write a reply from them. When it goes wrong, it rarely says "I don't know" or throws an error. It writes a fluent, confident answer from the wrong passage, an outdated passage or no passage at all, and the reply reads exactly like a right one.
That is why these failures are quiet. Uptime is fine, latency is fine, users get an answer every time. Most teams find out from a customer who acted on it. The seven signs below are the ones we look for first when we are asked to review a RAG system someone else built.
What are the signs a RAG chatbot is giving wrong answers?
1. It never says it doesn't know
Ask it something your documents do not cover - a product you do not sell, a policy you do not have. A healthy system says it cannot find that. A system that answers anyway is filling gaps from the model's general knowledge, and it is doing the same thing on real questions where you cannot tell.
2. Its citations don't support the answer
If it shows sources, open five of them. Does the cited passage actually say what the answer says? A citation that is on the right topic but does not contain the specific fact is one of the most common failures, and it is worse than no citation because it makes the answer look checked.
3. It quotes last year's version of a document
Change a price or a policy, then ask about it. If it still gives the old answer, the index is not being updated when documents change, or old and new versions are both indexed and the old one wins. Stale content is the single most common cause of wrong answers in systems that were correct at launch.
4. It gets tables, numbers and scanned PDFs wrong
Ask a question whose answer sits in a table - a price list, a specification sheet, a schedule. Many document pipelines flatten tables into a jumble of text or skip scanned pages entirely. The model then guesses the number, and a guessed number looks exactly like a real one.
5. It misses exact names, codes and acronyms
Search for a product code, an error number or an internal acronym. Retrieval based only on meaning (embeddings) is good at paraphrase and poor at exact strings, so it returns something that is about the right topic but not the right item. This is fixed by hybrid search, which adds keyword matching - covered in our vector database notes.
6. Rephrasing the question changes the answer
Ask the same question three ways. If the substance of the answer changes, the system is sensitive to wording in a way your customers will hit by accident. It usually means the right passage is only just making it into the results, and small changes push it out.
7. Users ask again, or ask for a human
Look at your conversation logs. A user who rephrases the same question straight after an answer, or asks for a person, is telling you the answer did not help. The share of conversations that end this way is the cheapest real-world accuracy signal you have, and most teams never measure it.
How can you check a RAG chatbot's accuracy in an afternoon?
You do not need an evaluation platform to find out whether you have a problem. You need thirty real questions and a few hours.
- 01
Collect 30 real questions
Pull them from support tickets, chat logs or sales emails - not ones you invent. Include ten awkward ones: tables, codes, recent changes, things you do not cover.
- 02
Write the right answer for each
And note which document it comes from. This takes longest and is the most valuable part. Keep the list - it becomes your test set.
- 03
Check retrieval on its own
For each question, look at the passages the system retrieved before it answered. Is the right passage there at all? If not, no prompt will fix it.
- 04
Check the answers
Mark each answer right, wrong or partly right, and whether the citation supports it. Run each question twice to see whether the answer changes.
Score retrieval and answers separately. If the right passage was retrieved but the answer was wrong, the problem is the prompt or the model. If the right passage never arrived, the problem is chunking, indexing or search, and that is where most fixes are. Our post on QA for RAG systems explains why testing this differs from testing ordinary software.
What causes each failure, and how is it fixed?
| Symptom | Usual cause | Usual fix |
|---|---|---|
| Answers questions it should refuse | No instruction or threshold for low-relevance results | A relevance cut-off, and an explicit 'not found' path |
| Citation doesn't support the answer | Model writes beyond the passage; chunks too large | Smaller chunks, answer-only-from-context prompting, citation checks |
| Quotes outdated content | Index not refreshed; old versions still indexed | Re-index on change, delete superseded versions |
| Wrong numbers from tables or scans | Parser flattens tables, skips images | Table-aware parsing and OCR for scanned pages |
| Misses product codes and acronyms | Embedding-only search | Hybrid search (keyword plus semantic) and a re-ranker |
| Answer changes with wording | Right passage only just retrieved | Better chunking, query rewriting, re-ranking |
| Users rephrase or ask for a human | Any of the above | Measure it, then trace sample conversations back to a cause |
For the build side of each fix, see how to implement retrieval augmented generation.
How do you stop a RAG chatbot from drifting back into wrong answers?
The thirty questions from your afternoon check are the start of an evaluation set. Keep adding every question users get wrong answers to, and re-run the set whenever documents, prompts or the model change. Three habits keep a system honest:
- Re-run the test set automatically on every change, and block the change if the score drops.
- Track the share of conversations that end in a rephrase or a request for a human, weekly.
- Re-index whenever source documents change, and remove what they replace.
How we build and run that test set is in evaluating LLM output in production.
How do you know if your RAG chatbot is giving wrong answers?
Ask it things your documents do not cover, open its citations, change a document and ask again, and test tables, codes and rephrased questions. Then check your logs for users who ask again or ask for a person. Thirty real questions with known answers, scored for retrieval and for the answer separately, will tell you in an afternoon whether you have a problem and roughly where it is.
If the check turns up more than you want to fix yourself, our RAG development team reviews and repairs existing systems as well as building new ones.