When a retrieval-augmented generation system gives a bad answer, the instinct is to rewrite the prompt or switch to a larger model. Often the real problem sits earlier: the right passage never reached the model. Debugging goes faster when you separate the pipeline into stages and check each one in order.

Split the Problem in Two

Every bad answer falls into one of two buckets.

• Retrieval failure: the passages needed to answer were not in the context given to the model.
• Generation failure: the right passages were present, but the model ignored, misread, or contradicted them.

The first test is simple. For a failing question, find the passage that contains the answer by hand, then check whether it appears in what your system retrieved. If it does not, you have a retrieval problem, and changing the prompt will not help. If it does, focus on the generation step.

Build a Failure Log

Collect failing questions in a spreadsheet with columns for the question, the expected source, what was retrieved, and the diagnosed cause. Patterns emerge quickly. Ten failures that all trace to table content tell you something very different from ten failures that trace to ambiguous questions. This log also becomes your regression suite.

Retrieval Failure Causes

Chunking problems

Chunks that are too small lose the context needed to interpret them. Chunks that are too large dilute the signal and waste context space. Splitting mid-sentence, separating a heading from its content, or breaking a table across chunks are frequent culprits. Try structure-aware splitting that respects headings and paragraphs, and consider adding the document title or section heading to each chunk.

Poor document parsing

If PDFs, slides, or HTML pages are converted to text badly, the embedded content may contain scrambled columns, repeated headers, or missing tables. Inspect the extracted text itself before blaming the model. Garbage in the index guarantees weak retrieval.

Embedding mismatch

A general embedding model may represent your domain’s vocabulary poorly. Questions written as natural language may also sit far from formal document text in embedding space. Test alternative embedding models on your failure log, and keep in mind that changing models means re-embedding the whole corpus.

Missing keyword sensitivity

Exact identifiers, acronyms, and rare terms are weak spots for pure semantic search. Adding keyword search alongside vectors, then merging results, often helps.

Filters and metadata

An overly restrictive filter, a wrong tenant tag, or stale metadata can exclude the right chunk entirely. Check what filters were applied for the failing query.

Too few or too many results

Retrieving too few passages risks missing the answer. Retrieving too many buries it and increases the chance of the model getting distracted. Tune the number retrieved, then use a reranker to select the best subset to pass on.

Symptom and Fix Table

Symptom	Likely stage	Things to try
Answer exists in corpus but never retrieved	Chunking or embedding	Adjust chunk size, add headings, test other embedding models
Retrieved text is garbled	Parsing	Improve extraction, handle tables and columns explicitly
Exact codes or names missed	Search method	Add keyword or hybrid search
Right passage retrieved but ranked low	Ranking	Add a reranking step, tune the number of candidates
Right passage present, answer still wrong	Generation	Tighten the prompt, require citations, reduce irrelevant context
Answers cite outdated content	Index freshness	Fix update and deletion sync with the source

Generation Failure Causes

When the right context is present and the answer is still wrong, examine the prompt and the context layout. Instruct the model to answer only from the provided passages and to say when the information is missing. Ask it to cite which passage supports each claim, which makes unsupported statements easier to spot. Also consider whether conflicting passages are present; if two documents disagree, the model needs guidance about which to trust, such as preferring the more recent one.

A Hypothetical Example

A team’s assistant keeps failing on questions about a leave policy. Reviewing the failure log, they find the answer sits in a table inside a PDF. The extraction step flattened the table into a sequence of unlabeled cells, so the chunk text said nothing meaningful about which number applied to which category. Retrieval never surfaced it because the embedding of that chunk was essentially noise. They change the parser to convert tables into labeled row-by-row text and prepend the section heading to each chunk. After re-indexing, the failing questions retrieve the correct chunk, and no prompt changes were needed.

Measure Retrieval Separately

Evaluate retrieval on its own before judging end-to-end answers. For each test question, record whether the expected passage appears among the retrieved results and where it ranks. Tracking this independently from answer quality tells you which change helped. If you only look at final answers, a retrieval improvement and a prompt tweak blur together, and you cannot tell which one mattered.

Frequently Asked Questions

Should I fine-tune the embedding model first? Usually not. Fix parsing, chunking, and search method first, since they are cheaper and often account for more of the problem.

How big should chunks be? There is no universal answer. It depends on your content and the questions asked. Test a few sizes against your failure log and pick based on retrieval results.

Is reranking worth the added latency? Often it improves results noticeably when the first-stage retrieval is decent but ordering is off. Measure the gain against your latency budget.

Why does the model hallucinate even with good context? It may not be instructed to stay within the context, or the context may include conflicting or irrelevant passages. Constrain the prompt and trim noise.

How do I know when retrieval is good enough? Set a target on your test set for how often the expected source appears in the top results, based on what your use case tolerates, and track it over time.

Next Step

Take ten recent bad answers, check by hand whether the correct passage was retrieved for each, and sort them into retrieval and generation failures before changing anything.