How RAG works
RAG stands for retrieval augmented generation, and the name describes the order of operations. A question arrives. Before any answer is written, the system searches a defined set of material — a website, a document store, a help centre, an index of the open web — and pulls back the passages that look most relevant. Only then does a language model write, using those passages as its raw material.
The retrieval step usually works on meaning rather than exact words. Documents are cut into chunks, each chunk is turned into a numeric representation of what it is about, and the question is converted the same way, so a passage can be matched even when it shares no vocabulary with the query. The result is an answer that can quote a specific document and, in a well-built system, tell you which one.
Why RAG matters
It is the difference between a model answering from memory and answering from a source. A plain model has no store of facts to consult, so it produces the most plausible continuation and can invent freely. A retrieval system narrows the model’s job to summarising material that is actually in front of it, which cuts invention sharply and lets the answer be checked.
It matters commercially in two directions. Inside a business, retrieval is what makes an assistant answering from your own documents feasible without retraining anything. Outside it, retrieval is exactly what AI search does — which is why being retrievable, clearly written and correctly attributed on your own site is a visibility question and not only a technical one.
Where RAG goes wrong
Retrieval does not guarantee truth; it guarantees a source. If the documents are out of date, contradictory or badly written, the system will answer confidently from the wrong page, and the citation makes the wrong answer look more trustworthy than it is. Rubbish in, cited rubbish out.
The other failures are quieter. Chunking that splits a table or an argument in half retrieves fragments that mislead. Questions the material cannot answer often get answered anyway, because the model would rather summarise the nearest passage than say nothing. And a knowledge base nobody owns drifts out of date without any visible failure at all — the assistant carries on sounding perfectly fine.
How to build on it
Treat the documents as the product. Decide what is in the source set and what is deliberately excluded, remove superseded versions instead of leaving them beside the current one, and give each document an owner and a review date. Most disappointing assistants are content problems wearing a technology costume.
Then insist on visible sources and test the edges. Every answer should name the document it came from, so a person can check it, and your test set should include questions the material does not cover, to confirm the system says so rather than improvising. The same clarity that makes a document easy to retrieve internally is what makes a public page quotable by a large language model in search, and the applied version inside a business is AI chatbot work.
