What is Retrieval-Augmented Generation?

Retrieval-augmented generation (RAG) is a technique in which a language model retrieves relevant passages from a defined body of content and uses them as the basis for its answer, rather than relying on what it learned during training.

The purpose is to make a general model answer questions about specific content it was never trained on, and to make the answer checkable.

How it works

A question is converted into a search over a content collection, usually combining semantic similarity with conventional keyword matching. The passages that come back are supplied to the model together with the question, and the model is instructed to answer from those passages.

Content is prepared in advance by splitting it into chunks and indexing them. Chunking is more consequential than it appears. Chunks that are too small lose the context needed to interpret them; too large and the relevant sentence is diluted by surrounding text. Splitting mid-table or mid-clause produces fragments that retrieve well and mislead when read.

For scanned material, everything depends on the content being machine-readable first, which means optical character recognition and, for structured documents, data extraction. A retrieval system over unrecognized images retrieves nothing.

Why citations matter

A RAG answer should state which passages it came from, and the citations should be inspectable rather than decorative.

They serve three purposes. They let a reader verify the claim, which is the only reliable defense against a confident and wrong answer. They make the system auditable, which regulated processes require. And they surface retrieval failure: an answer citing an irrelevant passage reveals that retrieval, not generation, is the problem.

Citations are not a guarantee of correctness. A model can cite a real passage and still summarize it inaccurately, so citation supports verification rather than replacing it.

Grounding reduces hallucination without eliminating it

Grounding narrows the model’s scope to supplied text, which substantially reduces invention. It does not remove it.

Residual failure modes are worth naming, because vendor material tends to claim the problem is solved. A model asked something the retrieved passages do not answer will often produce a plausible answer anyway rather than declining. It may merge two passages into a claim neither makes. And it can reproduce content that is retrieved correctly but wrong at source, since grounding inherits the accuracy of the underlying content.

Mitigations are procedural rather than technical: instruct the model to decline when the passages are insufficient, show the retrieved passages next to the answer, and treat the retrieval step as the primary quality lever, because a generation failure downstream of bad retrieval cannot be fixed by a better model.

Access control must be enforced at retrieval time

This is the requirement that distinguishes enterprise RAG from a demonstration, and the one most often got wrong.

If the index contains everything and permissions are applied to the generated answer, the system has already failed. The model has seen content the user may not, and the answer is derived from it. Filtering the output does not undo that, because a summary of a restricted document is a disclosure of it. Even a refusal can leak, by confirming a document exists.

Correct behavior is to resolve the user’s permissions before retrieval and search only what they may see, so two users asking the same question receive different answers according to their access. That requires the retrieval layer to understand the permission model of the content services platform it searches, including inherited and group-derived rights, and to re-resolve them per query rather than caching them.

The failure is easy to miss in testing, because it only appears when a test is run as a low-privilege user. A system tested only by administrators looks correct.

What commonly goes wrong

Retrieval quality is not measured. Teams evaluate answers and never inspect whether the right passages were found, so generation gets blamed for retrieval failures.

Stale content is indexed. An index rebuilt on a schedule serves superseded documents with full confidence. Deleted or reclassified content must leave the index promptly, which is also a records management obligation.

Everything is indexed because it can be. Drafts, duplicates and superseded versions compete with the authoritative document and degrade every answer.

There is no path to the source. An answer a reader cannot trace to a document in its original form cannot be relied on for a decision that matters.

Contellect One applies permission-aware retrieval in content intelligence and AI-driven insights.