Retrieval-Augmented Generation

In 2020, Patrick Lewis and Ethan Perez from Facebook AI Research published a paper titled "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" at NeurIPS, proposing Retrieval-Augmented Generation (RAG) as a complete end-to-end framework. This framework allows language models to first look up relevant information from an external knowledge base before answering a question, then generate answers based on the retrieved material, much like a human writing a survey after consulting the literature.

The previous two chapters covered the evolution of embedding models from Word2Vec to BGE, the working mechanisms of IVF, HNSW, and PQ vector indexes, the design principles of BM25 and hybrid retrieval, evaluation metrics such as recall, precision, and NDCG, along with the two-stage architecture of initial retrieval followed by re-ranking. All of these are components of a RAG system. This chapter's task is to connect these components into a complete workflow from query entry to final answer generation, and to introduce advanced patterns such as query rewriting, iterative retrieval, and Self-RAG that take RAG systems from merely functional to truly effective.

Basic RAG Workflow

A complete RAG system consists of three basic stages: retrieval, injection, and generation. When a user query arrives, the system first encodes the query text into a vector using an embedding model, then retrieves the Top-K most relevant document fragments from a pre-built vector index. This step is detailed in Embedding and Vector Retrieval, covering the full pipeline from text embedding generation to approximate index search. After retrieval, the system formats these document fragments into the prompt to form an augmented context. Finally, the language model generates the final answer based on this augmented prompt. Under this basic RAG workflow, a typical prompt template looks like this:

Answer the question based on the following reference materials. If the answer cannot be found in the materials, state "Cannot answer based on the available materials."
Reference Materials:
{context}
Question: {query}
Answer:

This template explicitly requires the model to prioritize using retrieved content. Relying on retrieved results for generation is the fundamental difference between RAG and pure model generation. The template also includes a fallback strategy for when retrieval fails -- when the retrieved results truly lack relevant information, the model is required to honestly report this rather than infer on its own or, worse, fabricate. Placing the materials before the question leverages the LLM's heightened sensitivity to the beginning of the context to improve generation quality.

The three-stage basic RAG workflow performs well in simple Q&A scenarios but often struggles with complex questions. The bottleneck may lie at the query end. Users' original queries are often too brief or imprecisely worded -- for instance, entering "how to do RAG" when they actually mean "how to integrate RAG functionality into an existing Python web application." When the query is poorly expressed, a single retrieval can easily miss key information. Even when the query itself is clear enough, choosing the right Top-K value at the retrieval end presents a dilemma. A K that is too small risks missing critical documents (under-retrieval), while a K that is too large introduces noise that interferes with the model's judgment (retrieval noise). At the generation stage, when multiple retrieved documents contain contradictory information, the model faces the challenge of deciding which source to trust. From query expression and retrieval granularity to information source conflicts -- these practical difficulties have given rise to a series of improvements to RAG systems, which we will introduce one by one in this chapter.

Query Rewriting and Expansion

The basic RAG workflow assumes that the user's query itself sufficiently expresses the retrieval intent, but in practice this assumption often fails. A user might colloquially describe a professional question (e.g., "What's that AI that's really good at making images?"), while the knowledge base uses formal terminology (e.g., "diffusion models," "Stable Diffusion"). When the vocabulary in the query does not match the vocabulary in the documents, the retrieval system struggles to find the right content. The purpose of query rewriting is to preprocess the user's query before retrieval, bridging this semantic gap.

Multi-Query is the earliest query rewriting strategy to emerge. It first prompts the LLM to generate multiple sub-queries from different angles based on a single original query, then performs independent retrieval for each sub-query, and finally merges and deduplicates the results to produce a comprehensive set. Suppose a user asks "Transformer position encoding," the LLM might generate variants such as "the principle of sinusoidal position encoding in Transformers," "the difference between learnable position encoding and sinusoidal encoding," and "the application of relative position encoding in Transformers." Documents retrieved from different angles complement each other, and recall is typically significantly higher than with single-query retrieval.

Hypothetical Document Embedding (HyDE) takes a more creative approach to query rewriting. In 2022, a paper from the University of Waterloo titled "Precise Zero-Shot Dense Retrieval without Relevance Labels" found that having the LLM first fabricate a hypothetical answer document based on the query, then using the vector of this fabricated document to retrieve real documents, actually yielded better results than directly using the query vector for retrieval. The intuition behind HyDE is that the distance between an answer document and a truly relevant document in vector space is generally closer than the distance between a brief query and a document. Queries are often just a few keywords, whereas documents are complete paragraphs -- their semantic densities are mismatched. The LLM-generated hypothetical answer fills in the context implicit in the query, bringing it closer to the target document in semantic space. The risk of this strategy is that if the LLM's hypothetical answer deviates from the correct direction -- for instance, giving an inaccurate explanation of a concept in a specialized domain -- the generated vector will steer retrieval toward the wrong area, producing worse results than using the original brief query directly.

Conversely, when the query is not too simple but rather too complex -- for example, containing multiple mixed types of conditions -- it can also significantly affect result accuracy. Self-Query Retrieval handles hybrid queries that contain both semantic and structured conditions. Take "papers published in 2026 about improvements to Transformer attention mechanisms" as an example. In this query, "improvements to Transformer attention mechanisms" requires semantic matching, while "2026" requires precise year filtering. Pure vector retrieval struggles to handle the year condition, while pure scalar filtering struggles with semantic matching. Self-Query Retrieval uses the LLM to decompose the query into two parts: the semantic part ("improvements to Transformer attention mechanisms") is handed to the vector index, while the structured condition ("year = 2026") is handed to the metadata filter. The results from both paths are then intersected. Frameworks such as LangChain have built-in parsing and execution logic for this type of query.

Context Injection Strategies

After retrieval, injecting documents into the prompt is not simply a matter of string concatenation. The model's context window is a precious resource with limited length (although the latest models already support 1M tokens or more, longer contexts mean higher latency and cost), so the total amount of retrieved documents must be kept within the model's processing capacity. More subtly, LLMs do not attend uniformly to all positions in the context. The "Lost in the Middle" phenomenon reveals that models tend to focus on information at the beginning and end of the prompt, while content in the middle is easily overlooked. This means that relevance-based sorting alone is not enough -- we also need to strategically arrange the order during concatenation, placing the most relevant documents at the beginning and end of the context and less relevant ones in the middle, so that the model's attention focuses on the most important information.

Simply concatenating documents in descending order of retrieval score is the easiest approach, but there may be redundancy or contradictions between documents. A more refined approach is to first group documents structurally before concatenation. Documents are categorized by topic, and each group is given a summarizing title, allowing the model to get a global view before diving into details. If retrieved documents are too long, each document can be compressed into a summary before injection, trading information density for effective context window utilization. When multiple documents contain contradictory information (e.g., two documents claiming the same event occurred in 2023 and 2024 respectively), annotating the information source during injection and letting the model judge for itself is usually safer than disambiguating in advance -- models often outperform rule-based systems at understanding subtle informational differences in context.

Context Compression is another practical optimization direction that every developer who has used Vibe Coding tools will find familiar. In a RAG scenario, if 20 retrieved documents are stuffed into the prompt verbatim, the total length could reach tens of thousands of tokens, much of which consists of transition sentences, repeated information, or query-irrelevant details. The goal of compression is to retain only the parts of each document most relevant to the current query and remove irrelevant content. Implementation approaches range from simple key sentence extraction (sorting and truncating based on the similarity between query terms and document sentences) to having the LLM summarize each document paragraph by paragraph, balancing precision and latency. The difficulty of context compression lies in how to judge relevance. The compressor only sees a single document and does not know whether other documents already contain the same information, so cross-document deduplication typically needs to be handled separately during the concatenation phase after compression.

Iterative Retrieval

If a single retrieval can cover all the information needed to answer a question, that would be simplest for both the user and the RAG system designer. For simple factoid queries like "What is the capital of France," this assumption holds. But when a user asks, "Compare the design philosophies and performance differences of Transformer and Mamba for long-sequence modeling," a single retrieval can hardly cover the Transformer's position encoding mechanism, Mamba's state space model principles, and the long-sequence modeling evaluation data for both. This scenario calls for Iterative Retrieval. The idea behind iterative retrieval is to progressively gather the information needed to answer a question through multiple rounds of retrieval, where each round can be adjusted and focused based on the results of the previous round. Depending on how the iteration path is organized, iterative retrieval can be further divided into three modes: chain retrieval, sub-question decomposition, and iterative refinement.

Chain Retrieval proceeds step by step. The first round retrieves "Transformer long-sequence modeling," generating a preliminary answer based on the retrieved documents. From this answer, we discover that the bottleneck of Transformers on long sequences mainly comes from the $O(n^2)$ complexity of the self-attention mechanism, so the second round retrieves "methods to reduce self-attention complexity," introducing variants such as sparse attention and linear attention. The third round then targets the comparison object, retrieving "Mamba state space model long-sequence performance." Each round in chain retrieval is anchored to the output of the previous round, and the retrieval direction gradually focuses, making it suitable for problems with clear steps and well-defined dependencies.
Sub-Question Decomposition first breaks the complex question into multiple independent sub-questions, then retrieves for each sub-question in parallel, and finally merges all retrieval results for unified generation. Using the "compare Transformer and Mamba" example again, the decomposition yields three sub-questions: "Transformer long-sequence modeling mechanism," "Mamba state space model mechanism," and "long-sequence performance benchmark comparison of both." Each sub-question is retrieved independently without interference from the results of other sub-questions. After merging, the model obtains a complete information panorama in one go, avoiding the risk of errors from a previous round propagating to the next round, which can occur in chain retrieval.
Iterative Refinement combines the advantages of the previous two modes. Like chain retrieval, it directs complementary information gathering based on existing answers, while like sub-question decomposition, it uses a self-check mechanism to ensure the answer covers all aspects of the question. It begins by generating a preliminary answer, then has the LLM self-check whether this answer is complete. If some aspect is found to have insufficient information (e.g., the answer only discusses Transformers but lacks specific data on Mamba), the LLM generates a supplementary query to retrieve the missing information, updates the answer with the new results, and repeats this process until the answer covers all aspects of the question or reaches the iteration limit.

When to terminate iteration is an important design decision. The simplest approach is to set a fixed maximum number of iterations (typically 3 to 5 rounds) -- reliable but inflexible. A more refined approach is to have the LLM judge at each round whether the new retrieval results provide incremental information. If the documents returned by a new round largely overlap with those from previous rounds, it indicates convergence and no further rounds are needed. In latency-sensitive scenarios, a total time budget can also be set -- once the budget is exceeded, the final answer is generated using whatever information is currently available, ensuring the user experience is not compromised.

Self-RAG

All the RAG variants discussed so far share a common implicit premise: since it is retrieval-augmented generation, performing retrieval should be a given for a RAG system. But not every query needs retrieval. For questions like "What is 1 plus 1," the model can answer accurately on its own, and additional retrieval not only wastes computational resources but may also introduce irrelevant documents that interfere with judgment. In 2023, a paper from the University of Washington titled "Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection" proposed Self-RAG, which allows the model to autonomously decide during generation whether retrieval is needed, whether the retrieved results are relevant, and whether the generated content is faithful to the retrieved documents.

The mechanism by which Self-RAG achieves this autonomous decision-making is the introduction of four special Reflection Tokens into the model's vocabulary. The [Retrieve] token lets the model judge whether the current query needs retrieval, with possible values of Yes, No, or Continue -- equivalent to the model first asking itself, "Can I answer this question using my own knowledge?" After retrieval is performed, the [ISREL] token judges whether each retrieved document is truly relevant to the query, with possible values of Relevant or Irrelevant; irrelevant documents are ignored during subsequent generation. During the generation phase, the [ISSUP] token checks whether each sentence generated can be supported by evidence in the retrieved documents, with possible values of Fully Supported, Partially Supported, or No Support -- equivalent to real-time monitoring of hallucination occurrence. The [ISUSE] token evaluates whether the generated answer is actually helpful to the user, using a five-level rating from 1 to 5, where 5 is most useful, covering cases where the answer addresses the question but misses the mark.

These four tokens form a complete self-reflection loop. When a query arrives, the model first outputs a [Retrieve] decision. If the answer is No, it directly generates an answer and outputs it. If the answer is Yes, after retrieval, the model outputs [ISREL] judgments for each document, filtering out irrelevant content. After generating an answer based on the relevant documents, the model outputs [ISSUP] to verify faithfulness sentence by sentence, and selects the answer with the strongest factual support from multiple candidates. Finally, it outputs [ISUSE] to evaluate the overall quality of the answer, regenerating if not satisfied. The entire process is driven by the model itself, without the need for external rule intervention.

The training approach for Self-RAG treats these reflection tokens as regular tokens added to the vocabulary, inserting them at the correct positions in the training data to teach the model when to output them. Training data is constructed by having GPT-4 automatically generate reflection tokens, then using this annotated data to train a Critic model, which in turn batch-annotates the entire training dataset, avoiding the high cost of manual annotation. After training, the model acquires the metacognitive ability for reflection -- it not only answers questions but also knows what it knows, what it does not know, and when it should look up information.

Multi-Channel Retrieval

Multi-Channel Retrieval is an important means of combining multiple retrieval strategies to extend the query capability of a RAG system. Vector retrieval excels at semantic generalization, but when faced with exact match requirements like "technical specifications of product number AB-12345," relying solely on the cosine similarity of embedding vectors may return documents for similar but entirely different products. Sparse retrieval methods such as BM25 perform admirably in exact matching, yet they cannot understand the semantic relationship between "car" and "automobile." Different types of information naturally reside on different retrieval paths. Multi-channel retrieval simultaneously searches from multiple paths and then integrates the results. The hybrid retrieval (dense vectors + BM25) discussed in Embedding and Vector Retrieval is the most basic and widely used form of multi-channel retrieval. Beyond this, strategies closely related to multi-channel retrieval in RAG systems include query routing and knowledge graph-enhanced retrieval.

Query Routing often serves as a front-end decision-making component for multi-channel retrieval. Not every query needs to traverse all retrieval paths. The role of routing is to direct a query to the most appropriate path based on its characteristics. Factual queries (e.g., "Who won the 2024 Nobel Prize in Physics?") are routed to knowledge graphs or exact-match indexes, semantic queries (e.g., "How to handle vanishing gradients in deep learning?") are routed to vector indexes, and statistical queries (e.g., "Revenue comparison of each product line in Q3") are routed to SQL structured queries. The routing decision itself can be made by an LLM -- the query is presented to the LLM along with descriptions of each retrieval path, and the LLM determines which path to take, or whether to take multiple paths simultaneously with different weight assignments.

Graph RAG is another important retrieval architecture that complements multi-channel retrieval by combining the structured relationship reasoning capability of knowledge graphs with the semantic generalization capability of vector retrieval. Relationships between entities in a knowledge graph (e.g., "Drug A treats Disease B," "Paper C cites Paper D") cannot be directly obtained through vector similarity, yet these relationships are crucial in scenarios such as medical diagnosis and legal case analysis. The Graph RAG workflow involves extracting entities from the query, retrieving neighbor nodes and multi-hop relationship paths of these entities in the knowledge graph, simultaneously performing vector retrieval to obtain relevant unstructured text, and finally fusing the structured relationships with the unstructured text before presenting it to the LLM for answer generation. Microsoft's GraphRAG project has done systematic exploration in this direction, enhancing RAG answer quality by automatically building entity relationship graphs from source documents.

Citation Tracing and Trustworthiness

An important advantage of RAG over pure model generation is that answers can be verified. When a model answers, "The Transformer architecture was published in 2017," if it includes citation annotations pointing to the source document, users can click the link to confirm whether the information is accurate and whether there is any quote mining. In scenarios such as legal document assistance and medical literature review, citations are not just a nice-to-have feature but a basic compliance requirement.

The granularity of citation implementation has three common choices. Document-level citation inserts numeric markers like [1], [2] in the answer paragraphs and appends a numbered list of sources at the end -- simple to implement and suitable for quick verification. Paragraph-level citation annotates the source for each natural paragraph, where all statements within a paragraph point to the same document. Sentence-level citation offers the finest granularity, with each sentence having an independent source annotation. Sentence-level citation is the most challenging to implement because LLMs easily lose track of which sentence corresponds to which source when generating long paragraphs, requiring an additional attribution mechanism to trace.

Trustworthiness evaluation of RAG-generated results is generally carried out along two dimensions: the authority of the sources and the consistency of the information. Source authority depends on the reliability of the document origin. Peer-reviewed academic papers obviously have higher authority than personal blogs, and official documentation obviously has higher authority than community wikis. Production systems typically maintain a source trust level list, prioritizing retrieval of documents from high-authority sources during retrieval. Consistency checking is the realization of multi-source verification -- if three independent sources all confirm the same fact, the credibility of that fact is far higher than a statement supported by only a single source. When retrieval results are insufficient to support a definitive answer, the model should proactively express the boundary of uncertainty, such as "Based on available materials, there is currently no definitive answer to this question, but the following are relevant discussions." This is far more valuable than the model giving a seemingly definitive but potentially incorrect answer to an uncertain question.

Summary

RAG enhances the generative capabilities of language models by retrieving external knowledge, freeing the model's knowledge from the constraints of training data cutoff dates while providing verifiable sources for answers. Starting from the basic retrieve-inject-generate three-stage workflow, query rewriting bridges the semantic gap between user input and document expression, context injection strategies manage information layout within the limited window, iterative retrieval allows complex questions to progressively gather complete information through multiple rounds of retrieval, and Self-RAG endows the model with the self-reflective ability to decide whether to retrieve and whether to trust. Multi-channel retrieval and fusion combine the strengths of different retrieval paths, and citation tracing transforms answers from black-box outputs into verifiable conclusions. Combined with the retrieval evaluation content covered in the previous chapter, iterating on the retrieval engine, generation model, and evaluation pipeline as a whole is how RAG systems can continuously improve in practical applications.

Exercises

Compare the performance of single-query retrieval and Multi-Query retrieval on complex questions: select 5 queries that require multi-faceted information, compare the recall of the two strategies on Top-5 retrieval results, and analyze which types of queries benefit most from Multi-Query.
Reference Answer
For queries that require gathering information from multiple angles (e.g., "compare the pros and cons of A and B"), Multi-Query can cover more relevant documents through sub-queries from different perspectives, and recall is typically significantly improved. However, for simple factoid queries (e.g., "the date of a certain event in a certain year"), the original query is already precise enough, and Multi-Query may introduce noise, actually reducing precision.
Implement an iterative retrieval pattern: for a complex question like "Compare the design philosophies and performance of CNN and Vision Transformer on image classification tasks," first perform sub-question decomposition, retrieve independently for each sub-question, then merge the results to generate an answer. Compare the differences in answer completeness and faithfulness between single-pass retrieval and iterative retrieval.
Reference Answer
Sub-question decomposition is recommended to split into "CNN architecture design philosophy," "Vision Transformer architecture design philosophy," and "performance comparison of both on ImageNet." The answer completeness of iterative retrieval is typically significantly better than single-pass retrieval because each sub-question is supported by targeted retrieval results. However, the quality of sub-question decomposition directly affects the final outcome -- if the decomposition is too fine-grained, the retrieval results for each sub-question may lack correlation, making it difficult to form a coherent narrative in the answer.