Retrieval-augmented generation

Retrieval-augmented generation is the practice of looking up relevant information and placing it into the model’s context window before the model answers, so that the answer rests on fetched facts rather than on whatever the model happened to memorize during training.

The problem it exists to solve

A language model is trained once, on a large body of text, and then frozen. Whatever it learned during that training is baked in, and everything that happened afterward is invisible to it. If a model finished training last year, it does not know what shipped last week. It does not know the name of your new coworker, the contents of the document you saved this morning, or the exact shape of the code in the repository you are working in right now. Its knowledge has a date on it, and that date is in the past.

There is a second, quieter problem. Even for things the model did learn, its knowledge is general. It read a great deal of public text, so it knows a lot about how software is usually written, how contracts are usually worded, how a disease usually progresses. But it does not know the specifics of your situation. It never saw your private files, your internal wiki, your company’s particular way of doing things. Ask it a general question and it does well. Ask it about your world and it either guesses or, worse, states a plausible-sounding answer with confidence.

That confident guess has a name. When a model produces text that sounds right but is not grounded in any real source, we call it a hallucination. The model is not lying in any human sense. It is doing exactly what it was built to do, which is to produce the most likely next words. If it does not have the fact, the most likely next words can still read as a fact. The output is fluent and wrong at the same time, which is the most dangerous combination.

You could try to fix both problems by training the model again on newer and more specific data. People do this, and it has its place, but it is slow, expensive, and blunt. You cannot retrain a model every time a file changes. And a model that has memorized a fact still cannot tell you where that fact came from, so you cannot check it. Retrieval-augmented generation takes a different route entirely. Instead of trying to push more knowledge into the model, it leaves the model as it is and brings the right knowledge to the model at the moment of the question.

The retrieve-then-generate loop

The whole idea fits in one sentence: before the model answers, go find the relevant material and paste it into the prompt. The model then answers with that material sitting right in front of it. Everything else is detail about how to find the right material and how to hand it over well. Here is the loop, step by step.

1. A question or task arrives. This is the thing the model will ultimately answer.
2. A retriever searches a collection of documents for the pieces most likely to help with that question. It does not read everything. It selects a handful of relevant passages.
3. Those passages are stitched into the prompt, usually with a short instruction like: answer the question using the information below, and if the information is not there, say so.
4. The model reads the question together with the fetched passages and generates its answer, leaning on the passages for the specifics.

Notice what changed. The model did not have to know the answer in advance. It had to be good at reading a few passages and using them, which is something language models are genuinely good at. The hard knowledge lives in the documents, which can be updated any time without touching the model. Change a document, and the next answer that retrieves it reflects the change immediately. There is no retraining, no waiting.

The word to hold onto here is grounding. An answer is grounded when it can be traced back to a real source that was actually placed in front of the model. Retrieval-augmented generation is, at heart, a machine for grounding. It does not make the model smarter. It makes the model’s answers rest on something you can point at and check.

Chunking and indexing: preparing the library

The retriever cannot search a pile of documents efficiently if it has to read every word of every document each time a question comes in. So the documents are prepared ahead of time, the way a library prepares its shelves and its catalog long before any reader walks in. This preparation has two parts: splitting the documents into pieces, and building an index over those pieces.

Splitting is called chunking. A long document is cut into smaller passages, each a few sentences or a few paragraphs. There are two reasons to do this. First, the context window, the amount of text a model can read at once, is limited, so you cannot hand over whole books. Second, retrieval is more precise when the pieces are small. If someone asks a narrow question, you want to return the one paragraph that answers it, not the forty-page chapter it happens to live in. Chunk too large and you drown the real answer in surrounding noise. Chunk too small and you cut a single idea in half, so no chunk contains the whole thought. Choosing the chunk size is a real tradeoff, and it matters more than it first appears.

Once you have chunks, you build an index: a structure that lets the retriever find relevant chunks quickly. Think of the index at the back of a textbook. Nobody rereads the whole book to find where inflation is discussed. They look up inflation in the index and it points them straight to the pages. A retrieval index does the same job, and the most common kind today works by meaning rather than by exact words. Each chunk is turned into a list of numbers that captures roughly what it is about, so that chunks with similar meaning end up near each other. The retriever then converts the question into the same kind of numbers and looks for the nearest chunks. That machinery, turning text into numbers that capture meaning, is a topic of its own, and it is the subject of embeddings and vector space, along with the search technique built on top of it.

The important thing at this stage is the shape of the work. The slow, careful part happens once, up front: chunk the documents, turn each chunk into its numeric form, store them in the index. The fast part happens at question time: turn the question into numbers and look up the nearest chunks. A library that took months to organize can answer a reader in seconds, and for the same reason.

A worked example

Suppose you have a company handbook, and someone asks: how many days of vacation do new employees get in their first year? A plain language model, with no access to your handbook, will produce a number. It will sound reasonable. It will very likely be wrong, because it is guessing at a fact it never saw.

Now run the same question through retrieval-augmented generation. The retriever searches the chunked handbook and finds the passage about paid time off, which happens to read: employees in their first calendar year accrue vacation at a rate of one and a quarter days per month, beginning in their second full month. That passage gets pasted into the prompt above the question. The model now reads the real policy and answers from it, and it can even show its work: roughly twelve and a half days over the first year, starting in the second month. The answer is grounded. You can check it against the very passage the model was given.

Retrieval-augmented generation does not make the model know more. It changes what sits in front of the model at the moment it answers.

The example also shows why this beats retraining for this kind of task. Next year the policy changes to a flat fifteen days. You edit one passage in the handbook. You do not touch the model at all. The very next time someone asks, the retriever fetches the updated passage and the answer updates with it. The knowledge lives in the documents, where knowledge like this belongs, and the model stays a fixed, reliable reader of whatever it is handed.

What makes retrieval good, and what makes it bad

Everything now hangs on one link in the chain: retrieval. If the retriever fetches the right passages, the model has what it needs and tends to answer well. If the retriever fetches the wrong passages, or misses the crucial one, the model answers from bad material and produces a bad answer, usually with the same fluent confidence as always. This is the single most important fact about the whole approach, and it is worth being blunt about it.

Garbage retrieval yields garbage answers. The model can only reason over what it is given, and it cannot fetch what the retriever failed to find.

So what separates good retrieval from bad? A few things. Coverage: the answer has to actually exist somewhere in the documents, and the relevant chunk has to have been indexed. If the fact is not in the library, no retriever can find it. Precision: the retriever should return the passages that bear on the question and leave out the ones that merely look similar. A passage can share a lot of words with the question and still be about something else entirely. Ranking: the most relevant passage should come back near the top, because there is only so much room in the context window and only the top few chunks make it in. Freshness: the index has to reflect the current state of the documents, or the model answers from stale material even though the documents were updated.

There is a failure mode worth naming directly. When retrieval returns passages that are related in surface but wrong in substance, the model does not push back. It trusts what it is handed. Give it a passage that superficially matches the question but describes a different product, an old version, or an unrelated case, and it will answer earnestly from that passage and sound entirely sure. The failure is invisible in the output, because the output is just as fluent as a correct one. This is why the quality of a retrieval system is judged by the quality of what it retrieves, not by how good the model sounds.

Retrieval-augmented generation is grounding in practice

Step back and the pattern is really a discipline about where facts come from. A raw model answers from memory, and memory is frozen, general, and unciteable. A grounded model answers from sources that were fetched for the occasion, and those sources are current, specific, and checkable. The same underlying model, wrapped in retrieval, goes from confidently guessing to answering with receipts.

This is why so many practical systems built on language models are, underneath, retrieval systems with a model on the end. A question-answering tool over your documents, an assistant that cites your policies, a coding helper that reads your actual files before it suggests a change: all of them are the same shape. Find the relevant material, put it in the context window, then let the model generate. The intelligence people notice is real, but a large part of the usefulness comes from the humble step of fetching the right thing first.

The honest limit

Retrieval-augmented generation is not a cure for everything, and pretending otherwise leads to disappointment. It helps precisely when the right information exists somewhere and can be found. It does nothing for a question whose answer is not in the documents at all. It does not make the model reason better; it only feeds the reasoning better inputs. And it inherits every weakness of its retriever. A brilliant model with a poor retriever is a poor system.

There is also a subtler limit that matters a great deal for some kinds of data, and it points beyond this idea. The dominant way to retrieve, matching by meaning through embeddings, works wonderfully for prose, where things that mean the same tend to use similar words. It works far less well for material where the pieces that depend on each other do not share vocabulary at all. Code is the sharpest example. A function and the code that calls it can be genuinely, tightly connected while sharing almost no words in common, so a retriever that ranks by word-level similarity quietly walks right past the real relationship.

That gap is exactly why plain similarity is not the end of the story. When the connections you care about are explicit links rather than resemblances, you want to retrieve by following those links, not by measuring nearness in meaning. Understanding where similarity search shines and where it fails is the natural next step, and it leads to vector search and its limits, and then to retrieval that walks a graph instead, which is the subject of GraphRAG for code.