Why RAG — parametric vs. non-parametric memory

A large language model (LLM) is a function from a prompt to text, and everything it "knows" was baked into its billions of weights during training, then frozen. That single fact — the weights are fixed at inference time — is the root of almost every problem this course exists to solve: the model cannot have read your internal wiki, it cannot know what happened after its training cutoff, and when it lacks a fact it tends to invent a fluent, confident, wrong one rather than abstain. Retrieval-Augmented Generation (RAG) is the dominant engineering answer: instead of trying to cram more knowledge into the weights, you leave the weights alone and, at request time, retrieve relevant text from an external store and read it into the prompt as context. This lesson frames why that split works, using the parametric-vs-non-parametric distinction from the paper that named the pattern — Lewis et al.'s Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020). We stay conceptual here; the next lesson builds the full pipeline in code.

The frozen-knowledge problem

When you call an LLM API, you are calling a function whose parameters were estimated once, during a training run that ended on some cutoff date, and have not changed since. The model's factual knowledge lives inside those parameters — Lewis et al. open by noting that large pre-trained language models "store factual knowledge in their parameters" and act as a kind of implicit knowledge base (Lewis et al. 2020). This is remarkable, but it has three structural consequences that no amount of clever prompting removes.

First, staleness: anything that happened after the cutoff, or any document that simply was not in the training corpus, is invisible to the model. Your company's Q3 incident report, written last week, is not in there.

Second, no provenance: because knowledge is smeared across billions of weights rather than stored as discrete retrievable records, the model cannot point at where a claim came from. Lewis et al. name this directly — with purely parametric models, "providing provenance for their decisions and updating their world knowledge remain open research problems" (Lewis et al. 2020).

Third, imprecise recall: even for facts that are in the training data, the same authors observe that these models' "ability to access and precisely manipulate knowledge is still limited," so they "lag behind task-specific architectures" on knowledge-intensive tasks (Lewis et al. 2020). Knowing a fact diffusely is not the same as being able to recall it exactly on demand.

Hallucination — the failure mode that looks like success

The most dangerous symptom of the frozen-knowledge problem is hallucination: generated text that is fluent, grammatical, and confident, yet unfaithful to any real source. Ji et al., in their survey for ACM Computing Surveys, define hallucinated content as text that is "nonsensical or unfaithful to the provided source content" and document that this tendency is general across natural-language-generation tasks built on Transformer language models (Ji et al. 2023).

The reason hallucination is treacherous is that the model has no built-in signal distinguishing "I am recalling a fact" from "I am generating a plausible continuation." Both produce equally confident prose. An LLM is trained to predict likely next tokens, so when it lacks the real answer it will still emit the most probable-looking answer — a fabricated citation, an invented API method, a wrong date — because abstaining is rarely the highest-probability continuation. Ji et al. taxonomize this into intrinsic hallucination (output contradicts the source) and extrinsic hallucination (output cannot be verified against the source at all) (Ji et al. 2023).

For an application engineer this reframes the goal. You are not trying to make the model "smarter"; you are trying to make its outputs grounded — traceable to a source you control. That is precisely what RAG attacks: by putting the relevant source text in front of the model, you change the most-probable continuation from "a plausible-sounding guess" to "a faithful summary of what is right here in the context."

Why you can't just retrain or edit the weights

The naive fix — "if the knowledge is in the weights, update the weights" — fails on cost, latency, and controllability, which is exactly why an external memory is attractive.

Retraining or fine-tuning moves the cutoff forward but is the wrong tool for keeping facts fresh. Pre-training a frontier model is expensive and slow — measured in GPU-hours to weeks — and even fine-tuning is a batch process measured in hours and GPU budgets, not the milliseconds you have between a user's question and your response. You cannot fine-tune per request. Worse, fine-tuning teaches behavior and style well but is an unreliable way to inject specific facts — and it provides no provenance, so a fine-tuned model still cannot cite where an answer came from, leaving the open problems Lewis et al. flagged unsolved (Lewis et al. 2020).

Model editing (surgically rewriting individual weights to change one fact) is an active research area, but it does not scale to a corpus that changes daily, and edits can interact unpredictably with neighboring knowledge. There is no request-time knob that says "and also know this new document."

The structural insight is that mixing knowledge into the weights is the source of the trouble, not just its location. Lewis et al. motivate RAG precisely by combining a parametric model with a non-parametric memory whose contents can be "directly revised and expanded, and accessed knowledge can be inspected and interpreted" (Lewis et al. 2020). Updating that memory is a database write, not a training run.

The retrieve-then-read pattern

RAG decomposes a knowledge-intensive query into two stages: a retriever that finds the few most relevant pieces of text from an external store, and a reader (the LLM, also called the generator) that conditions its answer on both the query and those retrieved passages. Lewis et al. describe exactly this composition — a parametric seq2seq generator combined with a non-parametric memory that is "a dense vector index of Wikipedia, accessed with a pre-trained neural retriever" (Lewis et al. 2020).

The key shift is where the relevant knowledge enters the model. It is not pulled from the weights; it arrives as ordinary text in the prompt, the same channel the user's question uses. The LLM is extremely good at reading and synthesizing text it is given in-context — that capability is fully baked into the frozen weights and needs no retraining. So RAG borrows the model's strong reasoning over provided text while sidestepping its weak recall from parameters.

Two payoffs follow immediately. Freshness becomes a write to the store: index today's incident report and the system can answer about it within seconds, no training involved. Provenance becomes free: because the answer was conditioned on specific retrieved passages, you can show the user which passages, satisfying the attribution gap parametric models leave open (Lewis et al. 2020). The two later modules on retrieval quality and grounding exist because each of these stages — retrieve, then read — has its own failure modes.

Parametric vs. non-parametric memory

The cleanest mental model from the RAG paper is a deliberate split of memory into two kinds, and knowing which kind owns which job is the organizing idea for this whole course.

Lewis et al. frame their contribution as exactly this hybrid: models that "combine pre-trained parametric and non-parametric memory for language generation" (Lewis et al. 2020). "Parametric" means the knowledge is encoded in the parameters $\theta$ of the network — to use it, you run a forward pass; to change it, you change $\theta$. "Non-parametric" means the knowledge is not compressed into a fixed-size parameter set; it lives as a growable collection of records you look up. Adding the millionth document does not enlarge the model — it enlarges the index.

A useful intuition: parametric memory is what a well-read person carries in their head — fluent, general, fuzzy on specifics. Non-parametric memory is the open reference book on the desk — exact, current, and you can point at the page. RAG is the person reading from the book. The whole engineering discipline that follows is about making the right page easy to find (retrieval) and making the person quote it faithfully (grounding).

What RAG does and doesn't solve

RAG is powerful but not magic, and being precise about its boundaries prevents the most common disappointments.

What it solves well. It eliminates staleness for anything you index, supplies provenance because answers are tied to retrieved passages, and measurably improves factuality — Lewis et al. report that on open-domain question answering their RAG models set a new state of the art, and that for language generation RAG produces output that is "more specific, diverse and factual" than a comparable parametric-only baseline (Lewis et al. 2020). It does all of this without touching the LLM's weights.

What it does not solve. RAG cannot answer from a document it failed to retrieve — a miss at the retrieval stage is invisible to the reader, which is why retrieval quality dominates real systems and gets its own modules later. It does not guarantee faithfulness either: an LLM handed correct passages can still ignore or misread them, so grounding is a request you make of the model, not a property you get for free — measuring and enforcing that is the job of the evaluation module. And RAG does not improve reasoning that does not depend on external facts; for purely deductive or stylistic tasks, retrieval adds noise, not value.

The honest summary is that RAG converts a hard, unsolved problem (precise, current, attributable recall from frozen weights) into two more tractable engineering problems (find the right text; faithfully read it). The rest of this course is the disciplined pursuit of those two problems.

Try it: the same question, with and without context

The snippet below uses a tiny stub "LLM" to make the mechanism visible — no model, no network. Without retrieved context the stub has nothing grounded to say; with one retrieved passage in the prompt, the answer becomes specific and traceable. This is the entire idea of RAG in miniature.

Sources

• Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) — https://arxiv.org/abs/2005.11401
• Ji et al. — Survey of Hallucination in Natural Language Generation, ACM Computing Surveys (2023) — https://arxiv.org/abs/2202.03629

Practice

1. Why does fine-tuning the model on last week's documents not give you the freshness and citability that RAG does? Fine-tuning is a slow, costly batch process — you cannot run it per request, so the index of "what the model knows" still lags reality. More fundamentally, it folds facts back into the diffuse weights, so it restores neither provenance nor precise recall; Lewis et al. list updating world knowledge and providing provenance as open problems for purely parametric models (Lewis et al. 2020).

2. Hallucination is often described as the model "lying." Why is that framing misleading, and what is actually happening? The model is not modeling truth at all — it predicts the most probable next tokens. When it lacks the real fact, a confident fabrication is often a more probable continuation than abstaining, so it produces fluent, unfaithful text (Ji et al.'s definition of hallucination) (Ji et al. 2023). It has no internal signal separating recall from invention.

3. Classify each as parametric or non-parametric memory, and say why it matters: (a) the grammar the model uses to form sentences, (b) your company's incident database. (a) Parametric — encoded in the frozen weights, used via a forward pass, changed only by retraining. (b) Non-parametric — an external store you can update with a write and cite by record. The split matters because it tells you to keep general language ability in the weights and route specific, changing, citable facts to the index (Lewis et al. 2020).

4. A teammate says "we added RAG, so the system can no longer hallucinate." Where is this wrong? RAG only helps for facts that are successfully retrieved; a retrieval miss leaves the reader to guess, and even with correct passages the LLM can ignore or misread them. RAG makes grounding possible, not guaranteed — which is why retrieval quality and grounding evaluation are separate engineering problems addressed later in the course.

5. Explain the retrieve-then-read pattern in one sentence, and identify which LLM capability it leans on and which it sidesteps. Retrieve relevant text from an external store, then have the LLM answer conditioned on that text — leaning on the model's strong ability to read and synthesize provided context while sidestepping its weak, un-citable recall from parameters (Lewis et al. 2020).