RAG Explained: Giving a Model Access to Documents It Has Never Seen

The Confident Wrong Answer

Ask a model about your company's deployment policy and it will answer. Fluently, in the right tone, with the right shape — two approvals, a staging soak, a change window. It will sound exactly like a deployment policy.

It will also be invented, because the model has never read yours.

This is the failure mode that makes language models unusable for internal tools out of the box. Not refusing to answer, which you could handle, but answering plausibly about documents it has never seen. The model has no way to know the difference between recalling something and constructing something that fits the shape of the question.

Fixing that is the subject of the next four posts. This one is the map: what the options are, what RAG actually is, and what each stage does before we build any of them.

Three Ways to Give a Model Your Data

There are only three, and they are often discussed as though they were interchangeable.

Put it in the prompt. Paste the document into the context window and ask about it. No infrastructure, no pipeline, nothing to maintain.

Fine-tune the model. Train on your own examples so the behaviour is baked into the weights.

Retrieve it. Keep the documents outside the model, find the relevant pieces at question time, and include just those in the prompt. This is retrieval-augmented generation.

They are not alternatives to the same problem. They solve different problems, and choosing by habit is how projects end up fine-tuning when they needed a search index.

The row that decides most real projects is the last-but-one. Fine-tuning cannot cite. A model with your handbook in its weights cannot tell you which paragraph it is drawing on, and for an internal tool that answers policy questions, an uncitable answer is close to worthless — the first thing a user wants is the link.

The row that decides the rest is update cost. Facts change. A handbook edited weekly against a model retrained monthly is a model that is confidently out of date.

Which One Do You Need

graph TD
    A[The model answers wrongly] --> B{Wrong facts,<br/>or wrong behaviour?}
    B -->|Wrong behaviour: tone, format, task| F[Better prompting first,<br/>then fine-tuning]
    B -->|Wrong facts| C{Does the whole corpus fit<br/>in one prompt, affordably?}
    C -->|Yes, and it rarely changes| P[Put it in the prompt]
    C -->|No, or it changes often| R[Retrieval]
    R --> S[Build the two paths below]

Two notes on that first branch, because it is the one people get wrong.

If the model's problem is that it will not answer in the format you need, or sounds wrong, or keeps ignoring an instruction — that is behaviour, and retrieval will not help. Try the prompt work from posts 5 and 6 first; a schema and two examples fix a surprising amount of what looks like it needs training.

And "does it fit in one prompt" is a real question now in a way it was not two years ago. Context windows run to a few hundred thousand tokens, so a modest handbook genuinely does fit. But the input is billed on every single call, and the long-context tiers cost more per token above the standard threshold — so "it fits" and "you can afford it at ten thousand questions a day" are different claims. Post 3 has the arithmetic.

RAG Is Two Paths, Not One Pipeline

Here is the thing most introductions blur, and the single most useful idea in this post.

RAG is not one flow. It is two flows that share one component, and they run at completely different times.

graph LR
    subgraph idx["INDEXING — runs once, offline, when documents change"]
        D[Your documents] --> L[Loader]
        L --> S[Splitter]
        S --> E1[Embedding model]
        E1 --> V[("Vector store")]
    end
    subgraph qry["QUERYING — runs on every single request"]
        Q[User question] --> E2[Embedding model]
        E2 --> RT[Retrieve top-k chunks]
        RT --> ST[Stuff into prompt]
        ST --> M[Chat model]
        M --> AN[Grounded answer]
    end
    V -.->|searched by| RT

The indexing path turns documents into something searchable. You run it when you add a handbook, not when a user asks a question. If it takes four minutes, nobody notices.

The querying path turns a question into an answer. It runs while someone waits, so every stage in it is on your latency and cost budget.

The vector store is the only thing both paths touch — written by one, read by the other. That is the whole architecture.

Keeping these separate in your head answers a lot of questions before you ask them. Why is embedding in both lanes? Because the question has to be turned into a vector with the same model that produced the document vectors, or the comparison is meaningless. Why is chunking an indexing concern? Because by query time it has already happened, and a bad chunk boundary is baked in. Why does adding a document not need a redeploy? Because indexing is a separate path.

The Five Stages, Briefly

Each of these gets a proper treatment in the next three posts. Here is what each one is for, so the sequence makes sense.

Loading turns a file into text plus metadata. A PDF becomes a list of Document objects, one per page, each carrying where it came from. The metadata matters more than it first appears — it is what lets you cite a source and filter a search later.

Splitting cuts those documents into chunks. This is the stage that quietly determines whether your RAG system is good, and the one tutorials rush. A chunk is the unit that gets retrieved, so a chunk that splits an answer in half retrieves half an answer, and no amount of clever prompting downstream repairs that.

Embedding converts each chunk into a vector — a list of a few hundred to a few thousand numbers positioning that text in a space where distance means similarity of meaning. This is what makes search work on meaning rather than on keywords, so "who signs off a release" can find a passage that says "deployments require two approvals" without sharing a single word.

Storing puts those vectors somewhere you can search them quickly. That is a vector store, and if you read the Database Engineering series, this is the same machinery as the vector database post there — the index structures are the subject of that series, and the application layer is the subject of this one.

Retrieval and generation is the query path. Embed the question, find the nearest chunks, paste them into a prompt as context, and ask the model to answer from that context. The last part is a prompt instruction, and it is what converts a model that invents deployment policies into one that reports yours.

A Note on Chroma

The next two posts use Chroma as the vector store, because it runs locally with no service to provision and it is the right choice for learning. One thing to know before you install it.

pip install langchain-chroma pulled 44 packages into a clean environment here. Grouped by what they are actually for:

 2  the integration itself     langchain-chroma, chromadb
 7  observability              opentelemetry api / sdk / proto / 4 exporters
 6  cluster and RPC            kubernetes, grpcio, googleapis-common-protos,
                               websocket-client, requests-oauthlib, durationpy
 5  ML tooling                 tokenizers, huggingface-hub, hf-xet, fsspec, filelock
 7  CLI and server extras      typer, rich, click, shellingham, uvloop,
                               watchfiles, httptools
17  everything else            build tooling, hashing, pydantic, urllib3, ...

Chroma is not just a library you import; it is a database server that can also run embedded, and you get the whole server's dependency tree either way. In this environment the install also upgraded pydantic and left three unsatisfied version constraints that pip check reports — kubernetes wanting a newer certifi and urllib3 than were resolved. Nothing broke, and everything in the next posts ran correctly, but a 44-package install with transitive conflicts is worth knowing about before it lands in a service image you are trying to keep small.

Two practical consequences. Pin your versions, because a dependency graph this wide is one where an unpinned upgrade eventually surprises you. And if you are deploying rather than learning, look at what you are actually getting — a hosted vector store, or pgvector in a PostgreSQL instance you already run, may be a far smaller addition to a production image than an embedded database server with a Kubernetes client in it.

There is also a long-standing requirement on the system SQLite version, which Chroma uses for its local store. It was satisfied here and on any current system it will be, but it is the first thing to check if an install fails in an older container image.

🧭 What's Next

• Post 8b: Loading and Splitting — the indexing path, properly. Getting PDFs and Word documents into Document objects, what the langchain-community sunset means for the loaders everyone uses, and the chunking decisions that determine how good your retrieval can possibly be. Including a splitter that silently ignores the chunk size you gave it.