RAG Retrieval and Generation: Similarity Search, MMR and Stuffing

The Stage Where the Answer Is Decided

The model in a RAG system is doing less work than it appears to. It reads a few passages and writes a summary of them. It is good at that, and it will do it just as fluently whether the passages answer the question or not.

Which means the quality of your system is almost entirely the quality of your retrieval. Hand it the right three chunks and a small cheap model gives you a correct, citable answer. Hand it the wrong three and the best model available will give you a fluent, well-sourced, wrong one.

This post covers how those chunks get chosen, and then assembles the whole pipeline from posts 8b, 8c, 7a and 7b into something that works.

Similarity Search

results = store.similarity_search("who approves a deployment?", k=3)

k is how many chunks come back. The store embeds the query with the same model used for the documents, measures distance to every vector, and returns the nearest k.

With scores:

for doc, score in store.similarity_search_with_score("who approves a deployment?", k=3):
    print(round(score, 4), doc.page_content)

0.3670  Two approvals are required to approve any deployment.
1.1835  A deployment needs two approvals before release.
1.2929  Deployments require two approvals and a passing CI run.

The score is a distance, so lower is closer. This catches people out constantly — it reads like a relevance score, where bigger would be better, and it is the opposite. The best match above scored 0.3670 and the worst of the three scored 1.2929.

It is also not a probability and not comparable across stores or embedding models. Use it to rank within one result set, and to set a cut-off you tuned by looking at your own data. A threshold copied from a blog post is meaningless against your corpus.

k is a cost decision as much as a quality one. Every retrieved chunk is input tokens on every query, so k=10 with 400-token chunks adds 4,000 tokens to each request. Post 3's arithmetic applies directly: at 60,000 queries a month that is 240 million input tokens of context, and whether that is $4.80 or $1,200 depends on which model you picked.

The Redundancy Problem

Look at those three results again. They all say the same thing.

Three chunks, three paraphrases of "two approvals are required", and the on-call rule and the audit-log detail — both genuinely relevant to how approvals work — did not make the cut. The prompt is carrying one fact three times and paying for the privilege.

This is not a flaw in the embedding model. It is doing exactly what it was asked: find the nearest neighbours. Near-duplicates are, necessarily, near.

query: "who approves a deployment?"

similarity_search(k=3)                           MMR(k=3, fetch_k=6, lambda_mult=0.5)
──────────────────────────────────────────       ──────────────────────────────────────────
Two approvals are required to approve any        Two approvals are required to approve any
  deployment.                                      deployment.
A deployment needs two approvals before          During an incident the on-call engineer
  release.                               ┐         may waive the soak period.
Deployments require two approvals and a   ├ same  Expenses over 500 USD need manager
  passing CI run.                        ┘  fact    sign-off.

one fact, three times                            three different facts

Maximal Marginal Relevance

MMR fixes this by selecting results that are relevant to the query and different from what it has already selected.

results = store.max_marginal_relevance_search(
    "who approves a deployment?",
    k=3,            # how many to return
    fetch_k=6,      # how many candidates to consider
    lambda_mult=0.5 # relevance vs diversity
)

It works in two passes. First it fetches fetch_k candidates by plain similarity. Then it builds the result set one at a time, each time picking the candidate with the best combined score of close to the query and far from everything chosen so far.

lambda_mult is the dial between those two goals, and the measured behaviour tells you exactly what it does:

lambda_mult = 1.0   pure relevance
                    → Two approvals are required to approve any deployment.
                      A deployment needs two approvals before release.
                      Deployments require two approvals and a passing CI run.
                      identical to similarity_search

lambda_mult = 0.5   balanced
                    → Two approvals are required to approve any deployment.
                      During an incident the on-call engineer may waive the soak period.
                      Expenses over 500 USD need manager sign-off.

lambda_mult = 0.0   pure diversity
                    → best match, then whatever is furthest from it,
                      relevance to the query barely considered

At 1.0 MMR is similarity search — which is a useful thing to know, because it means lambda_mult is the only parameter that matters for deciding between them. The default in LangChain is 0.5.

fetch_k also matters and is easy to set wrong. MMR can only diversify among candidates it fetched, so fetch_k=k leaves it nothing to choose from and quietly degrades to similarity search. A reasonable starting point is two to four times k.

When to use which: similarity search for narrow factual questions where the best match probably is the answer. MMR for broad questions where several different pieces of context are needed, and for corpora with duplication — which, if you have ever indexed documentation alongside its own changelog, you have.

A Retriever Is a Runnable

Searching a store directly is fine in a notebook. For a pipeline you want the store wrapped as a retriever:

retriever = store.as_retriever(
    search_type="mmr",
    search_kwargs={"k": 3, "fetch_k": 6, "lambda_mult": 0.5},
)

retriever.invoke("who approves a deployment?")   # -> 3 Documents

That returns a VectorStoreRetriever, and the important thing about it is what it has: invoke, batch, stream and the async forms. It is a Runnable, exactly as defined in post 7a.

Which means retrieval composes with the pipe operator like everything else, and the search strategy becomes configuration held inside the object rather than arguments scattered across your call sites. Swapping similarity for MMR is one dict.

Stuffing Documents

"Stuffing" is the unglamorous name for the obvious approach: take the retrieved documents, turn them into one block of text, and put it in the prompt.

def format_docs(docs):
    return "\n\n".join(
        f"[{d.metadata['source']} p.{d.metadata['page'] + 1}] {d.page_content}"
        for d in docs
    )

Three decisions are hiding in that function, and they all matter more than the word "stuffing" suggests.

The separator is \n\n, so the model can see where one passage ends and the next begins. Join them with a space and you have invited it to read two unrelated sentences as one claim.

The source is prefixed to each passage. This is what makes citation possible at all — the model cannot attribute a sentence to a file it was never told about. Note page + 1: post 8b flagged that page is zero-indexed, and page_label is the more correct field when the loader provides it.

And the metadata from post 8b's splitter is still here, having travelled through the splitter, the embedding, the store and the retriever to end up in a prompt. That is the payoff for caring about metadata four posts ago.

One historical note. LangChain used to offer create_stuff_documents_chain for this, and you will find it in a great deal of material online. It now lives in langchain-classic — langchain.chains.combine_documents does not exist in v1. The composition below is the current way, and it is also more legible, since the formatting is a function you can read instead of a helper you have to look up.

The Whole Thing

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from langchain.chat_models import init_chat_model

prompt = ChatPromptTemplate.from_messages([
    ("system",
     "Answer the question using only the context below. "
     "If the context does not contain the answer, say you do not know. "
     "Cite the source in brackets after each claim.\n\nContext:\n{context}"),
    ("human", "{question}"),
])

model = init_chat_model("gpt-5.6-luna")

chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | model
    | StrOutputParser()
)

chain.invoke("who approves a deployment?")
# 'Two approvals are required [handbook.pdf p.2].'

Every piece of that is from an earlier post. The dict literal coercing to RunnableParallel is post 7b. RunnablePassthrough carrying the question past the retrieval branch is post 7b. The prompt template is post 5, StrOutputParser is post 6, and the composed object gets batch, stream and async from post 7a for free.

Two instructions in that system prompt are doing the real work. "Using only the context below" is what converts a model that invents deployment policies into one that reports yours. "If the context does not contain the answer, say you do not know" is what stops it filling a retrieval failure with invention — and it is the single highest-value sentence in a RAG prompt, because retrieval will fail sometimes and you need that failure to be visible rather than fluent.

Here is the graph of that chain, generated from the chain itself with chain.get_graph().draw_mermaid():

graph TD;
	ParallelInput([Parallel context,question Input]):::first
	VectorStoreRetriever(VectorStoreRetriever)
	format_docs(format_docs)
	Passthrough(Passthrough)
	ParallelOutput(Parallel context,question Output)
	ChatPromptTemplate(ChatPromptTemplate)
	ChatModel(ChatModel)
	StrOutputParser(StrOutputParser)
	StrOutputParserOutput([StrOutputParserOutput]):::last
	ParallelInput --> VectorStoreRetriever;
	VectorStoreRetriever --> format_docs;
	format_docs --> ParallelOutput;
	ParallelInput --> Passthrough;
	Passthrough --> ParallelOutput;
	ParallelOutput --> ChatPromptTemplate;
	ChatPromptTemplate --> ChatModel;
	ChatModel --> StrOutputParser;
	StrOutputParser --> StrOutputParserOutput;
	classDef default fill:#f2f0ff,line-height:1.2
	classDef first fill-opacity:0
	classDef last fill:#bfb6fc

The fan-out and rejoin are visible: the input goes down two branches, one retrieving and formatting, one passing the question through untouched, and they meet at the prompt. The raw generated output escapes <, , and > as \3c, \2c and \3e; the node labels above are tidied for readability, and the structure is exactly as produced.

What the Model Actually Receives

Worth printing once, because "stuffing" stays abstract until you see it:

--- SystemMessage (394 chars) ---
Answer the question using only the context below. If the context does not contain
the answer, say you do not know. Cite the source in brackets after each claim.

Context:
[handbook.pdf p.2] Two approvals are required to approve any deployment.

[handbook.pdf p.3] During an incident the on-call engineer may waive the soak period.

[policy.docx p.1] Expenses over 500 USD need manager sign-off.

--- HumanMessage (26 chars) ---
who approves a deployment?

total: 420 chars  (~105 tokens at roughly 4 characters per token)

That is the entire input to the model. There is no hidden state, no memory of the corpus, nothing from the other thousands of chunks in the store. The model's knowledge of your handbook, for this request, is those three passages.

Which reframes k and chunk size as what they are: a budget. 105 tokens here, with three short chunks. Three 400-token chunks puts you nearer 1,300 tokens of context per question, every question, forever.

Where This Breaks

An honest finish, because what you have built is a working pipeline and not a good one yet.

graph LR
    S1["User: 'the answer was cut off'"] --> C1[Chunk boundary split the answer]
    S2["User: 'it cannot find ticket ABC-1234'"] --> C2[Embeddings miss exact identifiers]
    S3["User: 'it quotes the old policy'"] --> C3[Stale index, no re-indexing]
    S4["User: 'confident and wrong'"] --> C4[Retrieval failed, prompt let it improvise]
    C1 --> F1[Larger chunks, more overlap,<br/>structure-aware splitting]
    C2 --> F2[Hybrid search: keyword + vector]
    C3 --> F3[Re-index on change,<br/>deterministic ids]
    C4 --> F4[Score threshold, and an<br/>evaluation set that catches it]

Four failures, in the order you will meet them.

Chunk boundaries split answers. The fix is upstream, in post 8b, and it is why that post argued chunking sets your ceiling. Bigger chunks, more overlap, or header-aware splitting.

Embedding-only retrieval misses exact terms. Ask for ticket ABC-1234 and semantic similarity is the wrong tool — that string has no meaning to embed. Production systems run hybrid search, combining keyword matching with vector similarity, because each catches what the other misses.

No reranking. Retrieval returns the nearest chunks, not the most useful ones. A reranker is a second, more expensive model that scores the retrieved candidates properly — fetch twenty, rerank, keep three. It is usually the single biggest quality improvement available after chunking.

No evaluation. This is the real gap. You have no number telling you whether yesterday's chunk-size change made retrieval better or worse, so every adjustment is a guess that feels like progress. A set of fifty real questions with known correct sources, run on every change, is what turns RAG from a demo into engineering.

None of those four are exotic. They are the normal next steps, and they are a separate series rather than a paragraph.

The Series, End to End

Twelve posts, and the chain in this one uses something from almost all of them.

You learned what a token costs before writing code that spends them, so k and chunk size read as budget decisions rather than tuning knobs. You called the raw API before touching the framework, so init_chat_model is a convenience rather than magic. You built prompts as templates, got typed objects out instead of prose, and composed the pieces with a pipe operator — and then the retriever turned out to be a Runnable too, so it joined the same pipeline without ceremony.

That is the thing worth taking from the series. There is no RAG abstraction in LangChain v1 — no RAGChain class you configure. There is a retriever, which is a Runnable, and a prompt, which is a Runnable, and a model, which is a Runnable. RAG is a shape you assemble out of parts you already understand.

Which is also why the gaps above are tractable. Adding a reranker is another step in the pipeline. Adding hybrid search is a different retriever behind the same interface. You are not reaching for a new framework; you are changing one node in a graph you can print.

Every code example in this series was executed against pinned versions on the date stated at the top of each post. If you are reading this much later, treat the code as a guide and the current documentation as the authority — and post 2 explains how to tell the difference.

🧭 What's Next

• Beyond this series: the four failure modes above are where real RAG work lives — hybrid retrieval, reranking, chunking experiments driven by evaluation, and the observability to know which change helped. That is the subject of the next series, which picks up exactly where this pipeline stops being good enough.