RAG Indexing, Part 2: Embeddings and the Chroma Vector Store

From Text to Coordinates

Post 8b ended with a pile of chunks. Each is a short passage with metadata recording where it came from. Searchable by keyword, but that is not what we want — the question "who signs off a release?" shares no words with the passage "deployments require two approvals", and a keyword index will never connect them.

An embedding model fixes that by converting text into a position. Feed it a passage and it returns a list of numbers — a vector — placing that text somewhere in a space with a few hundred to a few thousand dimensions. The model is trained so that passages meaning similar things land near each other, regardless of the words they use.

That is the whole trick, and it is worth being precise about what it does and does not mean. Distance in that space approximates similarity of meaning as the model understands it. It is not a semantic truth; it is a learned approximation, and it is why retrieval sometimes returns something that is topically adjacent but useless. The model put it nearby. Nearby was wrong.

Embedding Chunks with OpenAI

# pip install -U "langchain[openai]"
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

vectors = embeddings.embed_documents([c.page_content for c in chunks])
len(vectors)      # one vector per chunk
len(vectors[0])   # 1536

Specify the model. This is the first trap, and it is in the default: OpenAIEmbeddings() with no arguments resolves to model='text-embedding-ada-002', which is the legacy generation. It still works, so nothing warns you, and you have quietly chosen a model that costs five times as much as text-embedding-3-small for the same number of dimensions and worse quality. Name the model every time.

Two methods, not one. embed_documents takes a list and returns a list of vectors. embed_query takes a single string and returns one vector:

embeddings.embed_query("who signs off a release?")

They exist separately because some embedding models encode a document and a query differently — a document is a thing being described, a query is a request for one. With OpenAI's models the two paths are the same, but the interface keeps the distinction so that a model which needs it works without changing your code. Use the right one anyway; it costs nothing and it is correct if you ever swap models.

Watch the parameter named chunk_size. OpenAIEmbeddings has one, defaulting to 1000, and it has nothing to do with the text chunking from post 8b. Here it means how many texts to batch into a single API request. Two different concepts, the same word, one line apart in a RAG pipeline — one of the more unfortunate naming collisions in this stack.

dimensions lets you shrink the output, but only on text-embedding-3 and later. Fewer dimensions means less storage and faster search for some loss of fidelity, which is a real trade when a corpus gets large.

What This Stage Costs

graph LR
    C["10,000 chunks<br/>~400 tokens each"] --> E["Embedding model<br/>text-embedding-3-small"]
    E --> V["10,000 vectors<br/>1,536 floats each"]
    V --> S[("Chroma collection<br/>~61 MB")]
    Q["1 question"] -.-> E
    E -.-> QV["1 vector"]
    QV -.-> S

Indexing is a one-off cost; querying is forever. For a corpus of 10,000 chunks at roughly 400 tokens each — four million tokens:

Eight cents to index a corpus. Embedding is the cheapest stage in RAG by a wide margin, which has a useful consequence: re-indexing is affordable, so chunking strategy is not a one-way door. If post 8b left you worried about picking the wrong chunk size, this table is the reassurance — you can re-split and re-embed the whole corpus for less than a cup of coffee, and you should, as an experiment, rather than agonising up front.

The storage column is the part that scales into something you notice. 3-large is 6.5× the price and 2× the bytes for a quality improvement that may or may not matter to your corpus. Start with 3-small; move up only when you have a retrieval failure you can show.

Verify these prices before you rely on them. They move more often than the APIs do, and the figures above came from a third-party aggregator rather than the provider's own page.

Building the Store

# pip install langchain-chroma
from langchain_chroma import Chroma

store = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    collection_name="handbook",
    ids=[f"handbook-{i}" for i in range(len(chunks))],
    persist_directory="chroma_db",
)

One call embeds every chunk and writes the vectors, the text and the metadata to disk. Four arguments worth understanding.

persist_directory is what makes this survive the process. Leave it out and the collection lives in memory and vanishes — fine for a notebook, and the cause of a lot of "why is my store empty" confusion in scripts.

ids are optional and worth supplying. Omit them and Chroma assigns UUIDs, which means re-running your indexing script creates duplicates of everything rather than replacing it. A deterministic id derived from the source — file path plus chunk index — makes indexing idempotent, which matters the first time you re-index a corpus that is already loaded.

Reloading later uses the constructor instead:

store = Chroma(
    collection_name="handbook",
    embedding_function=embeddings,   # note: embedding_function, not embedding
    persist_directory="chroma_db",
)

That parameter name really does change between the two paths — embedding on from_documents, embedding_function on the constructor. It is the kind of inconsistency that costs twenty minutes the first time, so it is worth knowing in advance.

On disk you get a chroma.sqlite3 file and a directory named after the collection's internal id. The local store is SQLite, which is why the SQLite version note from post 8a exists.

What Is Actually in There

A vector store stops being a black box the moment you look inside it. get() returns the contents:

raw = store.get()
sorted(k for k, v in raw.items() if v is not None)
# ['documents', 'ids', 'included', 'metadatas']

Four parallel lists, aligned by position. Here is the shape, from a real collection:

                 ids            documents                              metadatas
  index 0   'handbook-0'   'Deployments require two approvals.'   {source: handbook.pdf, h2: Deployments}
  index 1   'handbook-1'   'The on-call rotation is weekly.'      {source: handbook.pdf, h2: On-call}
  index 2   'policy-0'     'Expenses over 500 USD need sign-off.' {source: policy.docx,  h2: Expenses}

            embeddings  ->  None          <- NOT returned by default

Note the missing column. Embeddings are not returned unless you ask, which is a sensible default — 1,536 floats per row is a lot of output you rarely want. When you do want them:

raw = store.get(include=["embeddings", "documents", "metadatas"])
len(raw["embeddings"][0])   # 1536

Those metadata dicts are the h1/h2 trails that MarkdownHeaderTextSplitter attached in post 8b. They travelled from the heading structure of a Markdown file, through the splitter, into the store — and in a moment they become a search filter.

Managing Documents

Indexing is not a one-time event. Documents get edited, deleted and added, and a store you cannot maintain is a store you will eventually rebuild from scratch every time anything changes. All four operations, verified:

# fetch one by id
store.get(ids=["handbook-1"])

# fetch by metadata
store.get(where={"source": "policy.docx"})

# add
store.add_documents([Document(page_content="Travel needs 14 days notice.",
                              metadata={"source": "policy.docx", "h2": "Travel"})],
                    ids=["policy-1"])

# replace the content behind an existing id
store.update_document("handbook-0",
    Document(page_content="Deployments require THREE approvals.",
             metadata={"source": "handbook.pdf", "h2": "Deployments"}))

# remove
store.delete(ids=["policy-1"])

A second naming inconsistency to note while you are here: get() filters with where=, while the similarity search in the next post filters with filter=. Same concept, two spellings, depending on which method you are calling.

The update path is where deterministic ids pay for themselves. With ids derived from the source, re-indexing a changed file overwrites exactly its chunks. With UUIDs, you have no way to find the old chunks of handbook.pdf to remove them, and your store slowly fills with stale duplicates that retrieval happily returns.

Everything above survived a reload from disk in a fresh process, including the update.

A Picture of the Space, With a Caveat

It helps to have a mental image of what the store holds, as long as the image comes with a warning attached.

        (a projection of 1,536 dimensions down to 2 — the real space
         has no such flat picture, and distances here are illustrative)

   ^
   │        · deployments require two approvals
   │      · approvals are recorded in the audit log
   │   ?  ← "who signs off a release?"
   │
   │                         · the on-call rotation is weekly
   │                       · on-call may waive the soak period
   │
   │                                        · expenses over 500 USD
   │                                      · receipts within 30 days
   └───────────────────────────────────────────────────────────────>

Chunks about the same subject cluster, because the model placed them near each other. The question lands near the cluster that answers it despite sharing no words with it — which is the entire reason this machinery exists.

The caveat is not pedantry. In 1,536 dimensions, nearly all pairs of random points are roughly the same distance apart, and "clusters" are far less tidy than any two-dimensional drawing suggests. Use the picture to reason about why retrieval works, never to predict what it will return. For that, run the query.

What You Have

An indexed corpus that survives restarts: text, metadata and vectors, addressable by id, filterable by metadata, and cheap enough to rebuild that your chunking decisions stay reversible.

What you do not have yet is a way to ask it anything. get() fetches by id and by metadata — it does not search by meaning. That is next.

🧭 What's Next

• Post 8d: Retrieval and Generation — similarity search and what its score actually means, maximal marginal relevance for when your top results all say the same thing, turning the store into a Runnable retriever, and composing the whole thing into an answer chain with LCEL. Plus an honest account of where this naive pipeline breaks.