RAG Indexing, Part 1: Loading Documents and Splitting Them Properly

The Stage That Decides Your Quality Ceiling

Of the five stages in post 8a, this one gets the least attention and causes the most disappointment.

Embedding and retrieval feel like where the intelligence is, so that is where people tune. But retrieval can only return chunks that exist, and a chunk is whatever your splitter produced. If the sentence answering a question sits half in chunk 14 and half in chunk 15, then no embedding model, no reranker and no prompt engineering will recover it. You will retrieve half an answer and the model will confidently complete the other half from imagination.

Chunking sets the ceiling. Everything after it operates underneath that ceiling.

The Document Object

Everything downstream speaks one type, and it is simpler than you would expect:

from langchain_core.documents import Document

Document(
    page_content="Deployments require two approvals and a passing CI run.",
    metadata={"source": "handbook.pdf", "page": 0},
)

Text plus a dict. That is the whole contract. A loader's job is to produce these; a splitter's job is to turn big ones into small ones; a vector store indexes them; a retriever returns them.

Keep the metadata in view as we go. It looks like bookkeeping and it is what later lets you cite a source and filter a search — the difference between a toy and something people will trust.

Loading a PDF

# pip install langchain-community pypdf
from langchain_community.document_loaders import PyPDFLoader

docs = PyPDFLoader("handbook.pdf").load()
len(docs)   # 2  -> one Document per page

One Document per page, and the metadata is richer than you might assume. Here is page two of a two-page test file, printed whole:

page_content : 'On-call\nThe on-call rotation is weekly and starts on Monday at 10:00.'

metadata:
  source         = 'handbook.pdf'      <- for citations
  page           = 1                   <- ZERO-indexed
  page_label     = '2'                 <- what the page says it is
  total_pages    = 2
  title          = 'untitled'
  author         = 'anonymous'
  creator        = 'anonymous'
  producer       = 'ReportLab PDF Library - (opensource)'
  subject        = 'unspecified'
  keywords       = ''
  creationdate   = '2026-10-10T08:51:29+00:00'
  moddate        = '2026-10-10T08:51:29+00:00'
  trapped        = '/False'

Thirteen keys. Note that page is zero-indexed while page_label is the human page number — cite the wrong one and every citation in your product is off by one.

The \n inside page_content is worth noticing too. PDF text extraction gives you lines, not paragraphs, and that has consequences for splitting that we will get to shortly.

Loading a Word Document

# pip install docx2txt
from langchain_community.document_loaders import Docx2txtLoader

docs = Docx2txtLoader("policy.docx").load()
len(docs)         # 1  -> the whole file, as one Document
docs[0].metadata  # {'source': 'policy.docx'}

This is the lesson of these two loaders taken together: they share an interface, not a behaviour. PyPDF gives you one Document per page with thirteen metadata keys. Docx2txt gives you one Document for the entire file with exactly one key.

Nor does it preserve structure. The headings in that file come back as bare lines:

'Engineering Handbook'
''
'Expenses'
''
'Expenses over 500 USD need manager sign-off.'

A heading and a paragraph are now indistinguishable. Which matters, because you were about to split this text, and the only thing the splitter can see is blank lines.

So before using any loader, load one representative file and print what you got. Not the count — the actual content and the actual metadata. Fifteen seconds that saves you from discovering at retrieval time that your citations have no page numbers.

A Note on langchain-community

Something you should know, because almost no tutorial mentions it. Importing that package prints:

DeprecationWarning: `langchain-community` is being sunset and is no longer
actively maintained.

The sunset notice says it takes effect immediately and directs users to standalone integration packages — but it names no replacement for document loaders. langchain_classic.document_loaders is only a forwarding shim; reaching through it raises ModuleNotFoundError telling you to install langchain-community anyway.

In practice it works. Both loaders above ran correctly, and pip check reports no conflicts against langchain-core 1.6.7. But you are building on a package in maintenance mode with no announced successor, so it is worth knowing how little the loaders are actually doing for you:

from pypdf import PdfReader
from langchain_core.documents import Document

reader = PdfReader("handbook.pdf")
docs = [
    Document(page_content=page.extract_text(), metadata={"source": "handbook.pdf", "page": i})
    for i, page in enumerate(reader.pages)
]

Two lines, verified working, no langchain-community at all. A Document is content plus a dict, and pypdf and docx2txt are the libraries doing the real work in both cases — the loader classes are thin wrappers that pick the metadata for you.

Use the loaders; they are convenient and they handle more formats than you want to write yourself. Just know that the escape hatch is this short, so a sunset package is a manageable risk rather than a dependency you are trapped in.

Chunking: The Theory

Two parameters, and both are trade-offs rather than settings with correct values.

Chunk size trades precision against context. Small chunks retrieve precisely — the returned text is almost entirely relevant — but may be too small to contain a complete answer. Large chunks almost certainly contain the answer somewhere, surrounded by material that dilutes the embedding and costs input tokens on every query that retrieves it. A 2,000-character chunk about deployments, on-call and expenses has an embedding that is specifically about none of those things, which makes it mediocre at matching all three.

Chunk overlap is insurance against bad boundaries. By repeating the tail of one chunk at the head of the next, a sentence cut in half appears intact in at least one of them. It costs duplicated storage and the chance of retrieving two chunks that say the same thing — which is exactly the redundancy problem MMR exists to solve in post 8d.

Both parameters are measured in characters by default, not tokens. Your cost and your context limit are in tokens. The ratio is roughly four characters per token for English prose and considerably worse for other languages and for code, so a 1,000-character chunk is about 250 tokens of English — and if your documents are Vietnamese, materially more.

Chunking: Measured

Here is what those parameters actually do. A 163-character passage, chunk_size=60, chunk_overlap=12:

position   0         20        40        60        80       100       120       140    163
           |---------|---------|---------|---------|---------|---------|---------|------|
chunk 0    [=====================================]                                        [  0: 55] 55 chars
chunk 1                                   [==================================]            [ 48:100] 52 chars
chunk 2                                             [====================================][ 92:150] 58 chars
chunk 3                                                                     [===========] [144:163] 19 chars
                                          ^^^^^^^   ^^^^^^^^                ^^^^^^
                                          overlap   overlap                 overlap
                                          'CI run.' 'log. The'              'during'

Read the numbers, because they are not what the parameters say.

Sizes came out 55, 52, 58 and 19 — never the 60 you asked for. Overlaps came out 7, 8 and 6 characters, not 12. That is not a bug: chunk_size and chunk_overlap are upper bounds that the splitter snaps to the nearest separator, because a chunk cut mid-word is worse than a chunk slightly under size. You asked for at most 60 characters ending somewhere sensible, and that is what you got.

The last chunk is 19 characters. Short trailing chunks are normal, and they are weak retrieval candidates — not enough text to embed meaningfully. One more reason overlap matters: 'during' at least ties it to the chunk before it.

The Splitter That Silently Does Nothing

Same input, same parameters, two splitters:

input: 163 characters, containing no blank lines

CharacterTextSplitter(chunk_size=60, chunk_overlap=12)
   -> 1 chunk of 163 characters        chunk_size IGNORED

RecursiveCharacterTextSplitter(chunk_size=60, chunk_overlap=12)
   -> 4 chunks of 55, 52, 58, 19       as expected

CharacterTextSplitter splits on exactly one separator, and its default is "\n\n". Hand it text with no blank lines and there is nowhere to split, so it returns the input whole — no error, no warning, your 163-character "chunk" sails into the vector store.

And recall what the PDF loader produced: lines separated by single \n, not blank lines. This failure is not a contrived example; it is the default behaviour on the most common input in RAG.

Giving it the right separator fixes it:

CharacterTextSplitter(chunk_size=60, chunk_overlap=12, separator=" ")   # 4 chunks

But now you are choosing a separator per document type and getting it wrong silently when a file does not match. Which is the argument for the other splitter.

Why Recursive Is the Default

RecursiveCharacterTextSplitter tries a list of separators in order of how much structure they represent, falling back only when the current one leaves a piece too big:

graph TD
    A[Text too long] --> B["Split on paragraph break: \n\n"]
    B --> C{Pieces now<br/>under chunk_size?}
    C -->|Yes| DONE[Done]
    C -->|No| D["Split the oversized piece on line break: \n"]
    D --> E{Under<br/>chunk_size?}
    E -->|Yes| DONE
    E -->|No| F["Split on space"]
    F --> G{Under<br/>chunk_size?}
    G -->|Yes| DONE
    G -->|No| H["Split on raw characters"]
    H --> DONE

Its default separator list is exactly ["\n\n", "\n", " ", ""] — paragraph, line, word, character. The empty string is the guaranteed terminal case, which is why it can never fail to split.

The important property is that it only descends when it has to. A document with clean paragraphs is cut at paragraph boundaries. A wall of text with no paragraphs gets cut at lines, then words. You get the most structure the document can offer, without configuring anything per format. That is why it is the default recommendation, and why the character splitter is best understood as a teaching tool for the parameters rather than the one you ship.

Splitting by Structure Instead of Size

Size-based splitting ignores meaning. A handbook has sections, and a chunk that spans the end of "Deployments" and the start of "On-call" is a chunk about neither.

MarkdownHeaderTextSplitter splits on headings instead, and puts the heading trail into metadata:

from langchain_text_splitters import MarkdownHeaderTextSplitter

splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")]
)
sections = splitter.split_text(markdown_text)

Headings become inherited metadata on every chunk beneath them:

# Engineering Handbook
│
├── ## Deployments
│    │   chunk → 'Two approvals and a passing CI run are required.'
│    │   meta  → {h1: 'Engineering Handbook', h2: 'Deployments'}
│    │
│    └── ### Hotfixes
│            chunk → 'A hotfix may ship with one approval.'
│            meta  → {h1: 'Engineering Handbook', h2: 'Deployments', h3: 'Hotfixes'}
│
└── ## On-call
         chunk → 'The rotation is weekly and starts Monday at 10:00.'
         meta  → {h1: 'Engineering Handbook', h2: 'On-call'}

Notice the Hotfixes chunk carries all three levels. Its own text never mentions deployments, but its metadata records that it lives under that heading — so a retrieved chunk arrives knowing where it came from, and you can filter a search to a section without having inferred anything.

By default the headings are stripped from the text and kept only in metadata. strip_headers=False leaves them inline as well, which is often worth doing: a chunk whose text begins ## On-call embeds slightly better, because the topic is stated in the content rather than only beside it.

What You Will Actually Use: Both

Header splitting respects meaning but produces sections of wildly uneven size — a two-line section and a six-page one. Size splitting produces even chunks but ignores meaning. The production answer is to run them in that order:

sections = MarkdownHeaderTextSplitter(headers_to_split_on=heads).split_text(md)
chunks = RecursiveCharacterTextSplitter(
    chunk_size=400, chunk_overlap=50
).split_documents(sections)

Note split_documents rather than split_text — it takes Documents and returns Documents, carrying the metadata through. Verified: three header sections became five sized chunks, and every one of the five still had its h1/h2/h3 trail.

So a long section gets cut to size, and each piece still knows it is part of "Deployments". You get even chunks that have not forgotten their structure, which is the best of both and costs one extra line.

That is the indexing path up to the point where text becomes numbers. Next, it becomes numbers.

🧭 What's Next

• Post 8c: Embeddings and the Chroma Vector Store — turning these chunks into vectors with OpenAI embeddings, building a store you can persist and reload, and inspecting exactly what Chroma holds so the vector store stops being a black box.