Chunking Is Product Design (And Most RAG Systems Prove It)

Words you need

• Chunk -- A small piece of a document. You cut a long text into chunks so the system can find and use the right piece when someone asks a question.
• Token -- Roughly a word or part of a word for the AI. Models read and count text in tokens, not always whole words.
• RAG -- Retrieval-Augmented Generation. The system fetches the right chunks first, then the AI answers using those chunks instead of guessing from memory.
• Metadata -- Extra info you save with each chunk (e.g. title, section name, when it was created). You use it to filter and show where an answer came from.

I watched a team spend three weeks tuning their reranker. Fancy vector search. Custom embeddings. The answers were still garbage.

The problem? They were splitting every document into 500-token blocks. Definitions got separated from examples. Steps got ripped apart. The retrieval was technically working -- it just had nothing good to retrieve.

Chunking is the first decision in a RAG system. Get it wrong and nothing downstream can save you.

What chunking actually is

You take a long document and cut it into smaller pieces. Each piece should stand on its own. When a user asks a question, the system finds the best pieces and feeds them to the AI as context.

That's the theory. In practice, most people treat it like a text-splitting exercise. "500 tokens with 50-token overlap." Done.

It's not done. Chunking is a product decision. It defines what "one unit of context" means for your system. It decides what the AI can cite as a source. It determines whether users get a full thought or half a sentence.

For the full picture of how RAG fits together, start with the series map and then RAG at inference time.

The "just do 500 tokens with overlap" lie

Every tutorial starts here because it's easy to code. Not because it works.

What actually happens:

• You split a definition from its example. The AI gets one without the other.
• You split code from its explanation. The AI sees code it can't explain.
• You return half a thought. The AI guesses the rest. Usually wrong.

Overlap doesn't fix this. It just makes the same bad split slightly less bad. You're still cutting through the middle of ideas.

Different content needs different chunking

This is the part most teams skip. One strategy for everything produces answers that feel random.

• Articles and essays -- Cut at headings. One section = one chunk (when it fits). If a section is long, cut at paragraph boundaries. Never mid-sentence.
• How-to guides and runbooks -- Cut at steps. One step per chunk. Keep "why we do this" attached to "how to do it." Save labels like "step", "warning", "prerequisite" so you can filter later.
• Code -- Cut at function or class boundaries. Keep the docstring or comment with its code block. If you can't parse structure, at least cut at blank lines -- still better than mid-line.
• Short posts -- A tweet or update is already one chunk. Don't split it further. You'll destroy the point.

The cost: maintaining content-type-specific chunking logic. It's more code. More configuration. But the alternative is universally mediocre retrieval.

What to save with each chunk

For every chunk, store:

• The text itself
• A stable ID that never changes
• Source document ID and a "title path" (e.g. "Article name > Section > Subsection")
• Content type (article, how-to, code, post)
• Which chunking rules produced it (version number or rule name)

That last one matters more than people think. You will change your chunking strategy. If you can't tell which chunks were made with old rules vs new rules, you can't compare results. You can't debug regressions. You're flying blind.

The title path trick

If your content has headings, save the heading hierarchy for each chunk. "Article title > Section name > Subsection name."

This unlocks three things:

1. Filtering -- Search only within a specific section (e.g. "Safety" or "Getting Started")
2. Citations -- Show users exactly where an answer came from
3. Grouping -- Cluster related chunks by topic

You don't need to change your model for this. Just save the path. Answer quality goes up because the user sees provenance, and retrieval gets more precise with metadata filters.

How to test if your chunking works

Don't trust a retrieval score in isolation. Test with real questions.

Write 20 questions your users would actually ask. Run your system. Look at the top 10 chunks returned for each question. Read them. Ask yourself: "Could I answer this question using only these chunks?"

If the answer is no, fix the chunking. Don't reach for a fancier reranker or a bigger model. The chunks are the foundation. Fix those first.

What to do next

Pick one content type you have the most of. Try two chunking strategies: (1) by headings or steps, (2) by fixed 500-token blocks. Run the same 10 real questions against both. See which chunks give better answers.

Use the winner as your baseline. Tag every chunk with a version number. When you change strategies later, you'll have a clean comparison.

The question to ask yourself: what does "one piece of context" mean for your content?