Chunking strategy matters more than embedding model choice

It's common to see a RAG project spend real engineering time comparing embedding models — benchmarking three or four candidates on retrieval accuracy — while the chunking step that produced the text being embedded was a single split every 512 tokens call nobody revisited. That ordering of effort is usually backwards, because no embedding model can encode context that was severed at the chunk boundary before the embedding step ever saw it.

What a bad boundary actually costs

Fixed-size chunking splits on token count with no regard for what's actually at that boundary. A chunk boundary that lands mid-sentence embeds a sentence fragment on each side, and a fragment carries less usable meaning than a complete sentence — the embedding model does its best with what it's given, but what it's given is worse than it needed to be. It gets worse with structured content: a table split across two chunks means neither chunk contains a complete row-to-column mapping, and a retrieval system that surfaces just one of those chunks hands the model a table it can't actually interpret correctly, regardless of how good the embedding was.

Chunking with structure in mind

The fix isn't a smarter embedding model, it's chunking that respects the document's actual structure: split on paragraph or section boundaries first, and only fall back to a fixed size when a section is too large to embed as one chunk. For structured documents — anything with headers, tables, or lists — chunk-aware-of-structure means keeping a table intact as its own chunk rather than letting a token-count splitter cut through the middle of it, even if that chunk ends up larger or smaller than your target size.

A useful reframe: the chunking step is doing information design, not just text splitting. The question isn't "how do I cut this into roughly-equal pieces," it's "what's the smallest unit of this document that still means something on its own, out of context." A single FAQ entry, one complete table, one section with its heading — these are chunk boundaries chosen because of what the content is, not because of where a token counter happened to land.

Overlap helps, but it's not the fix

Overlapping chunks (repeating the last N tokens of one chunk at the start of the next) is a common mitigation, and it does help — a sentence cut at a boundary in chunk one might appear whole in chunk two's overlap region. But overlap is a partial patch over a structural problem, not a substitute for chunking well in the first place. It also has a real cost: overlapping content means the same information gets embedded and stored multiple times, which increases both storage and the chance that retrieval returns near-duplicate chunks instead of diverse, complementary ones.

The practical order of operations, if chunking hasn't gotten real attention yet: fix chunk boundaries to respect document structure first, measure retrieval quality again, and only then decide whether embedding model choice is still the bottleneck. Often it isn't anymore.