What is document chunking?
Document chunking is the process of splitting long source documents into smaller text units before embedding and indexing – so each HNSW neighbor is a focused passage rather than an entire file, which is essential for precise RAG and semantic search over manuals, papers, and knowledge bases.
Why does chunking matter more than many teams expect?
Embedding models and generators both have practical length limits, but the deeper issue is representation quality. One vector for a whole report averages many topics; a query about a single procedure then matches a muddy centroid. Chunks that are too large still mix ideas and waste context windows when stuffed into a prompt. Chunks that are too small read like mid-sentence fragments – humans and generators lack the surrounding meaning, and retrieval may return incomplete facts. Good chunking aims for pieces that stand alone when read, stay small enough for sharp embeddings, and preserve the author’s train of thought across boundaries. When RAG answers are weak, the fault is often the chunk set HNSW is searching – not only ef or the generator.
There is no universal size; strategies differ by document shape and query style.
What chunking strategies are commonly used?
Fixed-size (or token-count) windows with overlap are a robust baseline – for example roughly 100-200 words with a substantial overlap so sentences at boundaries are not orphaned. Structure-aware splits follow headings, paragraphs, Markdown sections, HTML blocks, or code function boundaries when the format is reliable. Mixed rules clamp paragraph chunks between min and max lengths so headers alone do not become useless objects. Semantic chunking groups sentences by embedding similarity until a topic shift appears – more coherent, more compute. Heavier approaches use a language model to propose self-contained propositions or let an agent pick a strategy per document. Late chunking embeds a long context first, then pools token vectors into chunk embeddings so each piece retains more document-wide context. Skip chunking entirely when items are already short and complete (FAQs, tickets); splitting those can hurt.
Those pieces become Weaviate objects that HNSW indexes.
How should chunked documents be stored and queried in Weaviate?
Pre-chunk in your ingest pipeline, then insert one object per chunk with the text plus metadata: source id, title, chapter, chunk index, and any tenant or ACL fields. That structure supports filtered HNSW search, ordered reconstruction for prompts, and citations back to the original file. Configure a vectorizer on the chunk text (or bring-your-own embeddings) so each object gets a passage vector. At query time, retrieve top chunks with nearText or hybrid search, optionally reorder them by source position before generation. Post-chunking architectures keep document-level vectors first and only split high-value hits at query time to save storage – useful at huge scale, but most RAG starters pre-chunk everything for simple, fast retrieval. Re-chunk and re-embed when you change size, overlap, or embedding model; old and new chunk graphs are not interchangeable.
Evaluation should judge chunks, not only end-to-end chat demos.
How do you know your chunking strategy is working?
Spot-check: does each chunk make sense alone? Measure retrieval recall on labeled question-passage pairs while holding the embedder and HNSW settings fixed, then vary size and overlap. Watch for header-only chunks, tables shredded into nonsense, and near-duplicate overlapping windows crowding results. Track generator failures that cite partial instructions cut at a bad boundary. Prefer iterating chunk rules before endless prompt tweaks. Match strategy to corpus – code by AST units, policies by section headers, chatty blogs by sliding windows – and document the chosen parameters in your reproducibility notes.
Document chunking is how long text becomes a searchable HNSW corpus of passages; get the cuts right and retrieval and RAG both improve. Next, read "What is cross-encoder reranking?" for scoring the shortlist after chunk retrieval, or "What is multi-vector document retrieval?" when one document needs several vectors instead of a single chunk list.