RAG File Preparation and Chunking: From Messy Files to Reliable Evidence

A policy says overseas meals are reimbursed up to RM120, then excludes alcohol in the next sentence. Split those statements into unrelated chunks and a search may return the amount without the exception. Good file preparation preserves the whole rule, along with enough source information to check it. The worked example below follows that rule across a page break.

This chapter covers ingestion in our end-to-end RAG workflow: turning original files into trustworthy, retrievable passages. Once those passages are sound, continue to embeddings, retrieval and context management.

What you need before processing files

Keep an original-file store, a document inventory, a parser, a versioned transformation pipeline and a small set of real questions. Separate original bytes, extracted content and searchable chunks so you can repair one stage without losing evidence.

There is no single stack price: budget for conversion or OCR, optional model-based enrichment, embedding, storage and reprocessing. A local parser still consumes compute and maintenance time. A managed parser introduces usage charges and data-handling decisions. Measure cost per accepted document, including retries and manual correction.

1. Categorize files and establish access separately

Classify along several axes: file format, document purpose, language, business domain, freshness and extraction difficulty. A scanned invoice in Chinese needs a different route from an English Markdown handbook, even if both belong to finance.

Use explicit metadata where available. A folder or owner-provided document type is a useful signal. If a classifier proposes a category, keep its uncertainty and allow correction; do not silently discard documents because a predicted label looks irrelevant.

A category such as finance does not authorize everyone in finance. Copy permissions from the trusted source, record tenant and owner, and enforce the requester's current access before evidence reaches the model. Microsoft documents permission-aware search and security-filter patterns; support depends on the selected connector and service configuration. Azure access-control documentation.

Treat category filtering as a relevance choice. Overly narrow routing can miss cross-department answers. Treat permission filtering as mandatory. Test renamed groups, revoked access and deleted source files as well as successful queries.

2. Route each file to an appropriate extractor

These routing choices are editorial guidance, not a claim that a single product handles every file reliably. Keep extraction failures visible. An empty parse should not count as a successfully indexed document.

Compare parser options

See the primary Docling documentation, Unstructured partitioning guide, and Azure layout reference. Keep parser versions in the ingestion record so later regressions are reproducible.

3. Clean without rewriting the evidence

Extraction produces a draft representation of the source. Cleaning should remove artifacts introduced by the format while preserving what the document says. Start with repeatable rules: normalize encoding and whitespace, remove repeated navigation or page furniture, and repair line wraps only when the rule is reliable.

Do not globally delete hyphens: a wrapped word and a product identifier need different treatment. Do not remove every repeated line: a table header may be essential. Preserve negation, currency, measurement units, effective dates, superscripts and footnote relationships.

For tables, keep a machine-readable representation alongside a readable rendering. A retrieved row should identify its columns, units, document and table title. For a large table, repeat relevant headers with each row group. For totals across many rows, a database calculation can be more appropriate than asking a model to infer an aggregate from a few retrieved fragments.

If an LLM repairs extraction, save the original extraction and a change record. Flag uncertain cells rather than inventing values. A fluent rewritten paragraph can hide a missing exception or reverse a condition.

Deduplicate and version deliberately

An ingestion manifest should tell you exactly which source versions produced an index. For a changed file, replace its dependent chunks rather than appending another copy indefinitely. Keep the previous index available until the new one passes checks.

4. Attach metadata before splitting

Use stable document identity plus a version and chunk identity. Include source location, heading path and page or line references. Keep business classification and permissions as separate fields.

This is an illustrative record, not a database schema you can copy unchanged:

{
  "document_id": "travel-policy",
  "document_version": "2026-09-01",
  "chunk_id": "travel-policy:2026-09-01:004",
  "source_uri": "internal://policies/travel.pdf",
  "source_sha256": "<original-file-hash>",
  "category": "employee-policy",
  "language": "en",
  "tenant_id": "example-company",
  "allowed_group_ids": ["employees"],
  "heading_path": ["Travel", "Reimbursement", "Exceptions"],
  "page_start": 4,
  "page_end": 5,
  "parser_version": "<pinned-parser-version>",
  "chunker_version": "structure-v1",
  "text": "<verbatim cleaned source passage>"
}

The application must enforce the access fields; merely storing them changes nothing. Add bounding boxes when page highlighting matters. Record extraction time separately from a document's effective date. Never present the indexing date as proof that an old policy is current.

5. Compare chunking strategies

A chunk is a retrievable unit of evidence. Its size should follow the questions and document structure, with a hard cap based on the embedding service's tokenizer and request limits.

LangChain's recursive splitter uses an ordered separator list and permits custom length measurement. Its default character-based measurement should not be confused with an embedding-token budget. For languages without word spaces, check sentence punctuation and script-specific splitting. Recursive splitting documentation.

Docling's hybrid chunker begins with document hierarchy, splits oversized chunks using the selected tokenizer and can merge small adjacent chunks with matching headings. Its table options repeat headers, with a separate setting that permits omitting a header when it would overflow. Keep that setting in your evaluation record: a row fitting the limit does not establish that its meaning survived. Docling chunking.

Unstructured's by_title strategy uses detected section titles, with controls for page boundaries and combining small sections. Its size controls are character counts, so check embedding tokens after chunking. Inspect the detected elements if an ordinary list item becomes a heading and fragments the document. These concrete behaviors matter more than whether a feature is marketed as “semantic.” Unstructured chunking.

Late chunking versus contextual retrieval

Jina describes late chunking as deriving token representations with surrounding context, then pooling the representations corresponding to each chunk. It still requires boundary decisions. It is distinct from late-interaction retrieval, which compares multiple representations during search. Jina's technical explanation.

Anthropic's contextual retrieval adds a short document-aware explanation to a chunk before indexing it for embeddings and keyword search. The original passage remains the evidence; generated context needs verification. This changes the indexed text, whereas late chunking changes how a representation is produced. Anthropic's contextual retrieval article.

Do not combine unrelated tenants or permission scopes into a shared contextualization input. That applies both to generated explanations and long-context embedding passes. Record which source context contributed to a derived representation.

Worked example: preserve a policy exception across pages

Consider this invented extraction from a two-page policy. The amounts illustrate a parsing problem; they are not a real company's rules.

PAGE 4 — ACME INTERNAL — Travel policy
3.2 Meals
Meals are reimbursed up to RM80 per day. Original receipts are
required. For international travel, the limit is
PAGE 5 — ACME INTERNAL — Travel policy
RM120 per day. Alcohol is excluded from both limits.

A page-based splitter could index page 4 without the international limit and page 5 without the section heading. A naive cleanup could also delete “Alcohol is excluded” as a short trailing line. Neither problem is solved by choosing a larger embedding model.

First, identify the repeating page furniture and move it into provenance metadata. Join the interrupted sentence because the layout and sentence continuity support that join. Preserve the exclusion. The cleaned source becomes:

Travel policy > 3.2 Meals
Meals are reimbursed up to RM80 per day. Original receipts are
required. For international travel, the limit is RM120 per day.
Alcohol is excluded from both limits.

Create one structural chunk here: this short section is already a coherent answer unit. Set its page range to 4–5 and retain per-span page references if the interface must highlight the exact sentence. Index the title plus section path with the text, while retaining the unmodified cleaned passage separately for citation.

Now test two questions: “What is the international meal allowance?” and “Can I claim alcohol within the meal limit?” The retrieved evidence should support RM120 and the exclusion. For a general meal-budget question, returning only the RM80 sentence is incomplete because the travel type matters.

If a real section is too large, split it into children that repeat the heading path. Retrieve a matching child, then expand to the authorized parent or neighboring exception within the context budget. Check those expansions rather than assuming adjacent text is always useful. This example demonstrates why completeness and citation quality must be evaluated together.

6. Run a small chunking experiment

The policy example gives you a useful test: retrieval must preserve both the amount and the exception. Add it to a frozen sample containing prose, tables, scans, multilingual text and revised documents. Include unanswerable questions and restricted documents, and keep extraction and retrieval fixed while comparing chunkers.

For illustration, compare three candidate configurations: a 300-token window with 40-token overlap; a 600-token window with 80-token overlap; and section-based splitting with a 600-token cap. These are experiment inputs, not recommended universal defaults. Count tokens using the actual embedding tokenizer, including any heading prefix.

Inspect whether each configuration retrieves the complete evidence, preserves exceptions and produces a correct citation. Measure duplicate retrieval, final context tokens, indexing cost and answer quality. A longer chunk can improve completeness while wasting context; smaller chunks can match precisely while omitting the qualifying sentence.

Only add semantic boundaries, parent expansion or generated context when the failure examples justify the added work. Keep a held-out question set so you do not merely tune to the examples you have already inspected.

7. Set an ingestion acceptance gate

Before embedding the full corpus, manually compare a sample of rendered pages with their parsed output. Include the hardest files, not just the first ten. Check a table cell, a multi-column page, a footnote, a heading transition and a scanned page.

Reject empty outputs, missing access metadata, unresolved source references and chunks that exceed the supported token cap. Quarantine uncertain files with a reason and retry route. Track success by usable documents and answerable questions rather than raw chunk count.

Once the evidence is sound, continue to embedding selection, retrieval and context budgeting. Revisit ingestion whenever a retrieval failure turns out to be missing or damaged source material.