Structure-aware ingestion#

Use structural chunks when headings, code, tables and source provenance should survive ingestion. The engine normalizes explicit document representations, packs retained content, and measures the complete text sent to embeddings.

Install fastrag-ai[normalization] for the default BPE tokenizer and Markdown/reStructuredText adapters. HTML normalization uses the Python standard library. An explicitly selected byte tokenizer needs no optional dependency for plain text or HTML.

Choose a representation#

An ingestion Parser converts source bytes into Document.content and sets metadata["content_format"]. A Normalizer reads that already parsed string into an ordered tree and an independent source coverage partition. Normalizers keep the original string; they do not fetch URLs or execute markup.

Representation	Structural adapter	Important behavior
`text`	`TextNormalizer`	Retains the whole string; no MIME sniffing. Missing legacy format metadata uses text.
`markdown`	`MarkdownNormalizer`	CommonMark with tables; headings, fenced/indented code, lists, links and original syntax.
`rst`	`RstNormalizer`	Docutils structure with original source locations; includes/raw execution disabled.
`html`	`HtmlNormalizer`	Readable serialization of headings, lists, tables, code and links; script/style/template/comment exclusions recorded.
`docling`	Native `DoclingChunker`	Serialized `DoclingDocument`, routed separately; not a structural textual adapter.

StructureAwareChunker selects a registered adapter by explicit format. Unknown formats raise an actionable error. ChunkerRouter validates each selected route: use ContentFormatFilter to send native Docling to DoclingChunker, or request DoclingParser(output_format="markdown") and use structural Markdown chunks. This second path describes the converted Markdown, not original spreadsheet cell coordinates or PDF byte offsets. See Ingestion for binary routing.

Measure the actual embedding input#

This complete offline recipe exercises packing and independent validation. Its tokenizer counts UTF-8 bytes, so its limits are deliberately byte units.

from fastrag.ingestion.steps.chunking import (
    StructureAwareChunker,
    Utf8ByteTokenizer,
    build_embedding_text,
    validate_structure_chunks,
)
from fastrag.models import Document

parent = Document(
    "orchard-guide",
    "local-guide",
    "# Orchard\n## Water\nWater pears weekly.\n## Soil\nTest soil each spring.\n",
    metadata={"content_format": "markdown"},
)
chunker = StructureAwareChunker(
    tokenizer=Utf8ByteTokenizer(),
    target_tokens=100,
    max_tokens=160,
    min_preferred_tokens=20,
    cache=False,
)
normalized = chunker.normalizers.normalize(parent)
pieces = list(chunker.split_normalized(normalized))
report = validate_structure_chunks(normalized, pieces, chunker.tokenizer, 160)
assert report.chunk_count == len(pieces) > 0
for content, metadata in pieces:
    embedding_text = build_embedding_text(content, metadata)
    assert chunker.tokenizer.count(embedding_text) <= 160
    assert metadata["embedding_token_count"] == chunker.tokenizer.count(embedding_text)

In a lazy graph, replace a character splitter with StructureAwareChunker(cache=cache) and retain embedder(chunker(parser(sources))). The existing Embedder base uses build_embedding_text for marked structural chunks while preserving display content. Legacy chunks retain their previous embedding input. Constructing the graph still performs no work.

The default TiktokenTokenizer counts cl100k_base with encode_ordinary, including literal special-token strings as ordinary content. Tiktoken may need its encoding cache on first use; prepare that cache for offline deployments. This is a configured-tokenizer guarantee, not a promise about an arbitrary embedding model’s tokenizer. Choose the provider’s matching encoding or inject its tokenizer. Tiktoken’s encoding API describes the underlying counting and Unicode-offset behavior.

A custom Tokenizer supplies a stable nonsecret identity, a nonnegative integer count(text), and ordered Unicode character boundaries(text) including zero and the final offset. Counts need not be additive or monotonic. The engine checks complete serialized candidates, including repeated headings, fences, table headers and overlap. Built-in tokenizer hints reduce large-block work; arbitrary custom tokenizers retain exhaustive feasibility checks, which can be expensive for large documents.

Packing, atomic structures and overlap#

Fitting direct sections remain intact. Small adjacent sibling sections can merge toward the target; crossing top-level sections requires the explicit merge option. Oversized sections pack adjacent blocks, then split oversized prose, code, table rows or list items with context. Code prefers logical lines and function boundaries; tables repeat schema, and nested lists repeat parent labels when those fit. Preservation flags relax these preferences while keeping retained source coverage. They do not make oversized objects fit automatically.

The target is a packing preference, and the maximum measures final embedding input. The default split_with_context policy enforces the maximum or raises when mandatory context cannot fit. preserve_and_flag deliberately permits oversized atomic output and records overflow; consumers must handle that before sending it to a provider. Splitting an indivisible table row or code line can record atomic structure loss. Read the flags when deciding whether the result is suitable for your application.

Overlap repeats only eligible trailing prose at sentence boundaries within the configured ceiling. It does not cross sections or add overlap to code/tables. Repeated overlap has separate provenance and does not count as newly retained source. Heading-only documents may yield no body chunks; empty and hierarchy-only results are explicitly classified by validation.

Keep provenance and parents#

Offsets are half-open Unicode character ranges into the parsed parent’s original string. Node spans may overlap or be noncontiguous; CoverageUnit ownership plus explicit Exclusion ranges partitions the source once. HTML output is a readable serialization, so display offsets and source offsets are different coordinates.

Chunk metadata includes document/section lineage, heading paths, primary source spans, retained intervals, serialization maps, repeated context, token counts and overflow/atomic flags. Consecutive compatible mapping runs are coalesced losslessly. structure_metadata is a lightweight node-ID/attribute projection; complete interpreted node metadata stays on the normalized parent. Do not treat it as a full parent document. validate_structure_chunks independently checks coverage, maps, lineage, counts and ordering on fresh and persisted output.

Store normalized parents separately if retrieval needs larger context. This standalone recipe resolves a structural hit without depending on a database.

from fastrag.ingestion.steps.chunking import StructureAwareChunker, Utf8ByteTokenizer
from fastrag.models import Document
from fastrag.retrieval import SectionIndex

parent = Document(
    "orchard-guide",
    "local-guide",
    "# Orchard\n## Water\nWater pears weekly.\n",
    metadata={"content_format": "markdown"},
)
chunker = StructureAwareChunker(tokenizer=Utf8ByteTokenizer(), cache=False)
normalized = chunker.normalizers.normalize(parent)
content, metadata = next(chunker.split_normalized(normalized))
hit = Document("retrieved-hit", parent.source_id, content, metadata=metadata)
index = SectionIndex([normalized])
sections = index.resolve(hit, ancestors=True)
assert sections and all(section.document_id == parent.id for section in sections)
assert any("Water pears weekly." in section.content for section in sections)

SectionIndex.resolve returns every merged source section in source order and can deduplicate ancestors. A missing parent/section raises KeyError; choose missing="skip" for an available subset. Parent expansion changes neither stored vectors nor prompts automatically: select and budget that context in your retrieval/query application. The in-memory index has no persistence or eviction policy; provide those in the application when parents outlive the process.

Extend normalization#

Register an adapter on the registry passed to your chunker. A stable identity must distinguish output-changing options. Duplicate registrations require explicit replace=True. Validate source slices, lineage and a complete nonoverlapping retained/excluded coverage partition before returning.

This minimal adapter uses only public immutable models. A richer adapter adds section/heading/list/table nodes under the same contracts.

from fastrag.ingestion.steps.chunking import StructureAwareChunker, Utf8ByteTokenizer
from fastrag.models import Document
from fastrag.normalization import (
    CoverageUnit,
    NormalizedDocument,
    NormalizedNode,
    NormalizerRegistry,
    SourceSpan,
)


class RecordNormalizer:
    identity = "records:whole-source:v1"

    def normalize(self, document):
        span = SourceSpan(0, len(document.content))
        root_id = f"{document.id}:root"
        root = NormalizedNode(root_id, "document", document.content, (span,))
        units = (
            (CoverageUnit(f"{root_id}:unit", root_id, span),)
            if document.content
            else ()
        )
        result = NormalizedDocument(document, "records", (root,), units)
        result.validate()
        return result


registry = NormalizerRegistry()
registry.register("records", RecordNormalizer())
chunker = StructureAwareChunker(
    normalizers=registry, tokenizer=Utf8ByteTokenizer(), cache=False
)
parent = Document(
    "record-1", "local", "Pear: water weekly.", metadata={"content_format": "records"}
)
pieces = list(chunker.split(parent))
assert len(pieces) == 1 and pieces[0][0] == parent.content
assert "records" in chunker.accepted_content_formats

Customizing covers byte-parser and pipeline-step extension hooks. A normalizer extension adds interpretation of a representation; it does not require replacing the format-independent packing engine.

Optional semantic boundaries#

Heuristic structural splitting is deterministic and invokes no chat or embedding model for boundaries. semantic_mode="disabled" disables sentence preferences. For explicit chat assistance, inject ChatBoundaryRefiner(llm=chat, identity=..., allow_external_transmission=True) into StructureAwareChunker(semantic_mode="llm", semantic_refiner=...) and evaluate asynchronously. The refiner sends eligible plain leaf paragraphs in Markdown/RST, never code/tables/lists/HTML or rewritten content, and accepts only allowed source offsets in strictly validated JSON. Timeout, retry/backoff, concurrency, input/response limits and a bounded instance-local cache govern calls. Ordinary provider/validation failures use the same deterministic heuristic fallback; cancellation propagates. Set the chat provider’s output-token limit too. Diagnostic calls/cache/fallback observations are runtime reports, not identity-changing chunk metadata.

EmbeddingSemanticChunker(embeddings=embedder) is a separate asynchronous algorithm: buffered sentence vectors select cosine-distance breakpoints via percentile, standard-deviation, interquartile or gradient thresholds, or a requested chunk count. Its sentence regex, buffer, minimum character size, start-index and batching options map to that algorithm. It preserves original text but supplies neither structural hard-token limits nor protected code/table boundaries. Configure the injected SDK Embedder for source transmission and cost. Neither this class nor structural refinement renames the existing SemanticChunker(llm=...), whose legacy chat-grouping targets use characters.

Migration and limitations#

Existing Chunker.split subclasses, ChunkerRouter routes, explicit caches and lazy Pipeline events remain usable. Structural chunks have deterministic IDs including content, lineage, configuration and embedding context; changing engine, normalizer, tokenizer or refiners invalidates their cache identities. Chunk indices are contiguous in source order per parent on fresh and cached built-in output; concurrent parents retain completion ordering. Diagnostics do not alter IDs. Keep complete metadata on transport; the default Milvus store rejects metadata exceeding its supported size rather than silently discarding maps.

Normalization retains original syntax/provenance, not a rendered browser DOM or lossless reconstruction of original binary documents. Malformed markup can have conservative locations and diagnostics. RST includes/raw execution, HTML scripts and style content are intentionally excluded. Custom adapters and pathological large custom-tokenizer inputs can need application-specific limits. Optional semantic suggestions do not establish retrieval quality: compare configurations on your own labelled corpus with Benchmarking.