fastrag.normalization#

Extensible source-preserving normalization separate from ingestion parsers.

Classes

`NormalizerRegistry`(*[, builtins])	Instance-local format routing with lazily constructed built-in adapters.
`TextNormalizer`()	Conservative whole-source text block without MIME sniffing.

class fastrag.normalization.CoverageUnit(id, node_id, span)[source]#

Bases: object

A nonoverlapping retained source interval owned by a structural node.

Parameters:

id (str)
node_id (str)
span (SourceSpan)

class fastrag.normalization.Diagnostic(code, message, span=None)[source]#

Bases: object

Stable normalization diagnostic; no timing or secret values.

Parameters:

code (str)
message (str)
span (SourceSpan | None)

class fastrag.normalization.Exclusion(span, reason)[source]#

Bases: object

Explicitly omitted source with the policy reason.

Parameters:

span (SourceSpan)
reason (str)

class fastrag.normalization.HtmlNormalizer(*, exclude_tags=('script', 'style'), exclude_hidden=False)[source]#

Bases: object

Recover HTML while retaining tabs and explicitly recording exclusions.

Parameters:

exclude_tags (tuple[str, ...])
exclude_hidden (bool)

property identity: str#: Return output policy and parser contract version.

normalize(document)[source]#

Parse original source positions without rendering or executing HTML.

Parameters:: document (Document)
Return type:: NormalizedDocument

class fastrag.normalization.MarkdownNormalizer(*, tables=True)[source]#

Bases: object

Parse Markdown using markdown-it-py without rendering or execution.

Parameters:: tables (bool)

property identity: str#: Return parser version and dialect options.

normalize(document)[source]#

Build nested nodes from the parser token stream and original lines.

Parameters:: document (Document)
Return type:: NormalizedDocument

class fastrag.normalization.NormalizedDocument(document, source_format, roots, units, exclusions=(), diagnostics=())[source]#

Bases: object

Tree and independent exact source coverage partition.

Parameters:

document (Document)
source_format (str)
roots (tuple[NormalizedNode, ...])
units (tuple[CoverageUnit, ...])
exclusions (tuple[Exclusion, ...])
diagnostics (tuple[Diagnostic, ...])

property source: str#: Return the unchanged original string.

walk()[source]#

Yield all nodes in preorder.

Return type:: Iterator[NormalizedNode]

validate()[source]#

Check lineage, source slices and complete nonduplicated coverage.

Return type:: None

class fastrag.normalization.NormalizedNode(id, kind, content, spans, children=(), parent_id=None, level=None, title=None, metadata=<factory>)[source]#

Bases: object

Ordered structure; content is original syntax, not rewritten prose.

Parameters:

id (str)
kind (str)
content (str)
spans (tuple[SourceSpan, ...])
children (tuple[NormalizedNode, ...])
parent_id (str | None)
level (int | None)
title (str | None)
metadata (dict[str, Any])

walk()[source]#

Yield this node and descendants in source order.

Return type:: Iterator[NormalizedNode]

class fastrag.normalization.Normalizer(*args, **kwargs)[source]#

Bases: Protocol

Injectable format adapter with stable nonsecret cache identity.

property identity: str#: Describe every output-changing adapter option.

normalize(document)[source]#

Normalize a document without executing its markup.

Parameters:: document (Document)
Return type:: NormalizedDocument

class fastrag.normalization.NormalizerRegistry(*, builtins=True)[source]#

Bases: object

Instance-local format routing with lazily constructed built-in adapters.

Parameters:: builtins (bool)

register(content_format, adapter, *, replace=False)[source]#

Register a custom adapter; duplicates require explicit replacement.

Parameters:

content_format (str)
adapter (Normalizer)
replace (bool)

Return type:

None

property formats: frozenset[str]#: Registered representations, without loading optional dependencies.

adapter(content_format)[source]#

Resolve one adapter, with actionable dependency/unknown-format errors.

Parameters:: content_format (str)
Return type:: Normalizer

property identity: str#: Identify every route without requiring unselected optional dependencies.

normalize(document)[source]#

Route by explicit output representation, defaulting legacy docs to text.

Parameters:: document (Document)
Return type:: NormalizedDocument

class fastrag.normalization.RstNormalizer[source]#

Bases: object

Use the RST parser directly with local/remote file insertion disabled.

property identity: str#: Identify parser and security policy.

normalize(document)[source]#

Map Docutils block tree and conservative source locations.

Parameters:: document (Document)
Return type:: NormalizedDocument

class fastrag.normalization.SourceSpan(start_offset, end_offset)[source]#

Bases: object

Half-open Unicode offsets into the original Document content.

Parameters:

start_offset (int)
end_offset (int)

class fastrag.normalization.TextNormalizer[source]#

Bases: object

Conservative whole-source text block without MIME sniffing.

identity = 'text:original:v1'#

normalize(document)[source]#

Retain the entire text unchanged.

Parameters:: document (Document)
Return type:: NormalizedDocument

Modules

`html`	HTML structural normalization with precise source policy exclusions.
`markdown`	CommonMark with pipe tables, exact block source, and inline metadata.
`models`	Source-preserving structural models independent of ingestion and packing.
`rst`	Safe Docutils structural adapter retaining unsupported directive source.