fastrag.ingestion.steps.parsing.html#

HTML to Markdown with hidden panels, SVG labels and optional vision descriptions.

Module Attributes

DEFAULT_STRIP_SELECTORS

Repeated page furniture removed before content extraction.

HTML_MEDIA_TYPES

The media types this parser understands.

html_filter

The filter a ParserRouter routes HTML sources with.

Classes

HtmlParser(*[, strip_selectors, encoding, ...])

Convert HTML to Markdown, dropping the page furniture first.
fastrag.ingestion.steps.parsing.html.DEFAULT_STRIP_SELECTORS: tuple[str, ...] = ('script', 'style', 'noscript', 'nav', 'header', 'footer', 'aside')#

Repeated page furniture removed before content extraction. Meaningful SVGs and images are handled separately by the shared content preservation code.

fastrag.ingestion.steps.parsing.html.HTML_MEDIA_TYPES: frozenset[str] = frozenset({'application/xhtml+xml', 'text/html'})#

The media types this parser understands.

fastrag.ingestion.steps.parsing.html.html_filter = <fastrag.ingestion.filters.mediatype.MediaTypeFilter object>#

The filter a ParserRouter routes HTML sources with:

parser = ParserRouter(cache=cache).route(html_filter, HtmlParser())
Parameters:

item (T_contra)

Return type:

bool

class fastrag.ingestion.steps.parsing.html.HtmlParser(*, strip_selectors=('script', 'style', 'noscript', 'nav', 'header', 'footer', 'aside'), encoding=None, vision=None, cache=None, policy=None)[source]#

Bases: Parser

Convert HTML to Markdown, dropping the page furniture first.

Markdown rather than stripped text, because the heading levels, lists, and tables it preserves are structure worth keeping. Relative links are resolved against the source’s URI, so a chunk quoted back to a user carries links that still work.

from fastrag.ingestion.steps.parsing import HtmlParser

parser = HtmlParser(strip_selectors=("script", "style", ".cookie-banner"))
Parameters:
content_format: ContentFormat = 'markdown'#

The ContentFormat this parser’s documents carry by default, stamped by build_document() into metadata["content_format"]. Override on a subclass whose output is always one format (HtmlParser/DocusaurusParser set MARKDOWN); pass content_format= to a single build_document() call instead for a parser whose output format depends on its own configuration.

accepted_media_types: frozenset[str] | None = frozenset({'application/xhtml+xml', 'text/html'})#

None retains legacy permissiveness; MIME patterns may use type/*.

config()[source]#

This parser’s cache-identity configuration.

strip_selectors/encoding change what a given page’s bytes parse into, so both are part of the fingerprint.

Return type:

Mapping[str, Any]

async transform(item, context)[source]#

Yield one Markdown document per HTML page, or nothing if it is blank.

Parameters:
Return type:

AsyncIterator[Document]

build_document(source, content, metadata=None, *, content_format=None)#

Build one document from source’s parsed content.

Sets id (a hash of the source identity and parsed content), source_id, step_config_hash (this parser’s own config_hash), and the reserved metadata keys every parser gets right without writing them itself: source_uri, content_type (source’s own, as declared or guessed upstream), and content_format (what kind of string content is — content_format unless overridden for this one call). The site_name/favicon_url a fetcher found on source are carried over too, unless metadata sets its own.

Parameters:

  • source (Source) – The source content was parsed from.

  • content (str) – The parsed text.

  • metadata (dict[str, Any] | None) – Extra metadata to carry, e.g. an HTML page’s title.

  • content_format (ContentFormat | None) – Overrides content_format for this one document, for a parser whose output format depends on its own configuration rather than being fixed per class.

Return type:

Document

property cache: CacheStore | Literal[False] | None#

Return the cache explicitly configured for this step.

property config_hash: str#

Invalidate cached documents created before source-aware identities.

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:

context (ExecutionContext)

Return type:

Policy

async evaluate(context)#

Emit DocumentParsed per document.

Wraps DocumentStep’s concrete evaluate (which already resolved, per item, whether the document came from cache or a fresh transform() call — document.cached is trustworthy either way) rather than emitting inside a concrete parser’s own transform(), so the event fires the same way for a cache hit as for a fresh parse, and exactly once regardless of how many layers of routing (e.g. ParserRouter) produced the document.

Parameters:

context (ExecutionContext)

Return type:

AsyncIterator[Document]

async evaluate_items(items, context)#

Transform a bounded cache window independently of worker concurrency.

Parameters:
Return type:

AsyncIterator[Document]

event_name: str = 'Parsing'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

property policy: Policy | None#

Return the policy explicitly configured for this step.

replay_needs_content: bool = False#

Set by a step whose cached replay reads the full documents anyway (e.g. to re-validate them): its cache reads then skip the header-only projection rather than fetching every entry twice.

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:

list[T]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

validate_input(item)#

Validate declared source capabilities before work and cache replay.

Parameters:

item (Source)

Return type:

None

name: str#