fastrag.ingestion.steps.parsing.docusaurus#

Docusaurus documentation parsing: the article, not the site chrome.

A Docusaurus site (docs.pontus-x.eu, agrospai.udl.cat/*/docs/*) wraps its actual article in a container div carrying a markdown class, surrounded by navigation, sidebars, breadcrumbs, footers, and “edit this page” links that HtmlParser has no way to distinguish from content. Feeding the whole page to it buries the article in boilerplate that then gets chunked and embedded — noise a RAG index does not want. This parser extracts only the first such div and converts that subtree; a page carrying no such container is parsed whole, the way HtmlParser does, with a metadata marker recording the fallback.

Module Attributes

DEFAULT_DOCUSAURUS_SELECTORS

Article headers contain the page heading; permalink anchors are UI chrome.

FALLBACK_METADATA_KEY

Metadata key set on every produced document, True when the page carried no markdown-classed div and this parser fell back to parsing the whole document instead — the marker a caller reads to tell the two cases apart.

docusaurus_filter

The filter a ParserRouter routes Docusaurus pages with. It inspects content, so it is not header_safe::.

Classes

DocusaurusParser(*[, strip_selectors, ...])

Extract a Docusaurus page's article, discarding the surrounding chrome.
fastrag.ingestion.steps.parsing.docusaurus.DEFAULT_DOCUSAURUS_SELECTORS = ('script', 'style', 'noscript', 'nav', 'footer', 'aside', 'a.hash-link')#

Article headers contain the page heading; permalink anchors are UI chrome.

fastrag.ingestion.steps.parsing.docusaurus.FALLBACK_METADATA_KEY = 'docusaurus_fallback'#

Metadata key set on every produced document, True when the page carried no markdown-classed div and this parser fell back to parsing the whole document instead — the marker a caller reads to tell the two cases apart.

fastrag.ingestion.steps.parsing.docusaurus.docusaurus_filter: Filter[Source | Document] = <fastrag.ingestion.steps.parsing.docusaurus._DocusaurusFilter object>#

The filter a ParserRouter routes Docusaurus pages with. It inspects content, so it is not header_safe:

parser = ParserRouter(cache=cache).route(
    html_filter & docusaurus_filter, DocusaurusParser()
)
Parameters:

item (T_contra)

Return type:

bool

class fastrag.ingestion.steps.parsing.docusaurus.DocusaurusParser(*, strip_selectors=('script', 'style', 'noscript', 'nav', 'footer', 'aside', 'a.hash-link'), encoding=None, vision=None, cache=None, policy=None)[source]#

Bases: Parser

Extract a Docusaurus page’s article, discarding the surrounding chrome.

Finds the first div whose class list includes markdown (Docusaurus themes typically render it as class="theme-doc-markdown markdown") and converts only that subtree to Markdown, using the same HtmlParser machinery (strip selectors, link absolutization, HTML-to-Markdown conversion) — just scoped to the article rather than the whole document. A page with no such div is parsed whole, exactly as HtmlParser would, and the produced document’s metadata records that the fallback happened.

from fastrag.ingestion.steps.parsing import DocusaurusParser, ParserRouter

parser = ParserRouter(cache=cache).route(docs_filter, DocusaurusParser())
Parameters:
content_format: ContentFormat = 'markdown'#

The ContentFormat this parser’s documents carry by default, stamped by build_document() into metadata["content_format"]. Override on a subclass whose output is always one format (HtmlParser/DocusaurusParser set MARKDOWN); pass content_format= to a single build_document() call instead for a parser whose output format depends on its own configuration.

accepted_media_types: frozenset[str] | None = frozenset({'application/xhtml+xml', 'text/html'})#

None retains legacy permissiveness; MIME patterns may use type/*.

config()[source]#

This parser’s cache-identity configuration.

strip_selectors/encoding change what a given page’s bytes parse into, so both are part of the fingerprint. Which div is matched is not configurable, so it is not part of it.

Return type:

Mapping[str, Any]

async transform(item, context)[source]#

Yield one Markdown document per page, extracted or whole.

Parameters:
Return type:

AsyncIterator[Document]

build_document(source, content, metadata=None, *, content_format=None)#

Build one document from source’s parsed content.

Sets id (a hash of the source identity and parsed content), source_id, step_config_hash (this parser’s own config_hash), and the reserved metadata keys every parser gets right without writing them itself: source_uri, content_type (source’s own, as declared or guessed upstream), and content_format (what kind of string content is — content_format unless overridden for this one call). The site_name/favicon_url a fetcher found on source are carried over too, unless metadata sets its own.

Parameters:

  • source (Source) – The source content was parsed from.

  • content (str) – The parsed text.

  • metadata (dict[str, Any] | None) – Extra metadata to carry, e.g. an HTML page’s title.

  • content_format (ContentFormat | None) – Overrides content_format for this one document, for a parser whose output format depends on its own configuration rather than being fixed per class.

Return type:

Document

property cache: CacheStore | Literal[False] | None#

Return the cache explicitly configured for this step.

property config_hash: str#

Invalidate cached documents created before source-aware identities.

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:

context (ExecutionContext)

Return type:

Policy

async evaluate(context)#

Emit DocumentParsed per document.

Wraps DocumentStep’s concrete evaluate (which already resolved, per item, whether the document came from cache or a fresh transform() call — document.cached is trustworthy either way) rather than emitting inside a concrete parser’s own transform(), so the event fires the same way for a cache hit as for a fresh parse, and exactly once regardless of how many layers of routing (e.g. ParserRouter) produced the document.

Parameters:

context (ExecutionContext)

Return type:

AsyncIterator[Document]

async evaluate_items(items, context)#

Transform a bounded cache window independently of worker concurrency.

Parameters:
Return type:

AsyncIterator[Document]

event_name: str = 'Parsing'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

property policy: Policy | None#

Return the policy explicitly configured for this step.

replay_needs_content: bool = False#

Set by a step whose cached replay reads the full documents anyway (e.g. to re-validate them): its cache reads then skip the header-only projection rather than fetching every entry twice.

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:

list[T]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

validate_input(item)#

Validate declared source capabilities before work and cache replay.

Parameters:

item (Source)

Return type:

None

name: str#