fastrag.ingestion.steps.parsing#

Source parsing: raw bytes in, documents out.

HTML parsers preserve hidden content and SVG labels by default. JavaScript rendering and vision descriptions are optional. ParserRouter dispatches each source to a parser by filter, e.g. html_filter to HtmlParser.

class fastrag.ingestion.steps.parsing.AudioParser(*, model, api_key, base_url, language=None, prompt=None, timeout=600, extra_body=None, cache=None, policy=None)[source]#

Bases: Parser

Transcribe audio sources to text with an OpenAI-compatible Whisper endpoint.

base_url is the API root (e.g. https://host/v1); the request goes to {base_url}/audio/transcriptions as multipart form data. Credentials are excluded from cache identity; model, endpoint, language, prompt and extra form fields are included.

Parameters:

  • model (str)

  • api_key (str)

  • base_url (str)

  • language (str | None)

  • prompt (str | None)

  • timeout (float)

  • extra_body (Mapping[str, Any] | None)

  • cache (CacheStore | Literal[False] | None)

  • policy (Policy | None)

accepted_media_types: frozenset[str] | None = frozenset({'audio/*', 'video/mp4', 'video/webm'})#

None retains legacy permissiveness; MIME patterns may use type/*.

config()[source]#

Return transcription-affecting settings without credentials.

Return type:

Mapping[str, Any]

async transcribe(content, filename, content_type)[source]#

POST audio bytes to the transcription endpoint and return the text.

Parameters:
Return type:

str

async transform(item, context)[source]#

Yield one transcript document for an audio source.

Parameters:
Return type:

AsyncIterator[Document]

build_document(source, content, metadata=None, *, content_format=None)#

Build one document from source’s parsed content.

Sets id (a hash of the source identity and parsed content), source_id, step_config_hash (this parser’s own config_hash), and the reserved metadata keys every parser gets right without writing them itself: source_uri, content_type (source’s own, as declared or guessed upstream), and content_format (what kind of string content is — content_format unless overridden for this one call). The site_name/favicon_url a fetcher found on source are carried over too, unless metadata sets its own.

Parameters:

  • source (Source) – The source content was parsed from.

  • content (str) – The parsed text.

  • metadata (dict[str, Any] | None) – Extra metadata to carry, e.g. an HTML page’s title.

  • content_format (ContentFormat | None) – Overrides content_format for this one document, for a parser whose output format depends on its own configuration rather than being fixed per class.

Return type:

Document

property cache: CacheStore | Literal[False] | None#

Return the cache explicitly configured for this step.

property config_hash: str#

Invalidate cached documents created before source-aware identities.

content_format: ContentFormat = 'text'#

The ContentFormat this parser’s documents carry by default, stamped by build_document() into metadata["content_format"]. Override on a subclass whose output is always one format (HtmlParser/DocusaurusParser set MARKDOWN); pass content_format= to a single build_document() call instead for a parser whose output format depends on its own configuration.

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:

context (ExecutionContext)

Return type:

Policy

async evaluate(context)#

Emit DocumentParsed per document.

Wraps DocumentStep’s concrete evaluate (which already resolved, per item, whether the document came from cache or a fresh transform() call — document.cached is trustworthy either way) rather than emitting inside a concrete parser’s own transform(), so the event fires the same way for a cache hit as for a fresh parse, and exactly once regardless of how many layers of routing (e.g. ParserRouter) produced the document.

Parameters:

context (ExecutionContext)

Return type:

AsyncIterator[Document]

async evaluate_items(items, context)#

Transform a bounded cache window independently of worker concurrency.

Parameters:
Return type:

AsyncIterator[Document]

event_name: str = 'Parsing'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

property policy: Policy | None#

Return the policy explicitly configured for this step.

replay_needs_content: bool = False#

Set by a step whose cached replay reads the full documents anyway (e.g. to re-validate them): its cache reads then skip the header-only projection rather than fetching every entry twice.

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:

list[T]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

validate_input(item)#

Validate declared source capabilities before work and cache replay.

Parameters:

item (Source)

Return type:

None

name: str#
class fastrag.ingestion.steps.parsing.ChandraParser(*, api_key, base_url='https://chandra.agrospai.udl.cat', output_format='markdown', timeout=900, pages=None, cache=None, policy=None)[source]#

Bases: PDFParser

OCR PDFs with Chandra; one Markdown (or HTML) document per page.

parser = ChandraParser(api_key=CHANDRA_API_KEY, base_url=CHANDRA_URL)
parser = ChandraParser(..., output_format="html")
Parameters:

  • api_key (str)

  • base_url (str)

  • output_format (OutputFormat)

  • timeout (float)

  • pages (Pages | None)

  • cache (CacheStore | Literal[False] | None)

  • policy (Policy | None)

content_format: ContentFormat = 'text'#

The ContentFormat this parser’s documents carry by default, stamped by build_document() into metadata["content_format"]. Override on a subclass whose output is always one format (HtmlParser/DocusaurusParser set MARKDOWN); pass content_format= to a single build_document() call instead for a parser whose output format depends on its own configuration.

config()[source]#

Parse-affecting settings, without credentials.

Return type:

Mapping[str, Any]

async parse_pdf(content, filename, pages)[source]#

OCR the PDF and map Chandra’s per-page results.

Parameters:
Return type:

PDFParseResult

accepted_media_types: frozenset[str] | None = frozenset({'application/pdf'})#

None retains legacy permissiveness; MIME patterns may use type/*.

build_document(source, content, metadata=None, *, content_format=None)#

Build one document from source’s parsed content.

Sets id (a hash of the source identity and parsed content), source_id, step_config_hash (this parser’s own config_hash), and the reserved metadata keys every parser gets right without writing them itself: source_uri, content_type (source’s own, as declared or guessed upstream), and content_format (what kind of string content is — content_format unless overridden for this one call). The site_name/favicon_url a fetcher found on source are carried over too, unless metadata sets its own.

Parameters:

  • source (Source) – The source content was parsed from.

  • content (str) – The parsed text.

  • metadata (dict[str, Any] | None) – Extra metadata to carry, e.g. an HTML page’s title.

  • content_format (ContentFormat | None) – Overrides content_format for this one document, for a parser whose output format depends on its own configuration rather than being fixed per class.

Return type:

Document

property cache: CacheStore | Literal[False] | None#

Return the cache explicitly configured for this step.

property config_hash: str#

Invalidate cached documents created before source-aware identities.

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:

context (ExecutionContext)

Return type:

Policy

async evaluate(context)#

Emit DocumentParsed per document.

Wraps DocumentStep’s concrete evaluate (which already resolved, per item, whether the document came from cache or a fresh transform() call — document.cached is trustworthy either way) rather than emitting inside a concrete parser’s own transform(), so the event fires the same way for a cache hit as for a fresh parse, and exactly once regardless of how many layers of routing (e.g. ParserRouter) produced the document.

Parameters:

context (ExecutionContext)

Return type:

AsyncIterator[Document]

async evaluate_items(items, context)#

Transform a bounded cache window independently of worker concurrency.

Parameters:
Return type:

AsyncIterator[Document]

event_name: str = 'Parsing'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

property policy: Policy | None#

Return the policy explicitly configured for this step.

replay_needs_content: bool = False#

Set by a step whose cached replay reads the full documents anyway (e.g. to re-validate them): its cache reads then skip the header-only projection rather than fetching every entry twice.

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:

list[T]

async transform(item, context)#

Yield one document per non-empty PDF page, in page order.

Parameters:
Return type:

AsyncIterator[Document]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

validate_input(item)#

Validate declared source capabilities before work and cache replay.

Parameters:

item (Source)

Return type:

None

name: str#
class fastrag.ingestion.steps.parsing.DecisionsParser(*, api_format='decisions', **kwargs)[source]#

Bases: Parser

Select useful source content with typed Decisions/JEV choices.

api_format picks the wire format: "decisions" (OpenAI Decisions, /decisions) or "systemone" (TypeSafe System One, /systemone).

Parameters:

  • api_format (Literal['decisions', 'systemone'])

  • kwargs (Any)

protocol: str = 'decisions'#
api_format: Literal['decisions', 'systemone'] = 'decisions'#
accepted_media_types: frozenset[str] | None = frozenset({'application/xhtml+xml', 'text/html'})#

None retains legacy permissiveness; MIME patterns may use type/*.

build_document(source, content, metadata=None, *, content_format=None)#

Build one document from source’s parsed content.

Sets id (a hash of the source identity and parsed content), source_id, step_config_hash (this parser’s own config_hash), and the reserved metadata keys every parser gets right without writing them itself: source_uri, content_type (source’s own, as declared or guessed upstream), and content_format (what kind of string content is — content_format unless overridden for this one call). The site_name/favicon_url a fetcher found on source are carried over too, unless metadata sets its own.

Parameters:

  • source (Source) – The source content was parsed from.

  • content (str) – The parsed text.

  • metadata (dict[str, Any] | None) – Extra metadata to carry, e.g. an HTML page’s title.

  • content_format (ContentFormat | None) – Overrides content_format for this one document, for a parser whose output format depends on its own configuration rather than being fixed per class.

Return type:

Document

property cache: CacheStore | Literal[False] | None#

Return the cache explicitly configured for this step.

config()#

Return content-selection settings without credentials.

Return type:

Mapping[str, Any]

property config_hash: str#

Invalidate cached documents created before source-aware identities.

content_format: ContentFormat = 'markdown'#

The ContentFormat this parser’s documents carry by default, stamped by build_document() into metadata["content_format"]. Override on a subclass whose output is always one format (HtmlParser/DocusaurusParser set MARKDOWN); pass content_format= to a single build_document() call instead for a parser whose output format depends on its own configuration.

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:

context (ExecutionContext)

Return type:

Policy

async evaluate(context)#

Emit DocumentParsed per document.

Wraps DocumentStep’s concrete evaluate (which already resolved, per item, whether the document came from cache or a fresh transform() call — document.cached is trustworthy either way) rather than emitting inside a concrete parser’s own transform(), so the event fires the same way for a cache hit as for a fresh parse, and exactly once regardless of how many layers of routing (e.g. ParserRouter) produced the document.

Parameters:

context (ExecutionContext)

Return type:

AsyncIterator[Document]

async evaluate_items(items, context)#

Transform a bounded cache window independently of worker concurrency.

Parameters:
Return type:

AsyncIterator[Document]

event_name: str = 'Parsing'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

property policy: Policy | None#

Return the policy explicitly configured for this step.

replay_needs_content: bool = False#

Set by a step whose cached replay reads the full documents anyway (e.g. to re-validate them): its cache reads then skip the header-only projection rather than fetching every entry twice.

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:

list[T]

async transform(item, context)#

Yield useful Markdown from supplied source bytes without browser or page requests.

Parameters:
Return type:

AsyncIterator[Document]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

validate_input(item)#

Validate declared source capabilities before work and cache replay.

Parameters:

item (Source)

Return type:

None

name: str#
base_url: str#
extra_body: dict[str, Any]#
model: str#
api_key: str#
class fastrag.ingestion.steps.parsing.OpenAIParser(*, model, api_key, base_url, goal='Extract all useful page content for a searchable knowledge base.', timeout=600, max_snapshot_chars=24000, request_timeout=120, extra_body=None, fallback=None, cache=None, policy=None)[source]#

Bases: Parser

Select useful source content with an OpenAI-compatible model.

Parameters:
accepted_media_types: frozenset[str] | None = frozenset({'application/xhtml+xml', 'text/html'})#

None retains legacy permissiveness; MIME patterns may use type/*.

api_format: Literal['decisions', 'systemone'] = 'decisions'#
build_document(source, content, metadata=None, *, content_format=None)#

Build one document from source’s parsed content.

Sets id (a hash of the source identity and parsed content), source_id, step_config_hash (this parser’s own config_hash), and the reserved metadata keys every parser gets right without writing them itself: source_uri, content_type (source’s own, as declared or guessed upstream), and content_format (what kind of string content is — content_format unless overridden for this one call). The site_name/favicon_url a fetcher found on source are carried over too, unless metadata sets its own.

Parameters:

  • source (Source) – The source content was parsed from.

  • content (str) – The parsed text.

  • metadata (dict[str, Any] | None) – Extra metadata to carry, e.g. an HTML page’s title.

  • content_format (ContentFormat | None) – Overrides content_format for this one document, for a parser whose output format depends on its own configuration rather than being fixed per class.

Return type:

Document

property cache: CacheStore | Literal[False] | None#

Return the cache explicitly configured for this step.

config()#

Return content-selection settings without credentials.

Return type:

Mapping[str, Any]

property config_hash: str#

Invalidate cached documents created before source-aware identities.

content_format: ContentFormat = 'markdown'#

The ContentFormat this parser’s documents carry by default, stamped by build_document() into metadata["content_format"]. Override on a subclass whose output is always one format (HtmlParser/DocusaurusParser set MARKDOWN); pass content_format= to a single build_document() call instead for a parser whose output format depends on its own configuration.

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:

context (ExecutionContext)

Return type:

Policy

async evaluate(context)#

Emit DocumentParsed per document.

Wraps DocumentStep’s concrete evaluate (which already resolved, per item, whether the document came from cache or a fresh transform() call — document.cached is trustworthy either way) rather than emitting inside a concrete parser’s own transform(), so the event fires the same way for a cache hit as for a fresh parse, and exactly once regardless of how many layers of routing (e.g. ParserRouter) produced the document.

Parameters:

context (ExecutionContext)

Return type:

AsyncIterator[Document]

async evaluate_items(items, context)#

Transform a bounded cache window independently of worker concurrency.

Parameters:
Return type:

AsyncIterator[Document]

event_name: str = 'Parsing'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

property policy: Policy | None#

Return the policy explicitly configured for this step.

protocol: str = 'openai'#
replay_needs_content: bool = False#

Set by a step whose cached replay reads the full documents anyway (e.g. to re-validate them): its cache reads then skip the header-only projection rather than fetching every entry twice.

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:

list[T]

async transform(item, context)#

Yield useful Markdown from supplied source bytes without browser or page requests.

Parameters:
Return type:

AsyncIterator[Document]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

validate_input(item)#

Validate declared source capabilities before work and cache replay.

Parameters:

item (Source)

Return type:

None

name: str#
base_url: str#
extra_body: dict[str, Any]#
model: str#
api_key: str#
class fastrag.ingestion.steps.parsing.DefaultParser(*, encoding='utf-8', cache=None, policy=None)[source]#

Bases: Parser

Decode the bytes as text, without any parsing.

The text-only fallback parser ParserRouter falls back to for text that has no more specific route.

from fastrag.ingestion.steps.parsing import DefaultParser

parser = DefaultParser(encoding="latin-1")
Parameters:
content_format: ContentFormat = 'text'#

The ContentFormat this parser’s documents carry by default, stamped by build_document() into metadata["content_format"]. Override on a subclass whose output is always one format (HtmlParser/DocusaurusParser set MARKDOWN); pass content_format= to a single build_document() call instead for a parser whose output format depends on its own configuration.

accepted_media_types: frozenset[str] | None = frozenset({'application/javascript', 'application/json', 'application/octet-stream', 'application/x-yaml', 'application/xhtml+xml', 'application/xml', 'application/yaml', 'text/*'})#

None retains legacy permissiveness; MIME patterns may use type/*.

config()[source]#

This parser’s cache-identity configuration.

encoding is part of the fingerprint: decoding the same bytes differently is a different parse.

Return type:

Mapping[str, Any]

async transform(item, context)[source]#

Yield one document holding the decoded text, or nothing if blank.

Parameters:
Return type:

AsyncIterator[Document]

build_document(source, content, metadata=None, *, content_format=None)#

Build one document from source’s parsed content.

Sets id (a hash of the source identity and parsed content), source_id, step_config_hash (this parser’s own config_hash), and the reserved metadata keys every parser gets right without writing them itself: source_uri, content_type (source’s own, as declared or guessed upstream), and content_format (what kind of string content is — content_format unless overridden for this one call). The site_name/favicon_url a fetcher found on source are carried over too, unless metadata sets its own.

Parameters:

  • source (Source) – The source content was parsed from.

  • content (str) – The parsed text.

  • metadata (dict[str, Any] | None) – Extra metadata to carry, e.g. an HTML page’s title.

  • content_format (ContentFormat | None) – Overrides content_format for this one document, for a parser whose output format depends on its own configuration rather than being fixed per class.

Return type:

Document

property cache: CacheStore | Literal[False] | None#

Return the cache explicitly configured for this step.

property config_hash: str#

Invalidate cached documents created before source-aware identities.

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:

context (ExecutionContext)

Return type:

Policy

async evaluate(context)#

Emit DocumentParsed per document.

Wraps DocumentStep’s concrete evaluate (which already resolved, per item, whether the document came from cache or a fresh transform() call — document.cached is trustworthy either way) rather than emitting inside a concrete parser’s own transform(), so the event fires the same way for a cache hit as for a fresh parse, and exactly once regardless of how many layers of routing (e.g. ParserRouter) produced the document.

Parameters:

context (ExecutionContext)

Return type:

AsyncIterator[Document]

async evaluate_items(items, context)#

Transform a bounded cache window independently of worker concurrency.

Parameters:
Return type:

AsyncIterator[Document]

event_name: str = 'Parsing'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

property policy: Policy | None#

Return the policy explicitly configured for this step.

replay_needs_content: bool = False#

Set by a step whose cached replay reads the full documents anyway (e.g. to re-validate them): its cache reads then skip the header-only projection rather than fetching every entry twice.

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:

list[T]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

validate_input(item)#

Validate declared source capabilities before work and cache replay.

Parameters:

item (Source)

Return type:

None

name: str#
class fastrag.ingestion.steps.parsing.DoclingParser(*, output_format='markdown', describe_images=False, picture_description_options=None, allowed_formats=None, format_options=None, cache=None, policy=None)[source]#

Bases: Parser

Convert a binary document to Markdown or a serialized DoclingDocument.

Hands a Source’s bytes to Docling’s DocumentConverter, entirely from memory (never a temporary file): Docling infers the source format from a filename derived from the source’s URI, not from anything written to disk.

from fastrag.ingestion.steps.parsing import DoclingParser

# Markdown, chunked by whatever chunker the caller already uses.
parser = DoclingParser()

# The structured document itself, for a Docling-native chunker.
parser = DoclingParser(output_format="docling")

# Describe the pictures found in PDFs (runs a vision model).
parser = DoclingParser(describe_images=True)

In "markdown" mode, a paginated document (PDF, slides) is organized by page: one Document per non-empty page, each carrying metadata["page_no"] (1-based), metadata["page_count"] and metadata["source_url"]. For PDFs source_url is the source URI with a #page=N fragment, so every chunk inherits a link to the exact page it came from. Documents without pages (.docx, HTML, …) are yielded whole, without page_no.

Conversion is synchronous and CPU-bound, so it always runs through asyncio.to_thread rather than blocking the event loop it is awaited from.

Parameters:

  • output_format (OutputFormat)

  • describe_images (bool)

  • picture_description_options (Any | None)

  • allowed_formats (Sequence[InputFormat] | None)

  • format_options (Mapping[InputFormat, Any] | None)

  • cache (CacheStore | Literal[False] | None)

  • policy (Policy | None)

accepted_media_types: frozenset[str] | None = frozenset({'application/msword', 'application/pdf', 'application/vnd.ms-excel', 'application/vnd.ms-powerpoint', 'application/vnd.openxmlformats-officedocument.presentationml.presentation', 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', 'application/vnd.openxmlformats-officedocument.wordprocessingml.document', 'application/xhtml+xml', 'image/*', 'text/*'})#

None retains legacy permissiveness; MIME patterns may use type/*.

config()[source]#

This parser’s cache-identity configuration.

output_format changes the shape of what a given file parses into — Markdown text versus a serialized DoclingDocument — so it is part of the fingerprint just as much as allowed_formats (which formats Docling will even attempt) is: switching either invalidates the cache instead of serving a caller who asked for one shape the other shape cached earlier. format_options belongs for the same reason: it changes what Docling extracts from a given file (picture descriptions, say), so it must not serve a caller the pre-enrichment text cached earlier.

Return type:

Mapping[str, Any]

async transform(item, context)[source]#

Yield one document per non-empty page, or nothing if there is no text.

Parameters:
Return type:

AsyncIterator[Document]

build_document(source, content, metadata=None, *, content_format=None)#

Build one document from source’s parsed content.

Sets id (a hash of the source identity and parsed content), source_id, step_config_hash (this parser’s own config_hash), and the reserved metadata keys every parser gets right without writing them itself: source_uri, content_type (source’s own, as declared or guessed upstream), and content_format (what kind of string content is — content_format unless overridden for this one call). The site_name/favicon_url a fetcher found on source are carried over too, unless metadata sets its own.

Parameters:

  • source (Source) – The source content was parsed from.

  • content (str) – The parsed text.

  • metadata (dict[str, Any] | None) – Extra metadata to carry, e.g. an HTML page’s title.

  • content_format (ContentFormat | None) – Overrides content_format for this one document, for a parser whose output format depends on its own configuration rather than being fixed per class.

Return type:

Document

property cache: CacheStore | Literal[False] | None#

Return the cache explicitly configured for this step.

property config_hash: str#

Invalidate cached documents created before source-aware identities.

content_format: ContentFormat = 'text'#

The ContentFormat this parser’s documents carry by default, stamped by build_document() into metadata["content_format"]. Override on a subclass whose output is always one format (HtmlParser/DocusaurusParser set MARKDOWN); pass content_format= to a single build_document() call instead for a parser whose output format depends on its own configuration.

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:

context (ExecutionContext)

Return type:

Policy

async evaluate(context)#

Emit DocumentParsed per document.

Wraps DocumentStep’s concrete evaluate (which already resolved, per item, whether the document came from cache or a fresh transform() call — document.cached is trustworthy either way) rather than emitting inside a concrete parser’s own transform(), so the event fires the same way for a cache hit as for a fresh parse, and exactly once regardless of how many layers of routing (e.g. ParserRouter) produced the document.

Parameters:

context (ExecutionContext)

Return type:

AsyncIterator[Document]

async evaluate_items(items, context)#

Transform a bounded cache window independently of worker concurrency.

Parameters:
Return type:

AsyncIterator[Document]

event_name: str = 'Parsing'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

property policy: Policy | None#

Return the policy explicitly configured for this step.

replay_needs_content: bool = False#

Set by a step whose cached replay reads the full documents anyway (e.g. to re-validate them): its cache reads then skip the header-only projection rather than fetching every entry twice.

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:

list[T]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

validate_input(item)#

Validate declared source capabilities before work and cache replay.

Parameters:

item (Source)

Return type:

None

name: str#
class fastrag.ingestion.steps.parsing.DoclingServeParser(*, api_key, base_url='https://docling.agrospai.udl.cat', timeout=600, pages=None, cache=None, policy=None)[source]#

Bases: PDFParser

Parse PDFs to Markdown (one document per page) with docling-serve.

The server is asked to put a page-break marker between pages, and the Markdown is split on it. Non-contiguous pages cost one request per contiguous run.

parser = DoclingServeParser(api_key=DOCLING_API_KEY, base_url=DOCLING_URL)
Parameters:
content_format: ContentFormat = 'markdown'#

The ContentFormat this parser’s documents carry by default, stamped by build_document() into metadata["content_format"]. Override on a subclass whose output is always one format (HtmlParser/DocusaurusParser set MARKDOWN); pass content_format= to a single build_document() call instead for a parser whose output format depends on its own configuration.

config()[source]#

Parse-affecting settings, without credentials.

Return type:

Mapping[str, Any]

async parse_pdf(content, filename, pages)[source]#

Convert the PDF and split the Markdown on page breaks.

Parameters:
Return type:

PDFParseResult

accepted_media_types: frozenset[str] | None = frozenset({'application/pdf'})#

None retains legacy permissiveness; MIME patterns may use type/*.

build_document(source, content, metadata=None, *, content_format=None)#

Build one document from source’s parsed content.

Sets id (a hash of the source identity and parsed content), source_id, step_config_hash (this parser’s own config_hash), and the reserved metadata keys every parser gets right without writing them itself: source_uri, content_type (source’s own, as declared or guessed upstream), and content_format (what kind of string content is — content_format unless overridden for this one call). The site_name/favicon_url a fetcher found on source are carried over too, unless metadata sets its own.

Parameters:

  • source (Source) – The source content was parsed from.

  • content (str) – The parsed text.

  • metadata (dict[str, Any] | None) – Extra metadata to carry, e.g. an HTML page’s title.

  • content_format (ContentFormat | None) – Overrides content_format for this one document, for a parser whose output format depends on its own configuration rather than being fixed per class.

Return type:

Document

property cache: CacheStore | Literal[False] | None#

Return the cache explicitly configured for this step.

property config_hash: str#

Invalidate cached documents created before source-aware identities.

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:

context (ExecutionContext)

Return type:

Policy

async evaluate(context)#

Emit DocumentParsed per document.

Wraps DocumentStep’s concrete evaluate (which already resolved, per item, whether the document came from cache or a fresh transform() call — document.cached is trustworthy either way) rather than emitting inside a concrete parser’s own transform(), so the event fires the same way for a cache hit as for a fresh parse, and exactly once regardless of how many layers of routing (e.g. ParserRouter) produced the document.

Parameters:

context (ExecutionContext)

Return type:

AsyncIterator[Document]

async evaluate_items(items, context)#

Transform a bounded cache window independently of worker concurrency.

Parameters:
Return type:

AsyncIterator[Document]

event_name: str = 'Parsing'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

property policy: Policy | None#

Return the policy explicitly configured for this step.

replay_needs_content: bool = False#

Set by a step whose cached replay reads the full documents anyway (e.g. to re-validate them): its cache reads then skip the header-only projection rather than fetching every entry twice.

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:

list[T]

async transform(item, context)#

Yield one document per non-empty PDF page, in page order.

Parameters:
Return type:

AsyncIterator[Document]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

validate_input(item)#

Validate declared source capabilities before work and cache replay.

Parameters:

item (Source)

Return type:

None

name: str#
class fastrag.ingestion.steps.parsing.DocusaurusParser(*, strip_selectors=('script', 'style', 'noscript', 'nav', 'footer', 'aside', 'a.hash-link'), encoding=None, vision=None, cache=None, policy=None)[source]#

Bases: Parser

Extract a Docusaurus page’s article, discarding the surrounding chrome.

Finds the first div whose class list includes markdown (Docusaurus themes typically render it as class="theme-doc-markdown markdown") and converts only that subtree to Markdown, using the same HtmlParser machinery (strip selectors, link absolutization, HTML-to-Markdown conversion) — just scoped to the article rather than the whole document. A page with no such div is parsed whole, exactly as HtmlParser would, and the produced document’s metadata records that the fallback happened.

from fastrag.ingestion.steps.parsing import DocusaurusParser, ParserRouter

parser = ParserRouter(cache=cache).route(docs_filter, DocusaurusParser())
Parameters:
content_format: ContentFormat = 'markdown'#

The ContentFormat this parser’s documents carry by default, stamped by build_document() into metadata["content_format"]. Override on a subclass whose output is always one format (HtmlParser/DocusaurusParser set MARKDOWN); pass content_format= to a single build_document() call instead for a parser whose output format depends on its own configuration.

accepted_media_types: frozenset[str] | None = frozenset({'application/xhtml+xml', 'text/html'})#

None retains legacy permissiveness; MIME patterns may use type/*.

config()[source]#

This parser’s cache-identity configuration.

strip_selectors/encoding change what a given page’s bytes parse into, so both are part of the fingerprint. Which div is matched is not configurable, so it is not part of it.

Return type:

Mapping[str, Any]

async transform(item, context)[source]#

Yield one Markdown document per page, extracted or whole.

Parameters:
Return type:

AsyncIterator[Document]

build_document(source, content, metadata=None, *, content_format=None)#

Build one document from source’s parsed content.

Sets id (a hash of the source identity and parsed content), source_id, step_config_hash (this parser’s own config_hash), and the reserved metadata keys every parser gets right without writing them itself: source_uri, content_type (source’s own, as declared or guessed upstream), and content_format (what kind of string content is — content_format unless overridden for this one call). The site_name/favicon_url a fetcher found on source are carried over too, unless metadata sets its own.

Parameters:

  • source (Source) – The source content was parsed from.

  • content (str) – The parsed text.

  • metadata (dict[str, Any] | None) – Extra metadata to carry, e.g. an HTML page’s title.

  • content_format (ContentFormat | None) – Overrides content_format for this one document, for a parser whose output format depends on its own configuration rather than being fixed per class.

Return type:

Document

property cache: CacheStore | Literal[False] | None#

Return the cache explicitly configured for this step.

property config_hash: str#

Invalidate cached documents created before source-aware identities.

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:

context (ExecutionContext)

Return type:

Policy

async evaluate(context)#

Emit DocumentParsed per document.

Wraps DocumentStep’s concrete evaluate (which already resolved, per item, whether the document came from cache or a fresh transform() call — document.cached is trustworthy either way) rather than emitting inside a concrete parser’s own transform(), so the event fires the same way for a cache hit as for a fresh parse, and exactly once regardless of how many layers of routing (e.g. ParserRouter) produced the document.

Parameters:

context (ExecutionContext)

Return type:

AsyncIterator[Document]

async evaluate_items(items, context)#

Transform a bounded cache window independently of worker concurrency.

Parameters:
Return type:

AsyncIterator[Document]

event_name: str = 'Parsing'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

property policy: Policy | None#

Return the policy explicitly configured for this step.

replay_needs_content: bool = False#

Set by a step whose cached replay reads the full documents anyway (e.g. to re-validate them): its cache reads then skip the header-only projection rather than fetching every entry twice.

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:

list[T]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

validate_input(item)#

Validate declared source capabilities before work and cache replay.

Parameters:

item (Source)

Return type:

None

name: str#
class fastrag.ingestion.steps.parsing.HtmlParser(*, strip_selectors=('script', 'style', 'noscript', 'nav', 'header', 'footer', 'aside'), encoding=None, vision=None, cache=None, policy=None)[source]#

Bases: Parser

Convert HTML to Markdown, dropping the page furniture first.

Markdown rather than stripped text, because the heading levels, lists, and tables it preserves are structure worth keeping. Relative links are resolved against the source’s URI, so a chunk quoted back to a user carries links that still work.

from fastrag.ingestion.steps.parsing import HtmlParser

parser = HtmlParser(strip_selectors=("script", "style", ".cookie-banner"))
Parameters:
content_format: ContentFormat = 'markdown'#

The ContentFormat this parser’s documents carry by default, stamped by build_document() into metadata["content_format"]. Override on a subclass whose output is always one format (HtmlParser/DocusaurusParser set MARKDOWN); pass content_format= to a single build_document() call instead for a parser whose output format depends on its own configuration.

accepted_media_types: frozenset[str] | None = frozenset({'application/xhtml+xml', 'text/html'})#

None retains legacy permissiveness; MIME patterns may use type/*.

config()[source]#

This parser’s cache-identity configuration.

strip_selectors/encoding change what a given page’s bytes parse into, so both are part of the fingerprint.

Return type:

Mapping[str, Any]

async transform(item, context)[source]#

Yield one Markdown document per HTML page, or nothing if it is blank.

Parameters:
Return type:

AsyncIterator[Document]

build_document(source, content, metadata=None, *, content_format=None)#

Build one document from source’s parsed content.

Sets id (a hash of the source identity and parsed content), source_id, step_config_hash (this parser’s own config_hash), and the reserved metadata keys every parser gets right without writing them itself: source_uri, content_type (source’s own, as declared or guessed upstream), and content_format (what kind of string content is — content_format unless overridden for this one call). The site_name/favicon_url a fetcher found on source are carried over too, unless metadata sets its own.

Parameters:

  • source (Source) – The source content was parsed from.

  • content (str) – The parsed text.

  • metadata (dict[str, Any] | None) – Extra metadata to carry, e.g. an HTML page’s title.

  • content_format (ContentFormat | None) – Overrides content_format for this one document, for a parser whose output format depends on its own configuration rather than being fixed per class.

Return type:

Document

property cache: CacheStore | Literal[False] | None#

Return the cache explicitly configured for this step.

property config_hash: str#

Invalidate cached documents created before source-aware identities.

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:

context (ExecutionContext)

Return type:

Policy

async evaluate(context)#

Emit DocumentParsed per document.

Wraps DocumentStep’s concrete evaluate (which already resolved, per item, whether the document came from cache or a fresh transform() call — document.cached is trustworthy either way) rather than emitting inside a concrete parser’s own transform(), so the event fires the same way for a cache hit as for a fresh parse, and exactly once regardless of how many layers of routing (e.g. ParserRouter) produced the document.

Parameters:

context (ExecutionContext)

Return type:

AsyncIterator[Document]

async evaluate_items(items, context)#

Transform a bounded cache window independently of worker concurrency.

Parameters:
Return type:

AsyncIterator[Document]

event_name: str = 'Parsing'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

property policy: Policy | None#

Return the policy explicitly configured for this step.

replay_needs_content: bool = False#

Set by a step whose cached replay reads the full documents anyway (e.g. to re-validate them): its cache reads then skip the header-only projection rather than fetching every entry twice.

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:

list[T]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

validate_input(item)#

Validate declared source capabilities before work and cache replay.

Parameters:

item (Source)

Return type:

None

name: str#
class fastrag.ingestion.steps.parsing.MineruParser(*, api_key, base_url='https://mineru.agrospai.udl.cat', tier='standard', keep_images=False, concurrency=4, timeout=1000, pages=None, cache=None, policy=None)[source]#

Bases: PDFParser

Parse PDFs to Markdown (one document per page) with a MinerU API.

MinerU returns one Markdown blob per request, so the PDF’s pages are requested individually (concurrency at a time) to keep page provenance. Needs the pdf extra (pypdf) to count pages.

parser = MineruParser(api_key=MINERU_API_KEY, base_url=MINERU_URL)
Parameters:
content_format: ContentFormat = 'markdown'#

The ContentFormat this parser’s documents carry by default, stamped by build_document() into metadata["content_format"]. Override on a subclass whose output is always one format (HtmlParser/DocusaurusParser set MARKDOWN); pass content_format= to a single build_document() call instead for a parser whose output format depends on its own configuration.

config()[source]#

Parse-affecting settings, without credentials.

Return type:

Mapping[str, Any]

async parse_pdf(content, filename, pages)[source]#

Request each page and return its Markdown.

Parameters:
Return type:

PDFParseResult

accepted_media_types: frozenset[str] | None = frozenset({'application/pdf'})#

None retains legacy permissiveness; MIME patterns may use type/*.

build_document(source, content, metadata=None, *, content_format=None)#

Build one document from source’s parsed content.

Sets id (a hash of the source identity and parsed content), source_id, step_config_hash (this parser’s own config_hash), and the reserved metadata keys every parser gets right without writing them itself: source_uri, content_type (source’s own, as declared or guessed upstream), and content_format (what kind of string content is — content_format unless overridden for this one call). The site_name/favicon_url a fetcher found on source are carried over too, unless metadata sets its own.

Parameters:

  • source (Source) – The source content was parsed from.

  • content (str) – The parsed text.

  • metadata (dict[str, Any] | None) – Extra metadata to carry, e.g. an HTML page’s title.

  • content_format (ContentFormat | None) – Overrides content_format for this one document, for a parser whose output format depends on its own configuration rather than being fixed per class.

Return type:

Document

property cache: CacheStore | Literal[False] | None#

Return the cache explicitly configured for this step.

property config_hash: str#

Invalidate cached documents created before source-aware identities.

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:

context (ExecutionContext)

Return type:

Policy

async evaluate(context)#

Emit DocumentParsed per document.

Wraps DocumentStep’s concrete evaluate (which already resolved, per item, whether the document came from cache or a fresh transform() call — document.cached is trustworthy either way) rather than emitting inside a concrete parser’s own transform(), so the event fires the same way for a cache hit as for a fresh parse, and exactly once regardless of how many layers of routing (e.g. ParserRouter) produced the document.

Parameters:

context (ExecutionContext)

Return type:

AsyncIterator[Document]

async evaluate_items(items, context)#

Transform a bounded cache window independently of worker concurrency.

Parameters:
Return type:

AsyncIterator[Document]

event_name: str = 'Parsing'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

property policy: Policy | None#

Return the policy explicitly configured for this step.

replay_needs_content: bool = False#

Set by a step whose cached replay reads the full documents anyway (e.g. to re-validate them): its cache reads then skip the header-only projection rather than fetching every entry twice.

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:

list[T]

async transform(item, context)#

Yield one document per non-empty PDF page, in page order.

Parameters:
Return type:

AsyncIterator[Document]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

validate_input(item)#

Validate declared source capabilities before work and cache replay.

Parameters:

item (Source)

Return type:

None

name: str#
class fastrag.ingestion.steps.parsing.PDFParser(*, pages=None, cache=None, policy=None)[source]#

Bases: Parser

Base class of every PDF parser: one document per page.

Subclasses implement parse_pdf() and config(); they inherit media-type validation (application/pdf only), per-page document construction, page metadata and per-page identities.

Parameters:

  • pages (Pages | None)

  • cache (CacheStore | Literal[False] | None)

  • policy (Policy | None)

accepted_media_types: frozenset[str] | None = frozenset({'application/pdf'})#

None retains legacy permissiveness; MIME patterns may use type/*.

abstractmethod async parse_pdf(content, filename, pages)[source]#

Parse content (PDF bytes), returning per-page results.

pages is the sorted 1-based selection or None for all pages. Pages with no text may be omitted or returned empty.

Parameters:
Return type:

PDFParseResult

async transform(item, context)[source]#

Yield one document per non-empty PDF page, in page order.

Parameters:
Return type:

AsyncIterator[Document]

build_document(source, content, metadata=None, *, content_format=None)#

Build one document from source’s parsed content.

Sets id (a hash of the source identity and parsed content), source_id, step_config_hash (this parser’s own config_hash), and the reserved metadata keys every parser gets right without writing them itself: source_uri, content_type (source’s own, as declared or guessed upstream), and content_format (what kind of string content is — content_format unless overridden for this one call). The site_name/favicon_url a fetcher found on source are carried over too, unless metadata sets its own.

Parameters:

  • source (Source) – The source content was parsed from.

  • content (str) – The parsed text.

  • metadata (dict[str, Any] | None) – Extra metadata to carry, e.g. an HTML page’s title.

  • content_format (ContentFormat | None) – Overrides content_format for this one document, for a parser whose output format depends on its own configuration rather than being fixed per class.

Return type:

Document

property cache: CacheStore | Literal[False] | None#

Return the cache explicitly configured for this step.

abstractmethod config()#

Return this step’s own configuration, for cache-identity hashing.

Every concrete step declares this explicitly — steps are plain classes, not dataclasses, so there is no automatic field introspection. A step that forgets it cannot be instantiated: config is abstract, so construction itself fails loudly rather than config_hash silently collapsing to a constant.

Return type:

Mapping[str, Any]

property config_hash: str#

Invalidate cached documents created before source-aware identities.

content_format: ContentFormat = 'text'#

The ContentFormat this parser’s documents carry by default, stamped by build_document() into metadata["content_format"]. Override on a subclass whose output is always one format (HtmlParser/DocusaurusParser set MARKDOWN); pass content_format= to a single build_document() call instead for a parser whose output format depends on its own configuration.

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:

context (ExecutionContext)

Return type:

Policy

async evaluate(context)#

Emit DocumentParsed per document.

Wraps DocumentStep’s concrete evaluate (which already resolved, per item, whether the document came from cache or a fresh transform() call — document.cached is trustworthy either way) rather than emitting inside a concrete parser’s own transform(), so the event fires the same way for a cache hit as for a fresh parse, and exactly once regardless of how many layers of routing (e.g. ParserRouter) produced the document.

Parameters:

context (ExecutionContext)

Return type:

AsyncIterator[Document]

async evaluate_items(items, context)#

Transform a bounded cache window independently of worker concurrency.

Parameters:
Return type:

AsyncIterator[Document]

event_name: str = 'Parsing'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

property policy: Policy | None#

Return the policy explicitly configured for this step.

replay_needs_content: bool = False#

Set by a step whose cached replay reads the full documents anyway (e.g. to re-validate them): its cache reads then skip the header-only projection rather than fetching every entry twice.

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:

list[T]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

validate_input(item)#

Validate declared source capabilities before work and cache replay.

Parameters:

item (Source)

Return type:

None

name: str#
class fastrag.ingestion.steps.parsing.Parser(*, cache=None, policy=None)[source]#

Bases: DocumentStep, ABC

Turns raw Source bytes into text.

One source may yield several documents, which is why transform() (inherited abstract from DocumentStep) returns an async iterator rather than a single document. Binding an upstream (parser(sources), via the inherited __call__) and evaluating it (async for document in parser / run()) never runs a parse eagerly — only consuming the resulting stream does.

Every event a parser emits is labelled "Parsing", inherited by every concrete parser (HtmlParser, DocusaurusParser, ParserRouter, …) unless it declares an event_name of its own.

Parameters:
event_name: str = 'Parsing'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

content_format: ContentFormat = 'text'#

The ContentFormat this parser’s documents carry by default, stamped by build_document() into metadata["content_format"]. Override on a subclass whose output is always one format (HtmlParser/DocusaurusParser set MARKDOWN); pass content_format= to a single build_document() call instead for a parser whose output format depends on its own configuration.

accepted_media_types: frozenset[str] | None = None#

None retains legacy permissiveness; MIME patterns may use type/*.

property config_hash: str#

Invalidate cached documents created before source-aware identities.

validate_input(item)[source]#

Validate declared source capabilities before work and cache replay.

Parameters:

item (Source)

Return type:

None

async evaluate(context)[source]#

Emit DocumentParsed per document.

Wraps DocumentStep’s concrete evaluate (which already resolved, per item, whether the document came from cache or a fresh transform() call — document.cached is trustworthy either way) rather than emitting inside a concrete parser’s own transform(), so the event fires the same way for a cache hit as for a fresh parse, and exactly once regardless of how many layers of routing (e.g. ParserRouter) produced the document.

Parameters:

context (ExecutionContext)

Return type:

AsyncIterator[Document]

build_document(source, content, metadata=None, *, content_format=None)[source]#

Build one document from source’s parsed content.

Sets id (a hash of the source identity and parsed content), source_id, step_config_hash (this parser’s own config_hash), and the reserved metadata keys every parser gets right without writing them itself: source_uri, content_type (source’s own, as declared or guessed upstream), and content_format (what kind of string content is — content_format unless overridden for this one call). The site_name/favicon_url a fetcher found on source are carried over too, unless metadata sets its own.

Parameters:

  • source (Source) – The source content was parsed from.

  • content (str) – The parsed text.

  • metadata (dict[str, Any] | None) – Extra metadata to carry, e.g. an HTML page’s title.

  • content_format (ContentFormat | None) – Overrides content_format for this one document, for a parser whose output format depends on its own configuration rather than being fixed per class.

Return type:

Document

property cache: CacheStore | Literal[False] | None#

Return the cache explicitly configured for this step.

abstractmethod config()#

Return this step’s own configuration, for cache-identity hashing.

Every concrete step declares this explicitly — steps are plain classes, not dataclasses, so there is no automatic field introspection. A step that forgets it cannot be instantiated: config is abstract, so construction itself fails loudly rather than config_hash silently collapsing to a constant.

Return type:

Mapping[str, Any]

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:

context (ExecutionContext)

Return type:

Policy

async evaluate_items(items, context)#

Transform a bounded cache window independently of worker concurrency.

Parameters:
Return type:

AsyncIterator[Document]

property policy: Policy | None#

Return the policy explicitly configured for this step.

replay_needs_content: bool = False#

Set by a step whose cached replay reads the full documents anyway (e.g. to re-validate them): its cache reads then skip the header-only projection rather than fetching every entry twice.

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:

list[T]

abstractmethod transform(item, context)#

Transform one upstream item into zero or more output documents.

Parameters:

  • item (Any) – One item from this step’s upstream (a Source for a parser, or a Document for a chunker/embedder).

  • context (ExecutionContext) – Execution-scoped state and services.

Yields:

The documents this one upstream item produces.

Return type:

AsyncIterator[Document]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

name: str#
class fastrag.ingestion.steps.parsing.ParserRouter(*, cache=None, policy=None)[source]#

Bases: RoutedStep[Document], Parser

Route each source to the first matching .route parser, or .default.

Holds no parsing logic of its own: a source matching no route with no default registered is skipped with an ItemFailed.

from fastrag.ingestion.steps.parsing import (
    DefaultParser,
    HtmlParser,
    ParserRouter,
    html_filter,
)

parser = (
    ParserRouter(cache=cache)
    .route(html_filter, HtmlParser())
    .default(DefaultParser())
)
Parameters:
routing_requires_content: bool = False#

Unknown routing overrides conservatively require complete inputs. Opt out only when built-in selection uses identity/metadata alone.

accepted_media_types: frozenset[str] | None = None#

None retains legacy permissiveness; MIME patterns may use type/*.

build_document(source, content, metadata=None, *, content_format=None)#

Build one document from source’s parsed content.

Sets id (a hash of the source identity and parsed content), source_id, step_config_hash (this parser’s own config_hash), and the reserved metadata keys every parser gets right without writing them itself: source_uri, content_type (source’s own, as declared or guessed upstream), and content_format (what kind of string content is — content_format unless overridden for this one call). The site_name/favicon_url a fetcher found on source are carried over too, unless metadata sets its own.

Parameters:

  • source (Source) – The source content was parsed from.

  • content (str) – The parsed text.

  • metadata (dict[str, Any] | None) – Extra metadata to carry, e.g. an HTML page’s title.

  • content_format (ContentFormat | None) – Overrides content_format for this one document, for a parser whose output format depends on its own configuration rather than being fixed per class.

Return type:

Document

property cache: CacheStore | Literal[False] | None#

Return the cache explicitly configured for this step.

config()#

No identity-affecting configuration of its own.

A routed item’s cache identity is keyed by whichever child matched it (see _dispatch_route), never by the router itself.

Return type:

Mapping[str, Any]

property config_hash: str#

Invalidate cached documents created before source-aware identities.

content_format: ContentFormat = 'text'#

The ContentFormat this parser’s documents carry by default, stamped by build_document() into metadata["content_format"]. Override on a subclass whose output is always one format (HtmlParser/DocusaurusParser set MARKDOWN); pass content_format= to a single build_document() call instead for a parser whose output format depends on its own configuration.

default(step)#

Register a fallback step for items that match no route filters.

Parameters:

step (PipelineStep[T]) – Default child step.

Returns:

Self, for fluent chaining.

Return type:

Self

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:

context (ExecutionContext)

Return type:

Policy

async evaluate(context)#

Emit DocumentParsed per document.

Wraps DocumentStep’s concrete evaluate (which already resolved, per item, whether the document came from cache or a fresh transform() call — document.cached is trustworthy either way) rather than emitting inside a concrete parser’s own transform(), so the event fires the same way for a cache hit as for a fresh parse, and exactly once regardless of how many layers of routing (e.g. ParserRouter) produced the document.

Parameters:

context (ExecutionContext)

Return type:

AsyncIterator[Document]

async evaluate_items(items, context)#

Transform a bounded cache window independently of worker concurrency.

Parameters:
Return type:

AsyncIterator[Document]

event_name: str = 'Parsing'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

property policy: Policy | None#

Return the policy explicitly configured for this step.

replay_needs_content: bool = False#

Set by a step whose cached replay reads the full documents anyway (e.g. to re-validate them): its cache reads then skip the header-only projection rather than fetching every entry twice.

route(filter_rule, step)#

Register a conditional branch step matching a filter predicate.

Parameters:

  • filter_rule (Filter[T]) – Filter predicate to match against incoming items.

  • step (PipelineStep[T]) – Child step to run on match.

Returns:

Self, for fluent chaining.

Return type:

Self

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:

list[T]

async transform(item, context)#

Dispatch item to the first matching route, or the default.

An item matching no route with no default registered is skipped with an ItemFailed naming it — never silently dropped. A matched child that declares no explicit cache/policy of its own (cache is None) inherits this routed step’s effective cache and policy rather than the outer pipeline’s, since a routed child is conceptually scoped to its router. A child that does declare a policy has it applied to its own items, retries and timeout included — see _policy_for_item, which is where that half happens, since the retry wrap sits above this method rather than inside it.

Parameters:
Return type:

AsyncIterator[T]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

validate_input(item)#

Validate declared source capabilities before work and cache replay.

Parameters:

item (Source)

Return type:

None

routes: list[tuple[Filter[T], PipelineStep[T]]]#
name: str#
class fastrag.ingestion.steps.parsing.OpenAIVisionParser(*, model, api_key, base_url, prompt='Describe this image for a searchable documentation index. Transcribe meaningful text and explain diagram nodes, arrow directions and relationships. For screenshots describe the interface and visible values. Use concise English and only information visible in the image.', timeout=120, extra_body=None, max_tokens=2048, cache=None, policy=None)[source]#

Bases: Parser

Describe standalone images, or enrich images embedded in HTML parsers.

SVGs are rasterized with the optional dynamic extra. Credentials are excluded from cache identity; model, endpoint, prompt and extra body are included. extra_body adds JSON request fields and overrides matching defaults.

Parameters:
accepted_media_types: frozenset[str] | None = frozenset({'image/*'})#

None retains legacy permissiveness; MIME patterns may use type/*.

config()[source]#

Return description-affecting settings without credentials.

Return type:

Mapping[str, Any]

async describe(content, content_type)[source]#

Send image bytes as a data URL and return the generated description.

Parameters:
Return type:

str

async transform(item, context)[source]#

Yield one searchable description for an image source.

Parameters:
Return type:

AsyncIterator[Document]

build_document(source, content, metadata=None, *, content_format=None)#

Build one document from source’s parsed content.

Sets id (a hash of the source identity and parsed content), source_id, step_config_hash (this parser’s own config_hash), and the reserved metadata keys every parser gets right without writing them itself: source_uri, content_type (source’s own, as declared or guessed upstream), and content_format (what kind of string content is — content_format unless overridden for this one call). The site_name/favicon_url a fetcher found on source are carried over too, unless metadata sets its own.

Parameters:

  • source (Source) – The source content was parsed from.

  • content (str) – The parsed text.

  • metadata (dict[str, Any] | None) – Extra metadata to carry, e.g. an HTML page’s title.

  • content_format (ContentFormat | None) – Overrides content_format for this one document, for a parser whose output format depends on its own configuration rather than being fixed per class.

Return type:

Document

property cache: CacheStore | Literal[False] | None#

Return the cache explicitly configured for this step.

property config_hash: str#

Invalidate cached documents created before source-aware identities.

content_format: ContentFormat = 'text'#

The ContentFormat this parser’s documents carry by default, stamped by build_document() into metadata["content_format"]. Override on a subclass whose output is always one format (HtmlParser/DocusaurusParser set MARKDOWN); pass content_format= to a single build_document() call instead for a parser whose output format depends on its own configuration.

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:

context (ExecutionContext)

Return type:

Policy

async evaluate(context)#

Emit DocumentParsed per document.

Wraps DocumentStep’s concrete evaluate (which already resolved, per item, whether the document came from cache or a fresh transform() call — document.cached is trustworthy either way) rather than emitting inside a concrete parser’s own transform(), so the event fires the same way for a cache hit as for a fresh parse, and exactly once regardless of how many layers of routing (e.g. ParserRouter) produced the document.

Parameters:

context (ExecutionContext)

Return type:

AsyncIterator[Document]

async evaluate_items(items, context)#

Transform a bounded cache window independently of worker concurrency.

Parameters:
Return type:

AsyncIterator[Document]

event_name: str = 'Parsing'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

property policy: Policy | None#

Return the policy explicitly configured for this step.

replay_needs_content: bool = False#

Set by a step whose cached replay reads the full documents anyway (e.g. to re-validate them): its cache reads then skip the header-only projection rather than fetching every entry twice.

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:

list[T]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

validate_input(item)#

Validate declared source capabilities before work and cache replay.

Parameters:

item (Source)

Return type:

None

name: str#
class fastrag.ingestion.steps.parsing.SvgParser(*, strip_selectors=('script', 'style', 'noscript', 'nav', 'header', 'footer', 'aside'), encoding=None, vision=None, cache=None, policy=None)[source]#

Bases: HtmlParser

Extract titles, descriptions, text and foreignObject labels from SVG.

Parameters:
accepted_media_types: frozenset[str] | None = frozenset({'image/svg+xml'})#

None retains legacy permissiveness; MIME patterns may use type/*.

config()[source]#

Return label extraction settings for cache identity.

Return type:

Mapping[str, Any]

async transform(item, context)[source]#

Use the HTML content pipeline to preserve meaningful SVG labels.

Parameters:
Return type:

AsyncIterator[Document]

build_document(source, content, metadata=None, *, content_format=None)#

Build one document from source’s parsed content.

Sets id (a hash of the source identity and parsed content), source_id, step_config_hash (this parser’s own config_hash), and the reserved metadata keys every parser gets right without writing them itself: source_uri, content_type (source’s own, as declared or guessed upstream), and content_format (what kind of string content is — content_format unless overridden for this one call). The site_name/favicon_url a fetcher found on source are carried over too, unless metadata sets its own.

Parameters:

  • source (Source) – The source content was parsed from.

  • content (str) – The parsed text.

  • metadata (dict[str, Any] | None) – Extra metadata to carry, e.g. an HTML page’s title.

  • content_format (ContentFormat | None) – Overrides content_format for this one document, for a parser whose output format depends on its own configuration rather than being fixed per class.

Return type:

Document

property cache: CacheStore | Literal[False] | None#

Return the cache explicitly configured for this step.

property config_hash: str#

Invalidate cached documents created before source-aware identities.

content_format: ContentFormat = 'markdown'#

The ContentFormat this parser’s documents carry by default, stamped by build_document() into metadata["content_format"]. Override on a subclass whose output is always one format (HtmlParser/DocusaurusParser set MARKDOWN); pass content_format= to a single build_document() call instead for a parser whose output format depends on its own configuration.

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:

context (ExecutionContext)

Return type:

Policy

async evaluate(context)#

Emit DocumentParsed per document.

Wraps DocumentStep’s concrete evaluate (which already resolved, per item, whether the document came from cache or a fresh transform() call — document.cached is trustworthy either way) rather than emitting inside a concrete parser’s own transform(), so the event fires the same way for a cache hit as for a fresh parse, and exactly once regardless of how many layers of routing (e.g. ParserRouter) produced the document.

Parameters:

context (ExecutionContext)

Return type:

AsyncIterator[Document]

async evaluate_items(items, context)#

Transform a bounded cache window independently of worker concurrency.

Parameters:
Return type:

AsyncIterator[Document]

event_name: str = 'Parsing'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

property policy: Policy | None#

Return the policy explicitly configured for this step.

replay_needs_content: bool = False#

Set by a step whose cached replay reads the full documents anyway (e.g. to re-validate them): its cache reads then skip the header-only projection rather than fetching every entry twice.

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:

list[T]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

validate_input(item)#

Validate declared source capabilities before work and cache replay.

Parameters:

item (Source)

Return type:

None

name: str#
class fastrag.ingestion.steps.parsing.TrafilaturaParser(*, favor_precision=False, favor_recall=False, include_tables=True, include_links=True, include_images=False, include_comments=False, prune_xpath=(), encoding=None, cache=None, policy=None)[source]#

Bases: Parser

Extract a web page’s main content as Markdown, with no per-site selectors.

Trafilatura locates the article heuristically and discards navigation, sidebars, footers, and related-link blocks around it. Relative links are resolved against the source’s URI, and the page’s title, author, date, description, and site name travel in Document.metadata.

from fastrag.ingestion.steps.parsing import (
    ParserRouter,
    TrafilaturaParser,
    html_filter,
)

parser = ParserRouter(cache=cache).route(
    html_filter, TrafilaturaParser(favor_precision=True)
)
Parameters:

  • favor_precision (bool)

  • favor_recall (bool)

  • include_tables (bool)

  • include_links (bool)

  • include_images (bool)

  • include_comments (bool)

  • prune_xpath (Sequence[str])

  • encoding (str | None)

  • cache (CacheStore | Literal[False] | None)

  • policy (Policy | None)

content_format: ContentFormat = 'markdown'#

The ContentFormat this parser’s documents carry by default, stamped by build_document() into metadata["content_format"]. Override on a subclass whose output is always one format (HtmlParser/DocusaurusParser set MARKDOWN); pass content_format= to a single build_document() call instead for a parser whose output format depends on its own configuration.

accepted_media_types: frozenset[str] | None = frozenset({'application/xhtml+xml', 'text/html'})#

None retains legacy permissiveness; MIME patterns may use type/*.

config()[source]#

This parser’s cache-identity configuration.

Every option changes what a given page’s bytes parse into, so all of them are part of the fingerprint.

Return type:

Mapping[str, Any]

async transform(item, context)[source]#

Yield one Markdown document per page, or nothing if it has no content.

Parameters:
Return type:

AsyncIterator[Document]

build_document(source, content, metadata=None, *, content_format=None)#

Build one document from source’s parsed content.

Sets id (a hash of the source identity and parsed content), source_id, step_config_hash (this parser’s own config_hash), and the reserved metadata keys every parser gets right without writing them itself: source_uri, content_type (source’s own, as declared or guessed upstream), and content_format (what kind of string content is — content_format unless overridden for this one call). The site_name/favicon_url a fetcher found on source are carried over too, unless metadata sets its own.

Parameters:

  • source (Source) – The source content was parsed from.

  • content (str) – The parsed text.

  • metadata (dict[str, Any] | None) – Extra metadata to carry, e.g. an HTML page’s title.

  • content_format (ContentFormat | None) – Overrides content_format for this one document, for a parser whose output format depends on its own configuration rather than being fixed per class.

Return type:

Document

property cache: CacheStore | Literal[False] | None#

Return the cache explicitly configured for this step.

property config_hash: str#

Invalidate cached documents created before source-aware identities.

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:

context (ExecutionContext)

Return type:

Policy

async evaluate(context)#

Emit DocumentParsed per document.

Wraps DocumentStep’s concrete evaluate (which already resolved, per item, whether the document came from cache or a fresh transform() call — document.cached is trustworthy either way) rather than emitting inside a concrete parser’s own transform(), so the event fires the same way for a cache hit as for a fresh parse, and exactly once regardless of how many layers of routing (e.g. ParserRouter) produced the document.

Parameters:

context (ExecutionContext)

Return type:

AsyncIterator[Document]

async evaluate_items(items, context)#

Transform a bounded cache window independently of worker concurrency.

Parameters:
Return type:

AsyncIterator[Document]

event_name: str = 'Parsing'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

property policy: Policy | None#

Return the policy explicitly configured for this step.

replay_needs_content: bool = False#

Set by a step whose cached replay reads the full documents anyway (e.g. to re-validate them): its cache reads then skip the header-only projection rather than fetching every entry twice.

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:

list[T]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

validate_input(item)#

Validate declared source capabilities before work and cache replay.

Parameters:

item (Source)

Return type:

None

name: str#

Modules

ai

Model-guided content selection from supplied HTML, without fetching.

audio

Speech transcription through an OpenAI-compatible /audio/transcriptions endpoint.

base

The Parser family base.

capabilities

MIME capability helpers; source MIME and output representation stay distinct.

chandra

PDF parsing through a remote Chandra OCR API.

default

DefaultParser: decode the bytes and keep them as they are.

docling

Binary-format parsing via Docling.

docling_serve

PDF parsing through a remote docling-serve.

docusaurus

Docusaurus documentation parsing: the article, not the site chrome.

dynamic

SVG rasterization for vision parsing.

html

HTML to Markdown with hidden panels, SVG labels and optional vision descriptions.

mineru

PDF parsing through a remote MinerU API.

pdf

PDFParser: the shared interface of every PDF-to-text parser.

router

ParserRouter: dispatch each source to the parser registered for it.

svg

Searchable SVG labels without requiring a vision model.

trafilatura

Main-content extraction for arbitrary web pages, via Trafilatura.

vision

Image descriptions through an OpenAI-compatible vision endpoint.
On this page
Edit on GitHub
Show Source