fastrag.ingestion.steps.parsing.docling#

Binary-format parsing via Docling.

The shipped parsers (HtmlParser, DefaultParser) only understand text and HTML. Spreadsheets, Word documents, and PDFs need an actual document converter, which is what DoclingParser hands off to.

Two output shapes, selected by output_format:

"markdown" (the default) — the converted text, so any existing chunker keeps working unmodified and never needs to know Docling was involved.
"docling" — the structured DoclingDocument itself, serialized as JSON, so a later native Docling chunker (task 012’s DoclingChunker) can split on the real structure instead of on flattened text.

metadata["content_format"] (~fastrag.models.ContentFormat.MARKDOWN or .DOCLING) is what tells a downstream step which shape it received.

Module Attributes

`DocumentStream`	The Docling classes, resolved on first use by `_load_docling()`.
`OFFICE_MEDIA_TYPES`	The "office" media types `DoclingParser` is meant to be routed to — spreadsheets, word-processor documents, and presentations, both the modern OOXML extensions and their legacy predecessors.
`office_filter`	The filter a `ParserRouter` route to `DoclingParser` matches against — the counterpart of `html_filter` for office documents.

Classes

DoclingParser(*[, output_format, ...])

Convert a binary document to Markdown or a serialized DoclingDocument.

fastrag.ingestion.steps.parsing.docling.DocumentStream: Any = None#: The Docling classes, resolved on first use by _load_docling(). Importing Docling pulls in torch and transformers (seconds), which a pipeline that never parses a binary document should not pay. They stay None when the optional “docling” extra is missing; DoclingParser.__init__ refuses to run until they exist.

fastrag.ingestion.steps.parsing.docling.OFFICE_MEDIA_TYPES: frozenset[str] = frozenset({'application/msword', 'application/vnd.ms-excel', 'application/vnd.ms-powerpoint', 'application/vnd.openxmlformats-officedocument.presentationml.presentation', 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', 'application/vnd.openxmlformats-officedocument.wordprocessingml.document'})#: The “office” media types DoclingParser is meant to be routed to — spreadsheets, word-processor documents, and presentations, both the modern OOXML extensions and their legacy predecessors. Deliberately narrower than every format Docling itself can convert (PDF and raw images among them): those have their own, better-fitting existing routes (or none yet), and “office” names what this filter is for, not an exhaustive list of DocumentConverter’s support matrix.

fastrag.ingestion.steps.parsing.docling.office_filter = <fastrag.ingestion.filters.mediatype.MediaTypeFilter object>#

The filter a ParserRouter route to DoclingParser matches against — the counterpart of html_filter for office documents.

parser = ParserRouter(cache=cache).route(office_filter, DoclingParser())

Parameters:: item (T_contra)
Return type:: bool

class fastrag.ingestion.steps.parsing.docling.DoclingParser(*, output_format='markdown', describe_images=False, picture_description_options=None, allowed_formats=None, format_options=None, cache=None, policy=None)[source]#

Bases: Parser

Convert a binary document to Markdown or a serialized DoclingDocument.

Hands a Source’s bytes to Docling’s DocumentConverter, entirely from memory (never a temporary file): Docling infers the source format from a filename derived from the source’s URI, not from anything written to disk.

from fastrag.ingestion.steps.parsing import DoclingParser

# Markdown, chunked by whatever chunker the caller already uses.
parser = DoclingParser()

# The structured document itself, for a Docling-native chunker.
parser = DoclingParser(output_format="docling")

# Describe the pictures found in PDFs (runs a vision model).
parser = DoclingParser(describe_images=True)

In "markdown" mode, a paginated document (PDF, slides) is organized by page: one Document per non-empty page, each carrying metadata["page_no"] (1-based), metadata["page_count"] and metadata["source_url"]. For PDFs source_url is the source URI with a #page=N fragment, so every chunk inherits a link to the exact page it came from. Documents without pages (.docx, HTML, …) are yielded whole, without page_no.

Conversion is synchronous and CPU-bound, so it always runs through asyncio.to_thread rather than blocking the event loop it is awaited from.

Parameters:

output_format (OutputFormat)
describe_images (bool)
picture_description_options (Any | None)
allowed_formats (Sequence[InputFormat] | None)
format_options (Mapping[InputFormat, Any] | None)
cache (CacheStore | Literal[False] | None)
policy (Policy | None)

accepted_media_types: frozenset[str] | None = frozenset({'application/msword', 'application/pdf', 'application/vnd.ms-excel', 'application/vnd.ms-powerpoint', 'application/vnd.openxmlformats-officedocument.presentationml.presentation', 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', 'application/vnd.openxmlformats-officedocument.wordprocessingml.document', 'application/xhtml+xml', 'image/*', 'text/*'})#: None retains legacy permissiveness; MIME patterns may use type/*.

config()[source]#

This parser’s cache-identity configuration.

output_format changes the shape of what a given file parses into — Markdown text versus a serialized DoclingDocument — so it is part of the fingerprint just as much as allowed_formats (which formats Docling will even attempt) is: switching either invalidates the cache instead of serving a caller who asked for one shape the other shape cached earlier. format_options belongs for the same reason: it changes what Docling extracts from a given file (picture descriptions, say), so it must not serve a caller the pre-enrichment text cached earlier.

Return type:: Mapping[str, Any]

async transform(item, context)[source]#

Yield one document per non-empty page, or nothing if there is no text.

Parameters:

item (Source)
context (ExecutionContext)

Return type:

AsyncIterator[Document]

build_document(source, content, metadata=None, *, content_format=None)#

Build one document from source’s parsed content.

Sets id (a hash of the source identity and parsed content), source_id, step_config_hash (this parser’s own config_hash), and the reserved metadata keys every parser gets right without writing them itself: source_uri, content_type (source’s own, as declared or guessed upstream), and content_format (what kind of string content is — content_format unless overridden for this one call). The site_name/favicon_url a fetcher found on source are carried over too, unless metadata sets its own.

Parameters:

source (Source) – The source content was parsed from.
content (str) – The parsed text.
metadata (dict[str, Any] | None) – Extra metadata to carry, e.g. an HTML page’s title.
content_format (ContentFormat | None) – Overrides content_format for this one document, for a parser whose output format depends on its own configuration rather than being fixed per class.

Return type:

Document

property cache: CacheStore | Literal[False] | None#: Return the cache explicitly configured for this step.

property config_hash: str#: Invalidate cached documents created before source-aware identities.

content_format: ContentFormat = 'text'#: The ContentFormat this parser’s documents carry by default, stamped by build_document() into metadata["content_format"]. Override on a subclass whose output is always one format (HtmlParser/DocusaurusParser set MARKDOWN); pass content_format= to a single build_document() call instead for a parser whose output format depends on its own configuration.

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:: context (ExecutionContext)
Return type:: Policy

async evaluate(context)#

Emit DocumentParsed per document.

Wraps DocumentStep’s concrete evaluate (which already resolved, per item, whether the document came from cache or a fresh transform() call — document.cached is trustworthy either way) rather than emitting inside a concrete parser’s own transform(), so the event fires the same way for a cache hit as for a fresh parse, and exactly once regardless of how many layers of routing (e.g. ParserRouter) produced the document.

Parameters:: context (ExecutionContext)
Return type:: AsyncIterator[Document]

async evaluate_items(items, context)#

Transform a bounded cache window independently of worker concurrency.

Parameters:

items (AsyncIterator[Any])
context (ExecutionContext)

Return type:

AsyncIterator[Document]

event_name: str = 'Parsing'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

property policy: Policy | None#: Return the policy explicitly configured for this step.

replay_needs_content: bool = False#: Set by a step whose cached replay reads the full documents anyway (e.g. to re-validate them): its cache reads then skip the header-only projection rather than fetching every entry twice.

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:: list[T]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

validate_input(item)#

Validate declared source capabilities before work and cache replay.

Parameters:: item (Source)
Return type:: None

name: str#