fastrag.ingestion.steps.parsing.pdf#

PDFParser: the shared interface of every PDF-to-text parser.

Concrete PDF parsers (MineruParser, DoclingServeParser, ChandraParser) differ only in how they turn PDF bytes into text. They all share this contract, so they can be swapped without touching the pipeline:

  • The input is always a PDF (application/pdf).

  • The output is one Document per PDF page, so the source and the page are tracked in every document’s metadata (source_uri, page_no, page_count, source_url).

  • metadata["content_format"] states what each page’s content is (Markdown, HTML, …) so a downstream step can pick the matching chunker.

Because pages are emitted as separate, ordered, 1-based-numbered documents linked by source_id, a later chunker can join pages back together (e.g. to keep a paragraph or table that spans a page break in one chunk) without the parser having to know about it.

A new parser only implements PDFParser.parse_pdf().

Module Attributes

pdf_filter

The filter a ParserRouter route to a PDFParser matches against.

Functions

contiguous_runs(pages)

[1, 2, 3, 7, 8] -> [(1, 3), (7, 8)].

normalize_pages(pages)

Turn "1-3,5" / [1, 2, 3, 5] into a sorted, de-duplicated list.

pdf_page_count(content)

Count a PDF's pages (needs the pdf extra: pypdf).

Classes

PDFParseResult(pages[, page_count, metadata])

What a backend produced for a whole PDF.

PDFParser(*[, pages, cache, policy])

Base class of every PDF parser: one document per page.

ParsedPage(page_no, content[, ...])

One page's parsed content, as returned by PDFParser.parse_pdf().
fastrag.ingestion.steps.parsing.pdf.pdf_filter = <fastrag.ingestion.filters.mediatype.MediaTypeFilter object>#

The filter a ParserRouter route to a PDFParser matches against.

parser = ParserRouter().route(pdf_filter, ChandraParser(...))
Parameters:

item (T_contra)

Return type:

bool

class fastrag.ingestion.steps.parsing.pdf.ParsedPage(page_no, content, content_format=ContentFormat.MARKDOWN, metadata=<factory>)[source]#

Bases: object

One page’s parsed content, as returned by PDFParser.parse_pdf().

Parameters:
class fastrag.ingestion.steps.parsing.pdf.PDFParseResult(pages, page_count=None, metadata=<factory>)[source]#

Bases: object

What a backend produced for a whole PDF.

Parameters:
fastrag.ingestion.steps.parsing.pdf.normalize_pages(pages)[source]#

Turn "1-3,5" / [1, 2, 3, 5] into a sorted, de-duplicated list.

Raises:

fastrag.exceptions.ConfigurationError – Malformed or non-positive pages.

Parameters:

pages (str | Sequence[int] | None)

Return type:

list[int] | None

fastrag.ingestion.steps.parsing.pdf.contiguous_runs(pages)[source]#

[1, 2, 3, 7, 8] -> [(1, 3), (7, 8)].

Parameters:

pages (Sequence[int])

Return type:

list[tuple[int, int]]

fastrag.ingestion.steps.parsing.pdf.pdf_page_count(content)[source]#

Count a PDF’s pages (needs the pdf extra: pypdf).

Parameters:

content (bytes)

Return type:

int

class fastrag.ingestion.steps.parsing.pdf.PDFParser(*, pages=None, cache=None, policy=None)[source]#

Bases: Parser

Base class of every PDF parser: one document per page.

Subclasses implement parse_pdf() and config(); they inherit media-type validation (application/pdf only), per-page document construction, page metadata and per-page identities.

Parameters:

  • pages (Pages | None)

  • cache (CacheStore | Literal[False] | None)

  • policy (Policy | None)

accepted_media_types: frozenset[str] | None = frozenset({'application/pdf'})#

None retains legacy permissiveness; MIME patterns may use type/*.

abstractmethod async parse_pdf(content, filename, pages)[source]#

Parse content (PDF bytes), returning per-page results.

pages is the sorted 1-based selection or None for all pages. Pages with no text may be omitted or returned empty.

Parameters:
Return type:

PDFParseResult

async transform(item, context)[source]#

Yield one document per non-empty PDF page, in page order.

Parameters:
Return type:

AsyncIterator[Document]

build_document(source, content, metadata=None, *, content_format=None)#

Build one document from source’s parsed content.

Sets id (a hash of the source identity and parsed content), source_id, step_config_hash (this parser’s own config_hash), and the reserved metadata keys every parser gets right without writing them itself: source_uri, content_type (source’s own, as declared or guessed upstream), and content_format (what kind of string content is — content_format unless overridden for this one call). The site_name/favicon_url a fetcher found on source are carried over too, unless metadata sets its own.

Parameters:

  • source (Source) – The source content was parsed from.

  • content (str) – The parsed text.

  • metadata (dict[str, Any] | None) – Extra metadata to carry, e.g. an HTML page’s title.

  • content_format (ContentFormat | None) – Overrides content_format for this one document, for a parser whose output format depends on its own configuration rather than being fixed per class.

Return type:

Document

property cache: CacheStore | Literal[False] | None#

Return the cache explicitly configured for this step.

abstractmethod config()#

Return this step’s own configuration, for cache-identity hashing.

Every concrete step declares this explicitly — steps are plain classes, not dataclasses, so there is no automatic field introspection. A step that forgets it cannot be instantiated: config is abstract, so construction itself fails loudly rather than config_hash silently collapsing to a constant.

Return type:

Mapping[str, Any]

property config_hash: str#

Invalidate cached documents created before source-aware identities.

content_format: ContentFormat = 'text'#

The ContentFormat this parser’s documents carry by default, stamped by build_document() into metadata["content_format"]. Override on a subclass whose output is always one format (HtmlParser/DocusaurusParser set MARKDOWN); pass content_format= to a single build_document() call instead for a parser whose output format depends on its own configuration.

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:

context (ExecutionContext)

Return type:

Policy

async evaluate(context)#

Emit DocumentParsed per document.

Wraps DocumentStep’s concrete evaluate (which already resolved, per item, whether the document came from cache or a fresh transform() call — document.cached is trustworthy either way) rather than emitting inside a concrete parser’s own transform(), so the event fires the same way for a cache hit as for a fresh parse, and exactly once regardless of how many layers of routing (e.g. ParserRouter) produced the document.

Parameters:

context (ExecutionContext)

Return type:

AsyncIterator[Document]

async evaluate_items(items, context)#

Transform a bounded cache window independently of worker concurrency.

Parameters:
Return type:

AsyncIterator[Document]

event_name: str = 'Parsing'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

property policy: Policy | None#

Return the policy explicitly configured for this step.

replay_needs_content: bool = False#

Set by a step whose cached replay reads the full documents anyway (e.g. to re-validate them): its cache reads then skip the header-only projection rather than fetching every entry twice.

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:

list[T]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

validate_input(item)#

Validate declared source capabilities before work and cache replay.

Parameters:

item (Source)

Return type:

None

name: str#