fastrag.ingestion.steps.fetching.base#

The Fetcher interface: docs/specification/subclasses.md, verbatim.

Classes

Fetcher(*args, **kwargs)

Base class for leaf-level data fetchers that extract raw content from a single protocol/medium.
class fastrag.ingestion.steps.fetching.base.Fetcher(*args, **kwargs)[source]#

Bases: SourceStep, ABC

Base class for leaf-level data fetchers that extract raw content from a single protocol/medium.

Every event a fetcher emits is labelled "Fetching" — restated here rather than only inherited from SourceStep so the family’s own module states the stage it belongs to.

Fetchers that discover URLs (crawlers) can delegate specific ones to other fetchers with route(), e.g. YouTube links to a YouTubeFetcher.

Parameters:

  • args (Any)

  • kwargs (Any)

event_name: str = 'Fetching'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

routes: list[tuple[Filter[Source], Fetcher]]#
route(filter_rule, fetcher)[source]#

Delegate the URLs matching filter_rule to fetcher.

The filter sees a content-less Source carrying only the URL, so URL-based filters such as URLFilter apply. Routes are tried in registration order, and the first match wins.

Parameters:

  • filter_rule (Filter[Source]) – Predicate over the URL about to be fetched.

  • fetcher (Fetcher) – Fetcher implementing fetch(), used on match.

Returns:

Self, for fluent chaining.

Return type:

Self

route_for(uri)[source]#

The fetcher routed to uri, or None to fetch it normally.

Parameters:

uri (str)

Return type:

Fetcher | None

canonical_uri(uri)[source]#

Identity under which uri is fetched, to deduplicate equivalent spellings.

Parameters:

uri (str)

Return type:

str

async fetch(uri)[source]#

Fetch one URL in-process, uncached. Needed to be a route() target.

Raises:

fastrag.exceptions.ConfigurationError – This fetcher cannot fetch an arbitrary URL.

Parameters:

uri (str)

Return type:

list[Source]

routes_config()[source]#

Routed fetchers’ fingerprints, for inclusion in a crawler’s config().

Return type:

list[dict[str, Any]]

property cache: CacheStore | Literal[False] | None#

Return the cache explicitly configured for this step.

abstractmethod config()#

Return this step’s own configuration, for cache-identity hashing.

Every concrete step declares this explicitly — steps are plain classes, not dataclasses, so there is no automatic field introspection. A step that forgets it cannot be instantiated: config is abstract, so construction itself fails loudly rather than config_hash silently collapsing to a constant.

Return type:

Mapping[str, Any]

property config_hash: str#

A stable hash of this step’s class name and config().

Two identically configured steps of the same class hash equal; changing any declared config value changes the hash; the hash is stable across processes. Used as part of every cache id this step’s evaluation reads or writes.

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:

context (ExecutionContext)

Return type:

Policy

abstractmethod evaluate(context)#

Lazily evaluate the step within an execution context.

Parameters:

context (ExecutionContext) – Execution-scoped state and services.

Yields:

Values produced by the step.

Return type:

AsyncIterator[T]

property policy: Policy | None#

Return the policy explicitly configured for this step.

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:

list[T]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

name: str#