fastrag.ingestion.pipeline#

Pipeline: the context manager that evaluates a lazy step graph.

Constructing a Pipeline never evaluates anything — it only records the root step and the pipeline-level defaults (cache/policy) a step falls back to when it declares none of its own. Entering it (async with) installs an event sink (via fastrag.ingestion.steps.base.event_sink_scope()) so that any step evaluated during the block — not just the ones this Pipeline drives itself — has its events routed here. __aiter__/run() drive the root step’s evaluation under one ExecutionContext built from this pipeline’s own cache/policy, concurrently draining the events it produces through a queue.

Classes

Pipeline(root, *[, cache, policy, ...])

Evaluates a lazy PipelineStep graph.
class fastrag.ingestion.pipeline.Pipeline(root, *, cache=None, policy=None, source_headers_only=False, headers_only=False)[source]#

Bases: AbstractAsyncContextManager[Pipeline]

Evaluates a lazy PipelineStep graph.

Pipeline owns the ExecutionContext for a whole run: it is responsible for creating it (with this pipeline’s own default cache/policy) and for installing the event sink every step in the graph — however deeply nested — ultimately emits into.

async with Pipeline(vector_store.save(embedded_documents)) as pipeline:
    await ConsoleLogger(pipeline).run()
Parameters:
property root: PipelineStep[Any]#

The step this pipeline evaluates — and the top of its upstream chain.

async run()[source]#

Evaluate root while discarding every event it produces.

For when event consumption is handled by something else — a Logger iterating this same pipeline concurrently, e.g. — rather than by the caller of run() itself.

Return type:

None