fastrag.ingestion.logger#

Pipeline event consumers for progress, documents and raw sources.

class fastrag.ingestion.logger.ChunkLogger(pipeline, *, format='rich', stream=None, configuration=None, stages=None, content_lines=None, content_chars=None, metadata_only=False, show_binary=False)[source]#

Bases: DocumentsLogger

Compatibility logger for chunking events anywhere in the graph.

New code should use DocumentsLogger. The summary’s documents and parent_documents both count chunking inputs, as in the former API.

Parameters:

pipeline (Pipeline)
format (Literal['rich', 'jsonl'])
stream (TextIO | None)
configuration (Mapping[str, Any] | None)
stages (Sequence[str] | None)
content_lines (int | None)
content_chars (int | None)
metadata_only (bool)
show_binary (bool)

label = 'Document inspection'#

async run()#

Inspect items and write a partial summary before propagating failures.

Return type:: None

pipeline: Pipeline#

class fastrag.ingestion.logger.ConsoleLogger(pipeline, *, show_progress=True, show_domain_events=False, show_errors=False, refresh_interval=0.5, column_order=None, stream=None, heartbeat_interval=15.0, slow_after=60.0, stall_after=30.0)[source]#

Bases: Logger

Renders a pipeline’s lifecycle and domain events to the console.

Aggregates item-level events into the per-source, per-stage progress table docs/specification/API.md’s “Example output” shows — one row per leaf fetcher, one column per stage, each cell new/cached/errors — refreshed in place on a TTY, or printed once at the end otherwise (piping to a file must never see escape sequences).

Parameters:

pipeline (Pipeline)
show_progress (bool)
show_domain_events (bool)
show_errors (bool)
refresh_interval (float)
column_order (Sequence[str] | None)
stream (Any)
heartbeat_interval (float)
slow_after (float)
stall_after (float)

show_progress: bool#

show_domain_events: bool#

show_errors: bool#

refresh_interval: float#

async run()[source]#

Consume the pipeline’s event stream and render progress as it arrives.

Iterating self.pipeline both drives the root step to completion and yields each event; a step failure under policy.on_error == "raise" propagates out of this call the same way it does out of Pipeline.run(), after every event already produced has been handled. If any ItemFailed could not be attributed to a known row, a trailing summary line is written after the final render so that failure is never silently dropped.

A background ticker keeps the output alive while no event arrives: on a TTY it redraws the table and status every refresh_interval; otherwise it prints a status line every heartbeat_interval. Failures are printed as they happen when show_errors is set, and listed again at the end.

Return type:: None

pipeline: Pipeline#

class fastrag.ingestion.logger.DocumentsLogger(pipeline, *, format='rich', stream=None, configuration=None, stages=None, content_lines=None, content_chars=None, metadata_only=False, show_binary=False)[source]#

Bases: Logger

Display complete text, metadata, embeddings, lineage and size statistics.

Accepts pipeline, format="rich" or "jsonl", stream, configuration and stages. By default only the pipeline root’s stage is inspected; stages=() inspects none, and an explicit sequence such as ("Parsing", "Chunking") inspects multiple stages. Streams without a root are inspected at every document stage. All failures are reported regardless of the stage selection. Rich requires the logging extra; JSON Lines needs no optional dependencies.

document_count counts output documents (including chunks), chunk_count counts chunk records. Counters reset for each run.

Set content_lines=N to show only the first and last N text lines, separated by an omitted-line marker. Short content is shown in full. The default None shows everything. This applies to Rich and JSONL text output; metadata, base64 bytes and size statistics remain complete.

metadata_only=True omits text and embeddings, keeping metadata and size statistics. show_binary=True disables binary content trimming.

Parameters:

pipeline (Pipeline)
format (Literal['rich', 'jsonl'])
stream (TextIO | None)
configuration (Mapping[str, Any] | None)
stages (Sequence[str] | None)
content_lines (int | None)
content_chars (int | None)
metadata_only (bool)
show_binary (bool)

label = 'Document inspection'#

async run()#

Inspect items and write a partial summary before propagating failures.

Return type:: None

pipeline: Pipeline#

class fastrag.ingestion.logger.Logger(pipeline)[source]#

Bases: ABC

Base class for components that consume a pipeline’s event stream.

A Logger is associated with one Pipeline at construction and does nothing until run() is awaited — matching every PipelineStep’s own “nothing happens at construction” contract.

Parameters:: pipeline (Pipeline)

pipeline: Pipeline#

abstractmethod async run()[source]#

Consume and process every event from pipeline.

Implementations iterate self.pipeline (async for event in self.pipeline:), which both drives the pipeline’s root step to completion and yields each event as it is emitted — this is also what propagates a step failure out of run(), exactly as it does out of run(), under policy.on_error == "raise".

Return type:: None

class fastrag.ingestion.logger.SourceListLogger(pipeline, *, stream=None, color=None, enrich_youtube=False)[source]#

Bases: Logger

Print a final list of unique source URLs, one per line, grouped by prefix.

The prefix of a URL is everything before its last path segment (scheme, domain and parent path). Groups are sorted by prefix and separated by one empty line; URLs inside a group are sorted. Each URL is colored by status: new is green, cached cyan and error red. With enrich_youtube=True, a YouTube video’s title follows its URL in parentheses. The last line, after an empty line, is the same summary printed by SourceTreeLogger. Event consumption and status rules are shared with SourceTreeLogger.

Parameters:

pipeline (Pipeline)
stream (TextIO | None)
color (bool | None)
enrich_youtube (bool)

async run()#

Consume events, then print the complete or interrupted summary.

Return type:: None

pipeline: Pipeline#

class fastrag.ingestion.logger.SourceLogger(pipeline, *, format='rich', stream=None, configuration=None, stages=None, content_lines=None, content_chars=None, metadata_only=False, show_binary=False)[source]#

Bases: Logger

Display unique fetched sources with complete metadata and content.

Accepts pipeline, format="rich" or "jsonl", stream, configuration and optional stages. UTF-8 content is displayed as literal text; other content is displayed as base64. JSON Lines always includes content_base64 for exact byte recovery. source_count counts unique source IDs, deduplicating events from Sources aggregates. Counters reset on each run; failures propagate after a partial summary.

Set content_lines=N to show only the first and last N text lines, separated by an omitted-line marker. Short content is shown in full. The default None shows everything. This applies to Rich and JSONL text output; metadata, base64 bytes and size statistics remain complete. content_chars=N additionally caps preview text at N characters, useful for minified HTML with very few lines. Both limits default to None.

metadata_only=True omits text, content_base64 and embeddings, keeping metadata and size statistics. Non-text content is trimmed to a short base64 preview (content_truncated) unless show_binary=True.

Parameters:

pipeline (Pipeline)
format (Literal['rich', 'jsonl'])
stream (TextIO | None)
configuration (Mapping[str, Any] | None)
stages (Sequence[str] | None)
content_lines (int | None)
content_chars (int | None)
metadata_only (bool)
show_binary (bool)

label = 'Source inspection'#

async run()#

Inspect items and write a partial summary before propagating failures.

Return type:: None

pipeline: Pipeline#

class fastrag.ingestion.logger.SourceTreeLogger(pipeline, *, max_depth=None, stream=None, color=None, enrich_youtube=False)[source]#

Bases: Logger

Print a final tree of unique source URLs and their subtree totals.

Domain nodes have depth 1; each path segment adds one level. Schemes, queries, fragments and trailing slashes distinguish URLs for counting, but share the same domain/path branch. A YouTube /watch URL adds its ?v= video ID as one more level. Every node includes URLs at that path and below it, even when descendants are hidden by max_depth. Path labels below domain roots start with a slash. Each line shows nonzero colored status numbers; a final summary names and totals them.

Status rules, enrich_youtube titles (shown in parentheses after the video’s label) and failure handling are those of the shared source summary base. No optional logging dependency is required.

Parameters:

pipeline (Pipeline)
max_depth (int | None)
stream (TextIO | None)
color (bool | None)
enrich_youtube (bool)

async run()#

Consume events, then print the complete or interrupted summary.

Return type:: None

pipeline: Pipeline#

Modules

`base`	`Logger`: the base class for a pipeline event stream consumer.
`chunk`	Compatibility import for the former chunk-only logger.
`console`	`ConsoleLogger`: renders a pipeline's event stream to the console.
`documents`	Inspect documents produced by parsing, chunking, embedding or indexing.
`source`	Inspect fetched source bytes without losing binary content.
`source_list`	Final flat URL list grouped by prefix.
`source_tree`	Final URL tree summaries grouped by domain and path.