fastrag.ingestion.logger.source#

Inspect fetched source bytes without losing binary content.

Classes

SourceLogger(pipeline, *[, format, stream, ...])

Display unique fetched sources with complete metadata and content.
class fastrag.ingestion.logger.source.SourceLogger(pipeline, *, format='rich', stream=None, configuration=None, stages=None, content_lines=None, content_chars=None, metadata_only=False, show_binary=False)[source]#

Bases: Logger

Display unique fetched sources with complete metadata and content.

Accepts pipeline, format="rich" or "jsonl", stream, configuration and optional stages. UTF-8 content is displayed as literal text; other content is displayed as base64. JSON Lines always includes content_base64 for exact byte recovery. source_count counts unique source IDs, deduplicating events from Sources aggregates. Counters reset on each run; failures propagate after a partial summary.

Set content_lines=N to show only the first and last N text lines, separated by an omitted-line marker. Short content is shown in full. The default None shows everything. This applies to Rich and JSONL text output; metadata, base64 bytes and size statistics remain complete. content_chars=N additionally caps preview text at N characters, useful for minified HTML with very few lines. Both limits default to None.

metadata_only=True omits text, content_base64 and embeddings, keeping metadata and size statistics. Non-text content is trimmed to a short base64 preview (content_truncated) unless show_binary=True.

Parameters:

  • pipeline (Pipeline)

  • format (Literal['rich', 'jsonl'])

  • stream (TextIO | None)

  • configuration (Mapping[str, Any] | None)

  • stages (Sequence[str] | None)

  • content_lines (int | None)

  • content_chars (int | None)

  • metadata_only (bool)

  • show_binary (bool)

label = 'Source inspection'#
async run()#

Inspect items and write a partial summary before propagating failures.

Return type:

None

pipeline: Pipeline#