fastrag.ingestion.logger.documents#

Inspect documents produced by parsing, chunking, embedding or indexing.

Classes

DocumentsLogger(pipeline, *[, format, ...])

Display complete text, metadata, embeddings, lineage and size statistics.
class fastrag.ingestion.logger.documents.DocumentsLogger(pipeline, *, format='rich', stream=None, configuration=None, stages=None, content_lines=None, content_chars=None, metadata_only=False, show_binary=False)[source]#

Bases: Logger

Display complete text, metadata, embeddings, lineage and size statistics.

Accepts pipeline, format="rich" or "jsonl", stream, configuration and stages. By default only the pipeline root’s stage is inspected; stages=() inspects none, and an explicit sequence such as ("Parsing", "Chunking") inspects multiple stages. Streams without a root are inspected at every document stage. All failures are reported regardless of the stage selection. Rich requires the logging extra; JSON Lines needs no optional dependencies.

document_count counts output documents (including chunks), chunk_count counts chunk records. Counters reset for each run.

Set content_lines=N to show only the first and last N text lines, separated by an omitted-line marker. Short content is shown in full. The default None shows everything. This applies to Rich and JSONL text output; metadata, base64 bytes and size statistics remain complete.

metadata_only=True omits text and embeddings, keeping metadata and size statistics. show_binary=True disables binary content trimming.

Parameters:

  • pipeline (Pipeline)

  • format (Literal['rich', 'jsonl'])

  • stream (TextIO | None)

  • configuration (Mapping[str, Any] | None)

  • stages (Sequence[str] | None)

  • content_lines (int | None)

  • content_chars (int | None)

  • metadata_only (bool)

  • show_binary (bool)

label = 'Document inspection'#
async run()#

Inspect items and write a partial summary before propagating failures.

Return type:

None

pipeline: Pipeline#