fastrag.ingestion.logger.source_list#

Final flat URL list grouped by prefix.

Classes

SourceListLogger(pipeline, *[, stream, ...])

Print a final list of unique source URLs, one per line, grouped by prefix.
class fastrag.ingestion.logger.source_list.SourceListLogger(pipeline, *, stream=None, color=None, enrich_youtube=False)[source]#

Bases: Logger

Print a final list of unique source URLs, one per line, grouped by prefix.

The prefix of a URL is everything before its last path segment (scheme, domain and parent path). Groups are sorted by prefix and separated by one empty line; URLs inside a group are sorted. Each URL is colored by status: new is green, cached cyan and error red. With enrich_youtube=True, a YouTube video’s title follows its URL in parentheses. The last line, after an empty line, is the same summary printed by SourceTreeLogger. Event consumption and status rules are shared with SourceTreeLogger.

Parameters:

  • pipeline (Pipeline)

  • stream (TextIO | None)

  • color (bool | None)

  • enrich_youtube (bool)

async run()#

Consume events, then print the complete or interrupted summary.

Return type:

None

pipeline: Pipeline#