fastrag.ingestion.steps.fetching.http#

Shared HTTP transport for the fetching steps that speak HTTP.

Non-public: _HttpTransport and its helpers are plumbing shared by concrete fetchers (SitemapCrawler and any future HTTP-speaking Fetcher), not part of this package’s public surface. Ported from src/fastrag/steps/fetching/http.py — connection reuse, user agent, timeouts, redirect following, and transient-status classification are unchanged; only the shape around it differs, since the new Fetcher is a plain SourceStep rather than a fingerprinted dataclass, so this transport is held as an attribute instead of mixed in via inheritance.

Functions

normalize_uri(uri)

Return a normalized identifier for uri, used as Source.id.

Classes

FetchedPage(uri, content, content_type, ...)

The raw HTTP response for one fetched URI, before it becomes a Source.

class fastrag.ingestion.steps.fetching.http.FetchedPage(uri, content, content_type, etag, last_modified, status_code)[source]#

Bases: object

The raw HTTP response for one fetched URI, before it becomes a Source.

Parameters:

uri (str)
content (bytes)
content_type (str)
etag (str | None)
last_modified (datetime | None)
status_code (int)

fastrag.ingestion.steps.fetching.http.normalize_uri(uri)[source]#

Return a normalized identifier for uri, used as Source.id.

Lowercases the scheme and host, adds https:// when the caller typed a bare host (no ://), drops a default port for the scheme, drops the fragment, and drops a trailing slash from the path. The scheme is kept (unlike API.md’s illustrative normalized("https://b.com/") == "b.com"): dropping it would collide http://a.com and https://a.com, which are not the same resource.

Parameters:: uri (str)
Return type:: str