fastrag.ingestion.models

Value types produced by the ingestion pipeline’s source-fetching stage.

A Source is raw fetched content, produced by a fetcher and consumed by a parser. What a Source becomes once parsed — Document — is shared by every later stage (parsing, chunking, embedding, indexing) as well as by retrieval, querying, serving and benchmarking, so it lives at the top level, in fastrag.models, rather than here.

Classes

Source(id, uri, content, content_type, ...)

Raw content fetched from somewhere, plus enough metadata to place it.

class fastrag.ingestion.models.Source(id, uri, content, content_type, content_hash, etag=None, last_modified=None, metadata=<factory>, last_validated=None, cached=False)

Bases: object

Raw content fetched from somewhere, plus enough metadata to place it.

A Source is produced by a fetcher and carries the original bytes together with enough information to identify, validate, cache, and later transform it into one or more Document instances. Sources are independently cacheable, and their freshness may be validated against the originating system using ETags, modification timestamps, content hashes, or equivalent mechanisms.

Parameters:

• id (str)

• uri (str)

• content (bytes)

• content_type (str)

• content_hash (str)

• etag (str | None)

• last_modified (datetime | None)

• metadata (dict[str, Any])

• last_validated (datetime | None)

• cached (bool)

id

Unique identifier for this source.

Type:

str

uri

Where the content came from.

Type:

str

content

The raw fetched bytes.

Type:

bytes

content_type

The source’s MIME media type.

Type:

str

content_hash

Deterministic hash of content, used as its address.

Type:

str

etag

HTTP validator for conditional refetching, when the origin supports it.

Type:

str | None

last_modified

Origin-reported modification time, when available.

Type:

datetime.datetime | None

metadata

Extended, source-specific metadata.

Type:

dict[str, Any]

last_validated

When this source was last confirmed fresh against its origin (or, for one that has never been revalidated, first fetched). Persisted: round-tripped through the cache so that SourceStep.cache_validation_interval means the same thing across process restarts, not just within one. None on a Source that has never been through a cache write, e.g. one just produced by evaluate() and not yet cached.

Type:

datetime.datetime | None

cached

Whether this instance was served from the cache rather than freshly fetched. Runtime-only: never persisted.

Type:

bool