fastrag.ingestion.steps.chunking.embedding_semantic#

Embedding-distance sentence boundaries over exact source slices.

Classes

EmbeddingSemanticChunker(*, embeddings[, ...])

Select sentence cuts with an existing asynchronous SDK Embedder.
class fastrag.ingestion.steps.chunking.embedding_semantic.EmbeddingSemanticChunker(*, embeddings, buffer_size=1, breakpoint_threshold_type='percentile', breakpoint_threshold_amount=None, number_of_chunks=None, sentence_split_regex='(?<=[.?!])\\s+', min_chunk_size=None, add_start_index=False, batch_size=64, cache=None, policy=None)[source]#

Bases: Chunker

Select sentence cuts with an existing asynchronous SDK Embedder.

Threshold modes mirror LangChain Experimental statistics. Requested counts instead select the strongest cuts exactly, with earlier offsets winning ties. Minimum size is in source characters and may reduce that count. This is a sentence splitter, with no syntax awareness or hard token budget. Both direct and pipeline outputs preserve all original whitespace and source order.

Parameters:

  • embeddings (Embedder)

  • buffer_size (int)

  • breakpoint_threshold_type (Literal['percentile', 'standard_deviation', 'interquartile', 'gradient'])

  • breakpoint_threshold_amount (float | None)

  • number_of_chunks (int | None)

  • sentence_split_regex (str)

  • min_chunk_size (int | None)

  • add_start_index (bool)

  • batch_size (int)

  • cache (CacheStore | Literal[False] | None)

  • policy (Policy | None)

accepted_content_formats: frozenset[str] | None = frozenset({'markdown', 'text'})#

None preserves permissive custom subclasses; sets may include custom strings.

config()[source]#

Return strategy settings and opaque nonsecret provider identity.

Return type:

Mapping[str, Any]

split(document)[source]#

Require asynchronous model evaluation before yielding any pieces.

Parameters:

document (Document)

Return type:

Iterator[str | tuple[str, Mapping[str, Any]]]

async asplit(document)[source]#

Embed neighboring source sentences, then slice at validated offsets.

Parameters:

document (Document)

Return type:

AsyncIterator[str | tuple[str, Mapping[str, Any]]]

async transform(item, context)[source]#

Use base-owned metadata/identity while retaining exact source whitespace.

Parameters:
Return type:

AsyncIterator[Document]

async validate_output(item, produced, context)[source]#

Restore source order after stores replay chunks in arbitrary ID order.

Return type:

None

property cache: CacheStore | Literal[False] | None#

Return the cache explicitly configured for this step.

property config_hash: str#

A stable hash of this step’s class name and config().

Two identically configured steps of the same class hash equal; changing any declared config value changes the hash; the hash is stable across processes. Used as part of every cache id this step’s evaluation reads or writes.

content_format: ContentFormat | None = None#

Overrides the parent document’s content_format in every chunk this chunker produces. None — the default — changes nothing: metadata inheritance in _build_chunk already carries the parent’s content_format (and everything else in its metadata) forward unchanged. Set this on a chunker whose output format differs from its input’s (e.g. one that flattens a DOCLING document into Markdown chunks) so the chunks it produces describe themselves truthfully.

effective_policy(context)#

Resolve this step’s policy over the one inherited from context.

The step’s own explicitly-set fields take priority; everything else falls through to the pipeline (or enclosing router/aggregator) policy.

Parameters:

context (ExecutionContext)

Return type:

Policy

async evaluate(context)#

Drive transform() over each upstream item, cached per item.

Raises:

fastrag.exceptions.ConfigurationError – No upstream is bound — call step(upstream) before evaluating.

Parameters:

context (ExecutionContext)

Return type:

AsyncIterator[Document]

async evaluate_items(items, context)#

Transform a bounded cache window independently of worker concurrency.

Parameters:
Return type:

AsyncIterator[Document]

event_name: str = 'Chunking'#

The stage this step’s work belongs to, e.g. "Chunking".

Declared as a plain class attribute by a family base class, and inherited by every concrete step in that family:

class Chunker(DocumentStep, ABC):
    event_name = "Chunking"


class ChunkerRouter(RoutedStep[Document], Chunker):
    ...  # -> "Chunking", inherited from `Chunker`

A concrete step overrides it to opt out of its family’s label:

class ChunkerRouter(RoutedStep[Document], Chunker):
    event_name = "ChunkerRouter"  # -> "ChunkerRouter"

Resolution walks the MRO for the first class that declares it in its own class body (so a mixin listed first, e.g. RoutedStep, contributes nothing unless it declares one itself), and falls back to the class’s own name when nothing in the MRO does — a step family that never declares an event_name still labels each of its steps distinctly rather than collapsing them under one blank heading. That fallback is computed per instance rather than assigned onto the class: assigning it would make it look “declared” to every subclass, which would then inherit their base’s name instead of getting their own.

property policy: Policy | None#

Return the policy explicitly configured for this step.

replay_needs_content: bool = False#

Set by a step whose cached replay reads the full documents anyway (e.g. to re-validate them): its cache reads then skip the header-only projection rather than fetching every entry twice.

async run()#

Execute this step independently and collect every item.

Builds a default ExecutionContext when none is installed (this step’s own policy, or a fresh Policy). Intended primarily for testing and standalone use — pipeline-wide event observation is Pipeline’s job.

Return type:

list[T]

property upstream: PipelineStep[Any] | None#

Return the upstream step bound via __call__, or None.

Read-only counterpart to __call__’s self._upstream assignment — lets external code (e.g. Pipeline, walking a graph from its root to classify each step by position) introspect composition without reaching into a private attribute.

validate_input(item)#

Reject incompatible representations before splitting or cache replay.

Parameters:

item (Document)

Return type:

None

name: str#