fastrag.ingestion.steps.chunking.validation#

Independent validation of persisted structural output, without a database.

Functions

validate_structure_chunks(normalized, ...[, ...])

Validate split tuples or cached Documents against authoritative source.

Classes

ChunkValidation(chunk_count, ...[, ...])

Runtime diagnostics kept outside chunk metadata and identity.
class fastrag.ingestion.steps.chunking.validation.ChunkValidation(chunk_count, retained_characters, embedding_tokens, classification, validation_seconds=0.0, token_counts=(), merged_chunks=0, split_chunks=0, fallback_chunks=0, oversized_chunks=0, excluded_characters=0, normalization_diagnostics=(), cached_chunks=0, small_chunks=0, model_calls=0, model_cache_hits=0, model_fallbacks=(), operation_seconds=0.0)[source]#

Bases: object

Runtime diagnostics kept outside chunk metadata and identity.

Parameters:

  • chunk_count (int)

  • retained_characters (int)

  • embedding_tokens (int)

  • classification (str)

  • validation_seconds (float)

  • token_counts (tuple[int, ...])

  • merged_chunks (int)

  • split_chunks (int)

  • fallback_chunks (int)

  • oversized_chunks (int)

  • excluded_characters (int)

  • normalization_diagnostics (tuple[str, ...])

  • cached_chunks (int)

  • small_chunks (int)

  • model_calls (int)

  • model_cache_hits (int)

  • model_fallbacks (tuple[str, ...])

  • operation_seconds (float)

fastrag.ingestion.steps.chunking.validation.validate_structure_chunks(normalized, pieces, tokenizer, max_tokens, max_overlap_tokens=50, min_preferred_tokens=100)[source]#

Validate split tuples or cached Documents against authoritative source.

Raises ChunkingError for invalid persisted metadata. Counts are recomputed; neither recorded budgets nor node IDs substitute for interval coverage.

Parameters:
Return type:

ChunkValidation