fastrag.benchmark.datasets

Evaluation datasets: a list of question/answer/relevant-id samples.

Classes

ColumnMapping([question, expected_answers, ...])	Override auto-detection for one or more of QADataset's columns.
EvalSample(question[, expected_answers, ...])	One question, with what a metric needs to score an answer to it.
QADataset(samples)	A list of EvalSample, loadable from several formats.

class fastrag.benchmark.datasets.EvalSample(question, expected_answers=<factory>, relevant_ids=<factory>, metadata=<factory>)

Bases: object

One question, with what a metric needs to score an answer to it.

relevant_ids is what enables the retrieval metrics (fastrag.benchmark.metrics.retrieval) — a sample with none is still usable for generation and operational metrics, just not those.

Parameters:

• question (str)

• expected_answers (list[str])

• relevant_ids (list[str])

• metadata (dict[str, Any])

question: str

expected_answers: list[str]

relevant_ids: list[str]

metadata: dict[str, Any]

class fastrag.benchmark.datasets.ColumnMapping(question=None, expected_answers=None, relevant_ids=None)

Bases: object

Override auto-detection for one or more of QADataset’s columns.

Each field left None is auto-detected by scanning its alias list against the loaded records’ keys; passing a value overrides only that field. Auto-detection raises ConfigurationError when two alias columns are both present, rather than silently picking one.

relevant_ids’s alias list includes contexts for compatibility with RAGAS-style eval sets — but a RAGAS contexts column holds retrieved text, not ids. Such a column matches nothing in Fastrag’s chunk ids and scores every retrieval metric 0.0: pass ColumnMapping(relevant_ids=...) pointing at an actual id column, or leave relevant_ids empty and skip the retrieval metrics for that dataset.

Parameters:

• question (str | None)

• expected_answers (str | None)

• relevant_ids (str | None)

question: str | None = None

expected_answers: str | None = None

relevant_ids: str | None = None

class fastrag.benchmark.datasets.QADataset(samples)

Bases: object

A list of EvalSample, loadable from several formats.

from fastrag.benchmark import ColumnMapping, QADataset

QADataset.load("eval.csv")  # auto-detected
QADataset.load("eval.csv", ColumnMapping(question="user_query"))

Parameters:

samples (list[EvalSample])

samples: list[EvalSample]

classmethod load(path, columns=None)

Load a dataset, dispatching on path’s suffix.

.json accepts either {"samples": [...]} (the native format, save()’s output) or a bare [...] of records. .jsonl reads one record per line. .csv reads with csv.DictReader.

.yaml/.parquet and from_huggingface() need the datasets extra — not yet implemented, kept behind the same gate this class already applies to its other deferred formats.

Raises:

ConfigurationError – path’s suffix is .yaml or .parquet (needs the datasets extra, not yet implemented), or is anything else this method does not read.

Parameters:

• path (str | Path)

• columns (ColumnMapping | None)

Return type:

QADataset

classmethod from_huggingface(path)

Load a dataset from Hugging Face Hub.

Not yet implemented — gated by the datasets extra, the same way the deferred .yaml/.parquet suffixes already are.

Raises:

ConfigurationError – Always — the datasets extra is not yet wired up, so this raises naming the install command rather than pretending a loading path exists.

Parameters:

path (str)

Return type:

QADataset

classmethod from_records(records)

Build a dataset from records already in memory, auto-detecting columns.

Parameters:

records (Iterable[dict[str, Any]])

Return type:

QADataset

save(path)

Write this dataset in the native {"samples": [...]} format.

Parameters:

path (str | Path)

Return type:

None