fastrag.benchmark#

Benchmarking: load a dataset, score a pipeline against it, compare runs.

class fastrag.benchmark.Benchmark(*, dataset, metrics, max_concurrent=4)[source]#

Bases: object

Runs every sample in a dataset through a pipeline and scores it.

from fastrag.benchmark import Benchmark, QADataset
from fastrag.benchmark.metrics import Latency, NDCG, Recall

benchmark = Benchmark(
    dataset=QADataset.load("eval.json"),
    metrics=[Recall(k=5), NDCG(k=5), Latency()],
    max_concurrent=4,
)
result = await benchmark.run(engine, name="recursive-500")

Parameters:

dataset (QADataset) – The samples to run.
metrics (Sequence[Metric]) – What to score each sample on. Every name must be unique — a duplicate would silently overwrite the first in BenchmarkResult.summary — and at least one metric’s needs_relevant_ids must be satisfiable by the dataset.
max_concurrent (int) – How many samples run at once.

dataset: QADataset#

metrics: Sequence[Metric]#

max_concurrent: int = 4#

async run(target, *, name)[source]#

Run every sample against target, scoring it with every metric.

Samples run under a concurrency-limited semaphore, gathered back in dataset order. A sample whose pipeline call raises is recorded as "errored" with every score None — no metric runs for it — so a failure is never scored as a zero; see StatusCounts.

Parameters:

target (QueryEngine | Retriever) – A QueryEngine for metrics that need a generated answer, or a bare Retriever for retrieval-only metrics — cheaper, and the signal to reach for first (see fastrag.benchmark.metrics.retrieval).
name (str) – This run’s identity, carried on the returned result.

Raises:

ConfigurationError – target is a bare Retriever and a metric needs an answer. Raised before any sample runs, since it is knowable without running one.

Return type:

BenchmarkResult

class fastrag.benchmark.BenchmarkResult(name, created_at, status, summary, samples, duration_s, higher_is_better=<factory>)[source]#

Bases: object

The outcome of one run().

name is the only identity a result carries — deliberately no serialized pipeline configuration, since the benchmark script is already the readable, version-controlled record of what was run. Keep the script and its results/ directory together, and name results after their configuration.

Parameters:

name (str)
created_at (datetime)
status (StatusCounts)
summary (dict[str, DistributionSummary])
samples (list[SampleResult])
duration_s (float)
higher_is_better (dict[str, bool])

name: str#

created_at: datetime#

status: StatusCounts#

summary: dict[str, DistributionSummary]#

samples: list[SampleResult]#

duration_s: float#

higher_is_better: dict[str, bool]#: Per metric name, whether a higher score is better. Read by best(), which is the only place a result file’s meaning — not just its numbers — is needed after the Metric objects that ran are gone.

save(path)[source]#

Write this result as JSON, with created_at as ISO-8601.

Parameters:: path (str | Path)
Return type:: None

classmethod load(path)[source]#

Read a result written by save().

Raises:: ConfigurationError – path is not a result file this version of the package can read.
Parameters:: path (str | Path)
Return type:: BenchmarkResult

class fastrag.benchmark.ColumnMapping(question=None, expected_answers=None, relevant_ids=None)[source]#

Bases: object

Override auto-detection for one or more of QADataset’s columns.

Each field left None is auto-detected by scanning its alias list against the loaded records’ keys; passing a value overrides only that field. Auto-detection raises ConfigurationError when two alias columns are both present, rather than silently picking one.

relevant_ids’s alias list includes contexts for compatibility with RAGAS-style eval sets — but a RAGAS contexts column holds retrieved text, not ids. Such a column matches nothing in Fastrag’s chunk ids and scores every retrieval metric 0.0: pass ColumnMapping(relevant_ids=...) pointing at an actual id column, or leave relevant_ids empty and skip the retrieval metrics for that dataset.

Parameters:

question (str | None)
expected_answers (str | None)
relevant_ids (str | None)

question: str | None = None#

expected_answers: str | None = None#

relevant_ids: str | None = None#

class fastrag.benchmark.Comparison(results)[source]#

Bases: object

Any number of BenchmarkResult, compared.

from fastrag.benchmark import Comparison

comparison = Comparison.load("results/*.json")  # or Comparison.of([r1, r2, r3])
print(comparison.to_markdown())
print(comparison.best("ndcg@5"))

Parameters:: results (list[BenchmarkResult])

results: list[BenchmarkResult]#

classmethod of(results)[source]#

Build a comparison from results already in memory.

Parameters:: results (Iterable[BenchmarkResult])
Return type:: Comparison

classmethod load(pattern)[source]#

Load every result file matching a glob pattern, sorted by path.

Raises:: ConfigurationError – pattern matched no files — an empty comparison is the silent-wrong-answer shape, so this raises rather than returning one.
Parameters:: pattern (str | Path)
Return type:: Comparison

metrics()[source]#

Every metric name across every result, first-seen order.

Return type:: list[str]

to_markdown(stat='mean')[source]#

Render one row per run, one column per metric, stat of each.

stat is any DistributionSummary field ("mean", "p95", …); an unknown one raises ConfigurationError rather than printing zeros. A run with no such metric prints —. When stat is not "mean" every header carries it, e.g. latency_ms p95.

Parameters:: stat (str)
Return type:: str

best(metric, stat='mean')[source]#

The result whose metric is best, per that metric’s own direction.

Raises:

ConfigurationError – no result carries metric.

Parameters:

metric (str)
stat (str)

Return type:

BenchmarkResult

save(path)[source]#

Write to_markdown()’s default (mean) table to path.

Parameters:: path (str | Path)
Return type:: None

class fastrag.benchmark.EvalSample(question, expected_answers=<factory>, relevant_ids=<factory>, metadata=<factory>)[source]#

Bases: object

One question, with what a metric needs to score an answer to it.

relevant_ids is what enables the retrieval metrics (fastrag.benchmark.metrics.retrieval) — a sample with none is still usable for generation and operational metrics, just not those.

Parameters:

question (str)
expected_answers (list[str])
relevant_ids (list[str])
metadata (dict[str, Any])

question: str#

expected_answers: list[str]#

relevant_ids: list[str]#

metadata: dict[str, Any]#

class fastrag.benchmark.QADataset(samples)[source]#

Bases: object

A list of EvalSample, loadable from several formats.

from fastrag.benchmark import ColumnMapping, QADataset

QADataset.load("eval.csv")  # auto-detected
QADataset.load("eval.csv", ColumnMapping(question="user_query"))

Parameters:: samples (list[EvalSample])

samples: list[EvalSample]#

classmethod load(path, columns=None)[source]#

Load a dataset, dispatching on path’s suffix.

.json accepts either {"samples": [...]} (the native format, save()’s output) or a bare [...] of records. .jsonl reads one record per line. .csv reads with csv.DictReader.

.yaml/.parquet and from_huggingface() need the datasets extra — not yet implemented, kept behind the same gate this class already applies to its other deferred formats.

Raises:

ConfigurationError – path’s suffix is .yaml or .parquet (needs the datasets extra, not yet implemented), or is anything else this method does not read.

Parameters:

path (str | Path)
columns (ColumnMapping | None)

Return type:

QADataset

classmethod from_huggingface(path)[source]#

Load a dataset from Hugging Face Hub.

Not yet implemented — gated by the datasets extra, the same way the deferred .yaml/.parquet suffixes already are.

Raises:: ConfigurationError – Always — the datasets extra is not yet wired up, so this raises naming the install command rather than pretending a loading path exists.
Parameters:: path (str)
Return type:: QADataset

classmethod from_records(records)[source]#

Build a dataset from records already in memory, auto-detecting columns.

Parameters:: records (Iterable[dict[str, Any]])
Return type:: QADataset

save(path)[source]#

Write this dataset in the native {"samples": [...]} format.

Parameters:: path (str | Path)
Return type:: None

class fastrag.benchmark.SampleResult(question, status, scores, answer=None, retrieved_ids=<factory>, retrieved_uris=<factory>, latency_ms=0.0, error=None)[source]#

Bases: object

One sample’s outcome: its scores, and enough detail to diagnose them.

Parameters:

question (str)
status (Literal['successful', 'errored', 'incomplete'])
scores (dict[str, float | None])
answer (str | None)
retrieved_ids (list[str])
retrieved_uris (list[str])
latency_ms (float)
error (str | None)

question: str#

status: Literal['successful', 'errored', 'incomplete']#

scores: dict[str, float | None]#

answer: str | None = None#

retrieved_ids: list[str]#

retrieved_uris: list[str]#

latency_ms: float = 0.0#

error: str | None = None#

class fastrag.benchmark.StatusCounts(successful, errored, incomplete)[source]#

Bases: object

How a run’s samples broke down — never collapsed into one pass/fail count.

A question that errored is never scored as zero: it is tallied in errored, not folded into incomplete or successful.

Parameters:

successful (int)
errored (int)
incomplete (int)

successful: int#: Ran, and every metric produced a score.

errored: int#: The pipeline raised.

incomplete: int#: Ran, but at least one metric returned None.

Modules

`benchmark`	The `Benchmark`: run a dataset through a pipeline, score every sample.
`comparison`	The `Comparison`: any number of runs, side by side.
`datasets`	Evaluation datasets: a list of question/answer/relevant-id samples.
`metrics`	Metrics: an object per measurement, scored per sample, aggregated per run.
`results`	What one benchmark run produced, and how it is saved and reloaded.