fastrag.benchmark.benchmark#

The Benchmark: run a dataset through a pipeline, score every sample.

Classes

Benchmark(*, dataset, metrics[, max_concurrent])

Runs every sample in a dataset through a pipeline and scores it.

class fastrag.benchmark.benchmark.Benchmark(*, dataset, metrics, max_concurrent=4)[source]#

Bases: object

Runs every sample in a dataset through a pipeline and scores it.

from fastrag.benchmark import Benchmark, QADataset
from fastrag.benchmark.metrics import Latency, NDCG, Recall

benchmark = Benchmark(
    dataset=QADataset.load("eval.json"),
    metrics=[Recall(k=5), NDCG(k=5), Latency()],
    max_concurrent=4,
)
result = await benchmark.run(engine, name="recursive-500")

Parameters:

dataset (QADataset) – The samples to run.
metrics (Sequence[Metric]) – What to score each sample on. Every name must be unique — a duplicate would silently overwrite the first in BenchmarkResult.summary — and at least one metric’s needs_relevant_ids must be satisfiable by the dataset.
max_concurrent (int) – How many samples run at once.

dataset: QADataset#

metrics: Sequence[Metric]#

max_concurrent: int = 4#

async run(target, *, name)[source]#

Run every sample against target, scoring it with every metric.

Samples run under a concurrency-limited semaphore, gathered back in dataset order. A sample whose pipeline call raises is recorded as "errored" with every score None — no metric runs for it — so a failure is never scored as a zero; see StatusCounts.

Parameters:

target (QueryEngine | Retriever) – A QueryEngine for metrics that need a generated answer, or a bare Retriever for retrieval-only metrics — cheaper, and the signal to reach for first (see fastrag.benchmark.metrics.retrieval).
name (str) – This run’s identity, carried on the returned result.

Raises:

ConfigurationError – target is a bare Retriever and a metric needs an answer. Raised before any sample runs, since it is knowable without running one.

Return type:

BenchmarkResult