fastrag.benchmark.metrics.base

The Metric contract, its input, and how scores aggregate.

Metrics consume the retrieval and query layers: Retriever returns Document — there is no ScoredDocument any more — so a metric reads a document’s score/rank out of its metadata dict (stamped by VectorStore.search) instead of off its .score/ .rank attributes.

Classes

DistributionSummary(count, missing, mean, ...)	A run's scores for one metric, never collapsed to a single mean.
Metric()	One measurement, computed per sample and aggregated over a run.
QueryOutcome(answer, sources, latency_ms[, ...])	What the pipeline produced for one sample — a metric's whole input.

class fastrag.benchmark.metrics.base.QueryOutcome(answer, sources, latency_ms, input_tokens=None, output_tokens=None)

Bases: object

What the pipeline produced for one sample — a metric’s whole input.

answer is "" and input_tokens / output_tokens are None when run() targets a bare Retriever rather than a QueryEngine.

Parameters:

• answer (str)

• sources (list[Document])

• latency_ms (float)

• input_tokens (int | None)

• output_tokens (int | None)

answer: str

sources: list[Document]

latency_ms: float

input_tokens: int | None = None

output_tokens: int | None = None

class fastrag.benchmark.metrics.base.Metric

Bases: ABC

One measurement, computed per sample and aggregated over a run.

A metric carries its own dependencies as class attributes, so Benchmark can refuse an unsatisfiable request at construction rather than silently returning a number that means nothing.

needs_relevant_ids: bool = False

True when score() reads sample.relevant_ids.

needs_answer: bool = False

True when score() reads outcome.answer — never satisfied by a bare Retriever.

higher_is_better: bool = True

Which direction improves. False for latency_ms, total_tokens, and cost_usd.

abstract property name: str

Key this metric occupies in BenchmarkResult.summary.

abstractmethod async score(sample, outcome)

This sample’s score, or None when it is not measurable.

Parameters:

• sample (EvalSample)

• outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:

scores (Sequence[float | None])

Return type:

DistributionSummary

class fastrag.benchmark.metrics.base.DistributionSummary(count, missing, mean, median, std, min, max, percentiles=<factory>)

Bases: object

A run’s scores for one metric, never collapsed to a single mean.

missing counts the None scores, dropped before mean/median/ std/min/max/percentiles are computed over what is left. An empty or all-None input yields count=0 and zeros everywhere, never a ZeroDivisionError.

Parameters:

• count (int)

• missing (int)

• mean (float)

• median (float)

• std (float)

• min (float)

• max (float)

• percentiles (dict[str, float])

count: int

missing: int

mean: float

median: float

std: float

min: float

max: float

percentiles: dict[str, float]

classmethod of(scores)

Build a DistributionSummary from raw per-sample scores.

Parameters:

scores (Sequence[float | None])

Return type:

DistributionSummary