fastrag.benchmark.metrics#

Metrics: an object per measurement, scored per sample, aggregated per run.

class fastrag.benchmark.metrics.MRR(*, k=5, key='id')[source]#

Bases: RetrievalMetric

1 / (position of the first relevant), else 0.0.

Parameters:

k (int)
key (Literal['id', 'source_uri'])

property name: str#: The summary key, e.g. "mrr@5".

async score(sample, outcome)[source]#

Reciprocal rank of the first relevant hit, or None with no relevant ids.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

k: int = 5#

key: Literal['id', 'source_uri'] = 'id'#

needs_answer: bool = False#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = True#: True when score() reads sample.relevant_ids.

class fastrag.benchmark.metrics.NDCG(*, k=5, key='id')[source]#

Bases: RetrievalMetric

Binary-gain NDCG, with the ideal ranking over min(len(relevant), k).

Parameters:

k (int)
key (Literal['id', 'source_uri'])

property name: str#: The summary key, e.g. "ndcg@5".

async score(sample, outcome)[source]#

Binary-gain NDCG at k, or None when the sample has no relevant ids.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

k: int = 5#

key: Literal['id', 'source_uri'] = 'id'#

needs_answer: bool = False#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = True#: True when score() reads sample.relevant_ids.

class fastrag.benchmark.metrics.AnswerRelevancy(*, judge)[source]#

Bases: JudgeMetric

How relevant the answer is to the question asked, per the judge.

Parameters:: judge (ChatModel)

name = 'answer_relevancy'#

async score(sample, outcome)[source]#

Ask the judge to score how relevant the answer is to the question.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

needs_answer: bool = True#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = False#: True when score() reads sample.relevant_ids.

judge: ChatModel#

class fastrag.benchmark.metrics.Correctness(*, judge)[source]#

Bases: JudgeMetric

How correct the answer is against the sample’s expected answers, per the judge.

Parameters:: judge (ChatModel)

name = 'correctness'#

async score(sample, outcome)[source]#

Ask the judge to score correctness, or None with no expected answers.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

needs_answer: bool = True#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = False#: True when score() reads sample.relevant_ids.

judge: ChatModel#

class fastrag.benchmark.metrics.Cost(*, input_price, output_price)[source]#

Bases: Metric

Dollar cost of one sample’s generation call.

Parameters:

input_price (float) – Price per 1,000,000 input tokens, in USD.
output_price (float) – Price per 1,000,000 output tokens, in USD.

input_price: float#

output_price: float#

name = 'cost_usd'#

higher_is_better: bool = False#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

async score(sample, outcome)[source]#

Dollar cost from token counts, or None when usage is unreported.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

needs_answer: bool = False#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = False#: True when score() reads sample.relevant_ids.

class fastrag.benchmark.metrics.DistributionSummary(count, missing, mean, median, std, min, max, percentiles=<factory>)[source]#

Bases: object

A run’s scores for one metric, never collapsed to a single mean.

missing counts the None scores, dropped before mean/median/ std/min/max/percentiles are computed over what is left. An empty or all-None input yields count=0 and zeros everywhere, never a ZeroDivisionError.

Parameters:

count (int)
missing (int)
mean (float)
median (float)
std (float)
min (float)
max (float)
percentiles (dict[str, float])

count: int#

missing: int#

mean: float#

median: float#

std: float#

min: float#

max: float#

percentiles: dict[str, float]#

classmethod of(scores)[source]#

Build a DistributionSummary from raw per-sample scores.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

class fastrag.benchmark.metrics.Faithfulness(*, judge)[source]#

Bases: JudgeMetric

How well the answer is supported by its retrieved sources, per the judge.

Parameters:: judge (ChatModel)

name = 'faithfulness'#

async score(sample, outcome)[source]#

Ask the judge to score how well the answer is grounded in its sources.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

needs_answer: bool = True#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = False#: True when score() reads sample.relevant_ids.

judge: ChatModel#

class fastrag.benchmark.metrics.HitRate(*, k=5, key='id')[source]#

Bases: RetrievalMetric

1.0 if any relevant document is in the window, else 0.0.

Parameters:

k (int)
key (Literal['id', 'source_uri'])

property name: str#: The summary key, e.g. "hit_rate@5".

async score(sample, outcome)[source]#

1.0 if any relevant document is in the window, else 0.0.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

k: int = 5#

key: Literal['id', 'source_uri'] = 'id'#

needs_answer: bool = False#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = True#: True when score() reads sample.relevant_ids.

class fastrag.benchmark.metrics.JudgeMetric(*, judge)[source]#

Bases: Metric

Shared machinery for a metric scored by asking a chat model to grade.

Parameters:: judge (ChatModel) – The chat model asked to grade each sample.

judge: ChatModel#

needs_answer: bool = True#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

abstract property name: str#: Key this metric occupies in BenchmarkResult.summary.

needs_relevant_ids: bool = False#: True when score() reads sample.relevant_ids.

abstractmethod async score(sample, outcome)#

This sample’s score, or None when it is not measurable.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

class fastrag.benchmark.metrics.Latency[source]#

Bases: Metric

Wall-clock time for one sample, in milliseconds.

Never None: Benchmark times every sample itself, so this metric works against a bare Retriever too, with no chat model involved.

name = 'latency_ms'#

higher_is_better: bool = False#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

async score(sample, outcome)[source]#

Return the sample’s wall-clock latency in milliseconds.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

needs_answer: bool = False#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = False#: True when score() reads sample.relevant_ids.

class fastrag.benchmark.metrics.Metric[source]#

Bases: ABC

One measurement, computed per sample and aggregated over a run.

A metric carries its own dependencies as class attributes, so Benchmark can refuse an unsatisfiable request at construction rather than silently returning a number that means nothing.

needs_relevant_ids: bool = False#: True when score() reads sample.relevant_ids.

needs_answer: bool = False#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

abstract property name: str#: Key this metric occupies in BenchmarkResult.summary.

abstractmethod async score(sample, outcome)[source]#

This sample’s score, or None when it is not measurable.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)[source]#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

class fastrag.benchmark.metrics.Precision(*, k=5, key='id')[source]#

Bases: RetrievalMetric

|relevant ∩ retrieved| / len(retrieved).

Parameters:

k (int)
key (Literal['id', 'source_uri'])

property name: str#: The summary key, e.g. "precision@5".

async score(sample, outcome)[source]#

Precision at k, or None when the sample has no relevant ids.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

k: int = 5#

key: Literal['id', 'source_uri'] = 'id'#

needs_answer: bool = False#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = True#: True when score() reads sample.relevant_ids.

class fastrag.benchmark.metrics.Recall(*, k=5, key='id')[source]#

Bases: RetrievalMetric

|relevant ∩ retrieved| / |relevant|.

Parameters:

k (int)
key (Literal['id', 'source_uri'])

property name: str#: The summary key, e.g. "recall@5".

async score(sample, outcome)[source]#

Recall at k, or None when the sample has no relevant ids.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

k: int = 5#

key: Literal['id', 'source_uri'] = 'id'#

needs_answer: bool = False#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = True#: True when score() reads sample.relevant_ids.

class fastrag.benchmark.metrics.RetrievalMetric(*, k=5, key='id')[source]#

Bases: Metric

Shared machinery for the five ranking metrics below.

Parameters:

k (int) – How many of the retrieved documents to consider, best first.
key (Literal['id', 'source_uri']) – Which field identifies a document for comparison against sample.relevant_ids. "id" is the chunk id (sha256(source_uri, chunk_index, content)), which changes with every chunker configuration. "source_uri" compares at the page level, which survives a chunk-size sweep — pass it whenever the ground truth was authored per page rather than per chunk.

k: int = 5#

key: Literal['id', 'source_uri'] = 'id'#

needs_relevant_ids: bool = True#: True when score() reads sample.relevant_ids.

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

abstract property name: str#: Key this metric occupies in BenchmarkResult.summary.

needs_answer: bool = False#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

abstractmethod async score(sample, outcome)#

This sample’s score, or None when it is not measurable.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

class fastrag.benchmark.metrics.TokenOverlap[source]#

Bases: Metric

Bag-of-words F1 between the answer and the best-matching expected answer.

A cheap smoke-test baseline, never accuracy: it rewards word overlap, not correctness, and a fluent answer that restates the question can score well while being wrong. Reach for the retrieval metrics or a judge-backed metric when the number needs to mean something.

name = 'token_overlap'#

needs_answer: bool = True#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

async score(sample, outcome)[source]#

Best F1 across expected answers, or None when there are none.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

needs_relevant_ids: bool = False#: True when score() reads sample.relevant_ids.

class fastrag.benchmark.metrics.TokenUsage[source]#

Bases: Metric

Input plus output tokens for one sample.

None when either count is missing — a bare Retriever never generates, and ask() does not report usage (in the new query layer ChatModel.invoke returns plain text, not a Completion carrying token counts).

name = 'total_tokens'#

higher_is_better: bool = False#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

async score(sample, outcome)[source]#

Total tokens, or None when the provider reported no usage.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

needs_answer: bool = False#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = False#: True when score() reads sample.relevant_ids.

Modules

`base`	The `Metric` contract, its input, and how scores aggregate.
`generation`	Generation metrics: a free bag-of-words baseline, and three judge-backed scores.
`operational`	Operational metrics: cost of running the pipeline, not quality of its output.
`retrieval`	Retrieval metrics: deterministic, free, fast — reach for these first.