fastrag.benchmark.metrics.retrieval#

Retrieval metrics: deterministic, free, fast — reach for these first.

Classes

`HitRate`(*[, k, key])	`1.0` if any relevant document is in the window, else `0.0`.
`MRR`(*[, k, key])	`1 / (position of the first relevant)`, else `0.0`.
`NDCG`(*[, k, key])	Binary-gain NDCG, with the ideal ranking over `min(len(relevant), k)`.
`Precision`(*[, k, key])	`\|relevant ∩ retrieved\| / len(retrieved)`.
`Recall`(*[, k, key])	`\|relevant ∩ retrieved\| / \|relevant\|`.
`RetrievalMetric`(*[, k, key])	Shared machinery for the five ranking metrics below.

class fastrag.benchmark.metrics.retrieval.RetrievalMetric(*, k=5, key='id')[source]#

Bases: Metric

Shared machinery for the five ranking metrics below.

Parameters:

k (int) – How many of the retrieved documents to consider, best first.
key (Literal['id', 'source_uri']) – Which field identifies a document for comparison against sample.relevant_ids. "id" is the chunk id (sha256(source_uri, chunk_index, content)), which changes with every chunker configuration. "source_uri" compares at the page level, which survives a chunk-size sweep — pass it whenever the ground truth was authored per page rather than per chunk.

k: int = 5#

key: Literal['id', 'source_uri'] = 'id'#

needs_relevant_ids: bool = True#: True when score() reads sample.relevant_ids.

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

abstract property name: str#: Key this metric occupies in BenchmarkResult.summary.

needs_answer: bool = False#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

abstractmethod async score(sample, outcome)#

This sample’s score, or None when it is not measurable.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

class fastrag.benchmark.metrics.retrieval.Recall(*, k=5, key='id')[source]#

Bases: RetrievalMetric

|relevant ∩ retrieved| / |relevant|.

Parameters:

k (int)
key (Literal['id', 'source_uri'])

property name: str#: The summary key, e.g. "recall@5".

async score(sample, outcome)[source]#

Recall at k, or None when the sample has no relevant ids.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

k: int = 5#

key: Literal['id', 'source_uri'] = 'id'#

needs_answer: bool = False#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = True#: True when score() reads sample.relevant_ids.

class fastrag.benchmark.metrics.retrieval.Precision(*, k=5, key='id')[source]#

Bases: RetrievalMetric

|relevant ∩ retrieved| / len(retrieved).

Parameters:

k (int)
key (Literal['id', 'source_uri'])

property name: str#: The summary key, e.g. "precision@5".

async score(sample, outcome)[source]#

Precision at k, or None when the sample has no relevant ids.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

k: int = 5#

key: Literal['id', 'source_uri'] = 'id'#

needs_answer: bool = False#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = True#: True when score() reads sample.relevant_ids.

class fastrag.benchmark.metrics.retrieval.MRR(*, k=5, key='id')[source]#

Bases: RetrievalMetric

1 / (position of the first relevant), else 0.0.

Parameters:

k (int)
key (Literal['id', 'source_uri'])

property name: str#: The summary key, e.g. "mrr@5".

async score(sample, outcome)[source]#

Reciprocal rank of the first relevant hit, or None with no relevant ids.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

k: int = 5#

key: Literal['id', 'source_uri'] = 'id'#

needs_answer: bool = False#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = True#: True when score() reads sample.relevant_ids.

class fastrag.benchmark.metrics.retrieval.NDCG(*, k=5, key='id')[source]#

Bases: RetrievalMetric

Binary-gain NDCG, with the ideal ranking over min(len(relevant), k).

Parameters:

k (int)
key (Literal['id', 'source_uri'])

property name: str#: The summary key, e.g. "ndcg@5".

async score(sample, outcome)[source]#

Binary-gain NDCG at k, or None when the sample has no relevant ids.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

k: int = 5#

key: Literal['id', 'source_uri'] = 'id'#

needs_answer: bool = False#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = True#: True when score() reads sample.relevant_ids.

class fastrag.benchmark.metrics.retrieval.HitRate(*, k=5, key='id')[source]#

Bases: RetrievalMetric

1.0 if any relevant document is in the window, else 0.0.

Parameters:

k (int)
key (Literal['id', 'source_uri'])

property name: str#: The summary key, e.g. "hit_rate@5".

async score(sample, outcome)[source]#

1.0 if any relevant document is in the window, else 0.0.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

k: int = 5#

key: Literal['id', 'source_uri'] = 'id'#

needs_answer: bool = False#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = True#: True when score() reads sample.relevant_ids.