fastrag.benchmark.metrics.generation#

Generation metrics: a free bag-of-words baseline, and three judge-backed scores.

Classes

`AnswerRelevancy`(*, judge)	How relevant the answer is to the question asked, per the judge.
`Correctness`(*, judge)	How correct the answer is against the sample's expected answers, per the judge.
`Faithfulness`(*, judge)	How well the answer is supported by its retrieved sources, per the judge.
`JudgeMetric`(*, judge)	Shared machinery for a metric scored by asking a chat model to grade.
`TokenOverlap`()	Bag-of-words F1 between the answer and the best-matching expected answer.

class fastrag.benchmark.metrics.generation.TokenOverlap[source]#

Bases: Metric

Bag-of-words F1 between the answer and the best-matching expected answer.

A cheap smoke-test baseline, never accuracy: it rewards word overlap, not correctness, and a fluent answer that restates the question can score well while being wrong. Reach for the retrieval metrics or a judge-backed metric when the number needs to mean something.

name = 'token_overlap'#

needs_answer: bool = True#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

async score(sample, outcome)[source]#

Best F1 across expected answers, or None when there are none.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

needs_relevant_ids: bool = False#: True when score() reads sample.relevant_ids.

class fastrag.benchmark.metrics.generation.JudgeMetric(*, judge)[source]#

Bases: Metric

Shared machinery for a metric scored by asking a chat model to grade.

Parameters:: judge (ChatModel) – The chat model asked to grade each sample.

judge: ChatModel#

needs_answer: bool = True#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

abstract property name: str#: Key this metric occupies in BenchmarkResult.summary.

needs_relevant_ids: bool = False#: True when score() reads sample.relevant_ids.

abstractmethod async score(sample, outcome)#

This sample’s score, or None when it is not measurable.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

class fastrag.benchmark.metrics.generation.Faithfulness(*, judge)[source]#

Bases: JudgeMetric

How well the answer is supported by its retrieved sources, per the judge.

Parameters:: judge (ChatModel)

name = 'faithfulness'#

async score(sample, outcome)[source]#

Ask the judge to score how well the answer is grounded in its sources.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

needs_answer: bool = True#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = False#: True when score() reads sample.relevant_ids.

judge: ChatModel#

class fastrag.benchmark.metrics.generation.AnswerRelevancy(*, judge)[source]#

Bases: JudgeMetric

How relevant the answer is to the question asked, per the judge.

Parameters:: judge (ChatModel)

name = 'answer_relevancy'#

async score(sample, outcome)[source]#

Ask the judge to score how relevant the answer is to the question.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

needs_answer: bool = True#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = False#: True when score() reads sample.relevant_ids.

judge: ChatModel#

class fastrag.benchmark.metrics.generation.Correctness(*, judge)[source]#

Bases: JudgeMetric

How correct the answer is against the sample’s expected answers, per the judge.

Parameters:: judge (ChatModel)

name = 'correctness'#

async score(sample, outcome)[source]#

Ask the judge to score correctness, or None with no expected answers.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

higher_is_better: bool = True#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

needs_answer: bool = True#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = False#: True when score() reads sample.relevant_ids.

judge: ChatModel#