fastrag.benchmark.metrics.operational#

Operational metrics: cost of running the pipeline, not quality of its output.

TokenUsage and Cost return None for every sample against a bare Retriever or a provider that reports no usage — “not measurable for this sample” (score()), not a zero, and it is why a retrieval-only benchmark should not list them: every sample would come back incomplete.

Classes

`Cost`(*, input_price, output_price)	Dollar cost of one sample's generation call.
`Latency`()	Wall-clock time for one sample, in milliseconds.
`TokenUsage`()	Input plus output tokens for one sample.

class fastrag.benchmark.metrics.operational.Latency[source]#

Bases: Metric

Wall-clock time for one sample, in milliseconds.

Never None: Benchmark times every sample itself, so this metric works against a bare Retriever too, with no chat model involved.

name = 'latency_ms'#

higher_is_better: bool = False#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

async score(sample, outcome)[source]#

Return the sample’s wall-clock latency in milliseconds.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

needs_answer: bool = False#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = False#: True when score() reads sample.relevant_ids.

class fastrag.benchmark.metrics.operational.TokenUsage[source]#

Bases: Metric

Input plus output tokens for one sample.

None when either count is missing — a bare Retriever never generates, and ask() does not report usage (in the new query layer ChatModel.invoke returns plain text, not a Completion carrying token counts).

name = 'total_tokens'#

higher_is_better: bool = False#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

async score(sample, outcome)[source]#

Total tokens, or None when the provider reported no usage.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

needs_answer: bool = False#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = False#: True when score() reads sample.relevant_ids.

class fastrag.benchmark.metrics.operational.Cost(*, input_price, output_price)[source]#

Bases: Metric

Dollar cost of one sample’s generation call.

Parameters:

input_price (float) – Price per 1,000,000 input tokens, in USD.
output_price (float) – Price per 1,000,000 output tokens, in USD.

input_price: float#

output_price: float#

name = 'cost_usd'#

higher_is_better: bool = False#: Which direction improves. False for latency_ms, total_tokens, and cost_usd.

async score(sample, outcome)[source]#

Dollar cost from token counts, or None when usage is unreported.

Parameters:

sample (EvalSample)
outcome (QueryOutcome)

Return type:

float | None

aggregate(scores)#

Aggregate this metric’s per-sample scores. Override to change how.

Parameters:: scores (Sequence[float | None])
Return type:: DistributionSummary

needs_answer: bool = False#: True when score() reads outcome.answer — never satisfied by a bare Retriever.

needs_relevant_ids: bool = False#: True when score() reads sample.relevant_ids.