fastrag.benchmark.results#

What one benchmark run produced, and how it is saved and reloaded.

Classes

BenchmarkResult(name, created_at, status, ...)

The outcome of one run().

SampleResult(question, status, scores[, ...])

One sample's outcome: its scores, and enough detail to diagnose them.

StatusCounts(successful, errored, incomplete)

How a run's samples broke down — never collapsed into one pass/fail count.
class fastrag.benchmark.results.StatusCounts(successful, errored, incomplete)[source]#

Bases: object

How a run’s samples broke down — never collapsed into one pass/fail count.

A question that errored is never scored as zero: it is tallied in errored, not folded into incomplete or successful.

Parameters:

  • successful (int)

  • errored (int)

  • incomplete (int)

successful: int#

Ran, and every metric produced a score.

errored: int#

The pipeline raised.

incomplete: int#

Ran, but at least one metric returned None.

class fastrag.benchmark.results.SampleResult(question, status, scores, answer=None, retrieved_ids=<factory>, retrieved_uris=<factory>, latency_ms=0.0, error=None)[source]#

Bases: object

One sample’s outcome: its scores, and enough detail to diagnose them.

Parameters:
question: str#
status: Literal['successful', 'errored', 'incomplete']#
scores: dict[str, float | None]#
answer: str | None = None#
retrieved_ids: list[str]#
retrieved_uris: list[str]#
latency_ms: float = 0.0#
error: str | None = None#
class fastrag.benchmark.results.BenchmarkResult(name, created_at, status, summary, samples, duration_s, higher_is_better=<factory>)[source]#

Bases: object

The outcome of one run().

name is the only identity a result carries — deliberately no serialized pipeline configuration, since the benchmark script is already the readable, version-controlled record of what was run. Keep the script and its results/ directory together, and name results after their configuration.

Parameters:
name: str#
created_at: datetime#
status: StatusCounts#
summary: dict[str, DistributionSummary]#
samples: list[SampleResult]#
duration_s: float#
higher_is_better: dict[str, bool]#

Per metric name, whether a higher score is better. Read by best(), which is the only place a result file’s meaning — not just its numbers — is needed after the Metric objects that ran are gone.

save(path)[source]#

Write this result as JSON, with created_at as ISO-8601.

Parameters:

path (str | Path)

Return type:

None

classmethod load(path)[source]#

Read a result written by save().

Raises:

ConfigurationError – path is not a result file this version of the package can read.

Parameters:

path (str | Path)

Return type:

BenchmarkResult