fastrag.benchmark.comparison#

The Comparison: any number of runs, side by side.

Classes

Comparison(results)

Any number of BenchmarkResult, compared.
class fastrag.benchmark.comparison.Comparison(results)[source]#

Bases: object

Any number of BenchmarkResult, compared.

from fastrag.benchmark import Comparison

comparison = Comparison.load("results/*.json")  # or Comparison.of([r1, r2, r3])
print(comparison.to_markdown())
print(comparison.best("ndcg@5"))
Parameters:

results (list[BenchmarkResult])

results: list[BenchmarkResult]#
classmethod of(results)[source]#

Build a comparison from results already in memory.

Parameters:

results (Iterable[BenchmarkResult])

Return type:

Comparison

classmethod load(pattern)[source]#

Load every result file matching a glob pattern, sorted by path.

Raises:

ConfigurationError – pattern matched no files — an empty comparison is the silent-wrong-answer shape, so this raises rather than returning one.

Parameters:

pattern (str | Path)

Return type:

Comparison

metrics()[source]#

Every metric name across every result, first-seen order.

Return type:

list[str]

to_markdown(stat='mean')[source]#

Render one row per run, one column per metric, stat of each.

stat is any DistributionSummary field ("mean", "p95", …); an unknown one raises ConfigurationError rather than printing zeros. A run with no such metric prints —. When stat is not "mean" every header carries it, e.g. latency_ms p95.

Parameters:

stat (str)

Return type:

str

best(metric, stat='mean')[source]#

The result whose metric is best, per that metric’s own direction.

Raises:

ConfigurationError – no result carries metric.

Parameters:
Return type:

BenchmarkResult

save(path)[source]#

Write to_markdown()’s default (mean) table to path.

Parameters:

path (str | Path)

Return type:

None