Benchmarking#

Score a retrieval or query pipeline against a labelled QA dataset, and compare runs. fastrag.benchmark loads a QADataset, runs every sample through a Benchmark against a Retriever (retrieval metrics) or a QueryEngine (generation and operational metrics too), and puts several runs side by side with Comparison.

examples/16_benchmarking.py demonstrates a chunk-size sweep, scored and saved in a loop. Give each configuration its own collection when comparing runs: a cache fingerprint does not change a store’s collection automatically. The script opens <FASTRAG_COLLECTION>_recursive_<size> explicitly per configuration and uses its matching retriever. Runtime size/dataset/result overrides and the owned local smoke workflow are documented in Examples:

dataset = QADataset.load(dataset_path)

benchmark = Benchmark(
    dataset=dataset,
    # key="source_uri": examples/eval/agrospai.json's ground truth is
    # page URLs, not chunk ids — chunk ids depend on the chunker
    # configuration, so they change with every chunk_size in the sweep,
    # and id-keyed ground truth would score 0.0 for most runs and look
    # like a real finding.
    metrics=[
        Recall(k=5, key="source_uri"),
        Precision(k=5, key="source_uri"),
        MRR(k=5, key="source_uri"),
        NDCG(k=5, key="source_uri"),
        HitRate(k=5, key="source_uri"),
        Latency(),
        # TokenUsage and Cost are absent on purpose: no generation runs
        # in this phase, so both would return None for every sample and
        # mark the whole run "incomplete".
    ],
    max_concurrent=4,
)

results = []
for size in sizes:
    # Store identity, not a save fingerprint, isolates each configuration.
    async with MilvusVectorStore(
        url=cfg.milvus_uri, collection=f"{cfg.collection}_recursive_{size}"
    ) as vector_store:
        await build_index(
            size, cfg=cfg, cache=cache, vector_store=vector_store, embedder=embedder
        )
        print(f"collection={vector_store.collection} chunk_size={size} synced")
        retriever = VectorRetriever(vector_store=vector_store, embedder=embedder)
        result = await benchmark.run(retriever, name=f"recursive-{size}")
        require_success(result)
        result.save(results_dir / f"recursive-{size}.json")
        results.append(result)

metrics=[...] takes the typed Metric objects directly (Recall(k=5), Precision(k=5), …) — there is no string-shorthand form. Metrics read a retrieved document’s score/rank/source_uri from its metadata (the keys Retrieval’s VectorStore.search stamps).

Datasets#

QADataset and EvalSample model one question with its expected contexts and answers. from_records builds one from rows; loading .yaml / .parquet files and from_huggingface raise ConfigurationError naming pip install "fastrag-ai[datasets]" until that extra’s loaders ship (see BACKLOG.md).

Targets and metrics#

BenchmarkTarget = Retriever | QueryEngine. A bare Retriever is scored with retrieval metrics; a QueryEngine adds generation metrics (Faithfulness, AnswerRelevancy, Correctness, TokenOverlap) and operational ones (Latency, TokenUsage, Cost). A judge-backed metric refuses a bare-Retriever target.
Comparison merges several saved BenchmarkResults into one table.

Benchmarking#

Datasets#

Targets and metrics#

See also#