fastrag.retrieval.language#

Retrieval that prefers documents in the language of the question.

A multilingual site holds each page once per language, and a multilingual embedder and reranker score a page’s translations about equally. So the top results for a question are often one passage repeated in several languages, and the answer cites a translation in a language the reader did not write in. LanguageAwareRetriever keeps one translation of each page, the one in the question’s language when it was retrieved, and ranks documents in other languages below documents of similar relevance in the question’s.

Classes

LanguageAwareRetriever(*, retriever, detector)

Prefer documents in the question's language, one translation per page.

LanguageDetector()

Names the language each text is written in.

LinguaLanguageDetector(languages, *[, ...])

Detects languages offline with Lingua, choosing among a corpus's own.
class fastrag.retrieval.language.LanguageDetector[source]#

Bases: ABC

Names the language each text is written in.

languages#

The ISO 639-1 codes detect() can return. A URL path segment naming one of them is read as a locale (see LanguageAwareRetriever).

Type:

frozenset[str]

languages: frozenset[str]#
abstractmethod detect(texts)[source]#

Return the ISO 639-1 code of each text’s language, or None if unsure.

Blocking: LanguageAwareRetriever runs it in a worker thread.

Parameters:

texts (Sequence[str])

Return type:

list[str | None]

class fastrag.retrieval.language.LinguaLanguageDetector(languages, *, minimum_relative_distance=0.0)[source]#

Bases: LanguageDetector

Detects languages offline with Lingua, choosing among a corpus’s own.

Needs the “language” extra (lingua-language-detector). Naming only the languages the corpus is written in keeps short questions reliable: a short Catalan question is told apart from Spanish far more often when those are among a handful of candidates rather than 75.

detector = LinguaLanguageDetector(["en", "es", "ca", "zh"])
detector.detect(["How can I obtain a credential?"])  # ["en"]
Parameters:

  • languages (frozenset[str]) – ISO 639-1 codes of the languages to choose among, at least two.

  • minimum_relative_distance (float) – How far ahead of the runner-up the most likely language must be, from 0.0 to 0.99; text that does not clear it gets None. 0.0 always names a language.

languages: frozenset[str]#
detect(texts)[source]#

Detect every text’s language, in parallel.

Parameters:

texts (Sequence[str])

Return type:

list[str | None]

class fastrag.retrieval.language.LanguageAwareRetriever(*, retriever, detector, candidate_k=40, other_language_weight=0.8)[source]#

Bases: Retriever

Prefer documents in the question’s language, one translation per page.

Over-fetches candidate_k documents from retriever, best first, and detects the language of the question and of each document (a document’s own metadata["language"] wins over detection). Then:

  1. Translations collapse. Documents whose source_uri differs only by a leading locale path segment naming one of detector.languages (/docs/a, /en/docs/a, /ca/docs/a) are copies of one page, and only one URL’s documents are kept: one written in the question’s language, preferably under its locale, if any was retrieved, else the best-ranked one. The same file served under every locale collapses too.

  2. Documents written in another language have their score scaled by other_language_weight and the rest are re-sorted, so they yield to documents of similar relevance in the question’s language but still outrank much weaker ones. Documents without a numeric score keep retriever’s order.

The first k are returned with their rank restated and their metadata["language"] set where known, so a prompt can show it. A question whose language is not detected skips step 2.

retriever = LanguageAwareRetriever(
    retriever=RerankingRetriever(retriever=hybrid, reranker=reranker),
    detector=LinguaLanguageDetector(["en", "es", "ca", "zh"]),
)
Parameters:

  • retriever (Retriever) – Ranks the candidates. A RerankingRetriever returns comparable scores across languages.

  • detector (LanguageDetector) – Detects the question’s and the documents’ languages.

  • candidate_k (int) – How many documents to fetch from retriever before choosing k.

  • other_language_weight (float) – Factor, in (0, 1], applied to the score of a document written in another language than the question. A negative score is divided by it instead, so it always demotes.

async retrieve(query, *, k=5)[source]#

Return the k best documents, preferring the question’s language.

Parameters:
Return type:

list[Document]