fastrag.query.llm#

Chat models: turning a conversation into a generated answer.

OpenAIChatModel is the one implementation: any endpoint speaking the OpenAI protocol, which also covers Ollama, vLLM and llama.cpp — only the model name and credential change.

class fastrag.query.llm.ChatModel[source]#

Bases: ABC

Turns a conversation into a generated answer.

invoke and stream are separate methods, not one method with a flag, because a streaming caller and a one-shot caller want different return shapes — tokens as they land versus one finished string — and forcing both through one signature would make one of the two callers reconstruct what it actually wanted from the other’s shape.

abstractmethod async invoke(messages)[source]#

Generate the whole answer at once, as plain text.

Parameters:: messages (Sequence[Message])
Return type:: str

abstractmethod stream(messages)[source]#

Generate the answer as text deltas, nothing else.

Yields plain text tokens only, so a provider that cannot report usage mid-stream never has to fabricate it.

Parameters:: messages (Sequence[Message])
Return type:: AsyncIterator[str]

abstract property model_id: str#: Stable identifier of the model, for logging and events.

class fastrag.query.llm.Message(role, content)[source]#

Bases: object

One turn in a chat conversation.

Parameters:

role (Literal['system', 'user', 'assistant'])
content (str)

role#

Who sent the turn.

Type:: Literal[‘system’, ‘user’, ‘assistant’]

content#

The turn’s text.

Type:: str

class fastrag.query.llm.OpenAIChatModel(*, model, api_key, base_url=None, temperature=0.0, max_tokens=None, timeout=60.0, extra_body=None)[source]#

Bases: ChatModel

Generate with any OpenAI-compatible endpoint.

Choose this for OpenAI’s models or for a self-hosted server that speaks the OpenAI protocol (vLLM, Ollama, text-generation-inference). Point base_url at the self-hosted endpoint; the only things that change are the model name and the credential the endpoint expects.

import os

from fastrag.query.llm import OpenAIChatModel

llm = OpenAIChatModel(
    model=os.environ["FASTRAG_CHAT_MODEL"],
    api_key=os.environ["FASTRAG_CHAT_API_KEY"],
    base_url=os.environ["FASTRAG_CHAT_BASE_URL"],
)

Parameters:

model (str) – Model name, e.g. gpt-4o-mini or llama3.2:3b.
api_key (str) – API key. Self-hosted endpoints usually accept any non-empty string.
base_url (str | None) – Endpoint base URL. None uses the OpenAI API.
temperature (float) – Sampling temperature passed to the provider.
max_tokens (int | None) – Cap on generated tokens. None leaves the provider’s own default in place.
extra_body (Mapping[str, Any] | None) – Additional JSON request fields; matching fields override defaults.
timeout (float) – Seconds per provider call. CPU-bound self-hosted models are far slower than a hosted API — raise this before raising retries.

Raises:

fastrag.exceptions.ConfigurationError – The “openai” extra is not installed, or max_tokens < 1.

property model_id: str#: The configured model name.

async invoke(messages)[source]#

Generate the whole answer at once, as plain text.

No provider exception escapes: everything is translated into the Fastrag hierarchy, with timeouts, rate limits, and server errors marked transient so a caller wrapping this in retries gets correct behaviour for free.

Parameters:: messages (Sequence[Message])
Return type:: str

async stream(messages)[source]#

Generate the answer as text deltas.

Not retried: tokens already yielded cannot be un-yielded, so a retry mid-stream would emit two overlapping partial answers. A stall still raises after timeout seconds with no delta.

Parameters:: messages (Sequence[Message])
Return type:: AsyncIterator[str]

Modules

`base`	The `ChatModel` contract: messages to a generated answer.
`openai`	`OpenAIChatModel`: OpenAI-compatible chat completions, via langchain-openai.