fastrag.query.llm.openai#

OpenAIChatModel: OpenAI-compatible chat completions, via langchain-openai.

A ChatModel whose generate() -> Completion collapses into invoke() -> str — token counts are not part of this contract. Any endpoint speaking the OpenAI protocol (Ollama, vLLM, llama.cpp) works by pointing base_url at it.

Classes

OpenAIChatModel(*, model, api_key[, ...])

Generate with any OpenAI-compatible endpoint.
class fastrag.query.llm.openai.OpenAIChatModel(*, model, api_key, base_url=None, temperature=0.0, max_tokens=None, timeout=60.0, extra_body=None)[source]#

Bases: ChatModel

Generate with any OpenAI-compatible endpoint.

Choose this for OpenAI’s models or for a self-hosted server that speaks the OpenAI protocol (vLLM, Ollama, text-generation-inference). Point base_url at the self-hosted endpoint; the only things that change are the model name and the credential the endpoint expects.

import os

from fastrag.query.llm import OpenAIChatModel

llm = OpenAIChatModel(
    model=os.environ["FASTRAG_CHAT_MODEL"],
    api_key=os.environ["FASTRAG_CHAT_API_KEY"],
    base_url=os.environ["FASTRAG_CHAT_BASE_URL"],
)
Parameters:

  • model (str) – Model name, e.g. gpt-4o-mini or llama3.2:3b.

  • api_key (str) – API key. Self-hosted endpoints usually accept any non-empty string.

  • base_url (str | None) – Endpoint base URL. None uses the OpenAI API.

  • temperature (float) – Sampling temperature passed to the provider.

  • max_tokens (int | None) – Cap on generated tokens. None leaves the provider’s own default in place.

  • extra_body (Mapping[str, Any] | None) – Additional JSON request fields; matching fields override defaults.

  • timeout (float) – Seconds per provider call. CPU-bound self-hosted models are far slower than a hosted API — raise this before raising retries.

Raises:

fastrag.exceptions.ConfigurationError – The “openai” extra is not installed, or max_tokens < 1.

property model_id: str#

The configured model name.

async invoke(messages)[source]#

Generate the whole answer at once, as plain text.

No provider exception escapes: everything is translated into the Fastrag hierarchy, with timeouts, rate limits, and server errors marked transient so a caller wrapping this in retries gets correct behaviour for free.

Parameters:

messages (Sequence[Message])

Return type:

str

async stream(messages)[source]#

Generate the answer as text deltas.

Not retried: tokens already yielded cannot be un-yielded, so a retry mid-stream would emit two overlapping partial answers. A stall still raises after timeout seconds with no delta.

Parameters:

messages (Sequence[Message])

Return type:

AsyncIterator[str]