Decisions#

Recorded so a future change cannot silently reverse one without someone noticing. Append-only: when a decision changes, add a new entry that supersedes the old one and says so, rather than editing the original away.

Non-goals#

Stated explicitly, because they bound every decision below:

  • Not a LangChain replacement. fastrag uses LangChain internally where it is the best available implementation and adapts at the edges (Conventions). It does not aim to cover LangChain’s surface.

  • Not a CLI. See decision 1.

  • Not a serving platform. fastrag.serve ships the chats API as a thin adapter over a QueryEngine and a ChatStore, with no mandatory infrastructure — the app owns no database engine and no ORM session; it borrows the store’s. Serving does not reimplement retrieval or prompting.

  • No global configuration. No settings singleton, no implicit .env loading, no ambient state. Credentials are arguments — see decision 14.

The SDK#

#

Decision

Rationale

1

No CLI. SDK only.

The value is typed, composable pipelines. A CLI re-flattens them into strings and doubles the surface to document and test.

2

Async at process boundaries; sync for computation. One name per operation, one run_sync door.

Duplicate run/arun pairs made the contract unreadable. See Conventions for the two narrow exceptions this settled into in practice (Metric.score, BenchmarkResult.save/.load).

3

Own types; LangChain adapted at the edges.

Delivers the “easy wrapper” pitch without inheriting upstream API churn into the public surface. fastrag.models.Document.content is deliberately not named page_content, so a future LangChain-object adapter can rename in both directions at zero cost when one is built.

4

Step cache and record manager are separate.

They answer different questions: “must I recompute?” and “what is in the collection that should not be?” Conflating them is why purely time-based caching cannot detect change.

5

Metrics are objects.

Strings cannot carry a judge model, cannot be type-checked, and cannot be extended without a registry. metrics=["faithfulness"] silently returning bag-of-words overlap is the worst failure an evaluation tool can have.

6

Collection name = human prefix + config fingerprint.

The prefix keeps it declarative and readable at the call site; the fingerprint makes mixing incompatible vectors impossible. Excluding fetcher configuration from the fingerprint keeps a live site indexing into one collection as it grows.

7

Deterministic content-derived document ids.

The precondition for VectorStore.upsert. Random ids per run are what made a second run duplicate the corpus.

8

incremental cleanup by default; full refuses after fetch errors.

A failed fetch must never be read as “this content was deleted.”

9

No global configuration.

A settings singleton hard-coding one vector DB and one endpoint contradicts a swappable-services SDK.

10

Benchmark results carry a name, not a config.

The benchmark script is already the readable, version-controlled record. Serializing the pipeline configuration too would duplicate state that will drift.

11

Metrics aggregate to a distribution, not a mean.[1]

A mean latency is close to useless and a mean faithfulness hides the tail. Percentiles are what a service level is written against.

12

Serving ships the chats API.

Originally kept serving WIP — /healthz only, no mandatory infrastructure. That requirement survives as create_app’s layering (neither argument → /healthz only), and serving now ships the full /v1 chats API as a thin adapter over a QueryEngine.

13

The step cache is a database when one is configured — PostgreSQL today.

A directory of files could not evict, size itself, or clear a prefix without walking the tree, and could hand a parser a truncated entry after a crash. In the current tree the cache is explicit and per-step: each step takes cache= (a CacheStore, or None to inherit, or False to opt out) — see Caching.

14

No settings module. fastrag.settings is deleted; every credential is an argument, with no exception.

The module existed only for alembic’s env.py, which now reads FASTRAG_DATABASE_URL directly — appropriate for a CLI entry point invoked by the alembic command, not code a caller imports. Keeping the module around “for Alembic” while telling users the SDK never reads the environment was a contradiction the documentation could not describe truthfully on one site; deleting it was cheaper than narrowing the rule, and the rule is load-bearing for a swappable-services SDK.

Serving#

Appended to the log above with its own numbering, serving-tagged, ported from the design this package’s serving layer was built against.

#

Decision

Rationale

S1

Streaming is stateless. No streaming state is persisted; a message is written only once its generation finishes.

Any replica can serve any request — no load balancer session affinity is required, and the database is the only shared state. A disconnect mid-generation is stored with finish_reason: "interrupted" rather than resumed.

S2

The server assigns every id. Clients never choose a session, chat, or message id.

A client-supplied id is untrusted input that would let a caller overwrite or probe rows it does not own. This is why message_start carries ids at all — it is the only way a client learns the id of the message it just sent.

S3

IDs are ULIDs.

Lexicographically sortable by creation time, so ORDER BY id is chronological and cursor pagination needs no secondary timestamp column.

S4

Chats are linear, not trees. Re-running a message truncates everything after it rather than branching.

The UI shows a single thread; a tree would need a “current leaf” pointer and merge rules for no product gain.

S5

Generations are serialized per chat, with a session-level Postgres advisory lock.

See Serving internals for the full lock design and why it is session-level rather than transaction-scoped.

S6

Sessions and chats expire after 30 days of inactivity, swept by an externally-scheduled job rather than on request.

Cleanup cadence stays independent of API traffic; see Serving internals’s Retention section for why not pg_cron.

S7

Expiry bumps are write-throttled. An expiry is extended on activity only when the new value is materially later than the stored one.

Keeps active conversations alive without a write on every single request.

S8

The session id is the bearer secret. There is no username, password, or OAuth layer.

The cookie is the credential; a bearer token is the same credential carried in a header instead — see Serving internals’s “Bearer tokens, in full.”

S9

The chat-store schema is self-managed, the same way the ingestion cache’s is. PostgresChatStore.__aenter__ creates its schema, tables, and ENUM types if absent (create_all(checkfirst=True)); nothing in fastrag touches Alembic or a shared declarative Base, though alembic is still a declared dependency from the design this was ported from.

Matches decision 13’s reasoning for the ingestion cache: a self-managing store needs no separate migration step, and a swappable ChatStore cannot assume the caller runs one. The ported design’s original “migrations only” rule was not carried into this implementation and never got its own decision entry recording the change — this row corrects that gap rather than adding a new one.

S10

Every generation carries the chat’s full history, unwindowed and unsummarized.

The alternative — condensing history into a standalone query before retrieval — needs an extra LLM round-trip before the first token; deferred (BACKLOG.md). Accepted trade: a follow-up pronoun resolves in generation but not in retrieval.

S11

History replay includes a row if and only if its content is not NULL.

The content_required_unless_error constraint permits a NULL body on a failed assistant row; replaying unconditionally would construct an empty message and fail the third turn of any chat that ever errored.

References#