Serving internals#

What a contributor needs to know about fastrag.serve beyond the wire contract (HTTP API): the layout, the lock design, and the model. fastrag.serve is the port of the pre-queue-4 CLI serving layer onto the new QueryEngine + ChatStore design; the wire contract is unchanged, so a client written against the old app works unmodified.

Layout#

Only five modules are public; everything else is _-prefixed and internal.

  • app.py — create_app, the app factory with no engine of its own.

  • config.py — ServeConfig, frozen and validated.

  • cli.py — CLI, the Typer client (HTTP API).

  • cleanup.py — cleanup_expired, the batched expiry sweep.

  • _auth.py, _errors.py, _generation.py, _guards.py, _locking.py, _logging.py, _metrics.py, _net.py, _openapi.py, _rate_limit.py, _routes.py, _schemas.py, _shield.py, and _sse.py are internal to serve. _openapi.py force-publishes the SSE event payload schemas into the OpenAPI document’s components/schemas — used only by scripts/generate_openapi.py (task openapi), see Frontend types. The ULID minting _generation.py uses for a stored assistant message id lives at fastrag._ids, shared with fastrag.query.chat (see Architecture’s “shared private layer”). _metrics.py holds ServeMetrics (the app-owned CollectorRegistry and its nine series) and MetricsMiddleware; both are wired up by app.py, never imported by a caller directly — see Metrics. _net.py holds resolve_client_ip (shared by the rate limiter and the session writer, so the two can never disagree about who a request is from). The optional GeoIP lookup is grouped by feature in geolocalization/: resolver.py holds CountryResolver (built by create_app, closed by its lifespan) and dependencies.py the get_country_resolver dependency get_or_create_session receives it through; _rate_limit.py holds the sliding-window RateLimiter itself, deliberately free of any import from _metrics.py, _errors.py, or _net.py — config.py imports _rate_limit.parse_rate to validate its own rate-limit fields, and a _rate_limit → _metrics → config edge would make that an import cycle. See Admission: rate limiting and client IP/country for the user-facing behaviour both modules produce.

create_app(engine=, chat_store=, config=) — engine is a QueryEngine. The app borrows the store’s engine (via the PostgresChatStore.engine property) for the advisory lock and /healthz; it creates no engine or session itself.

Design#

  • The session id is the bearer secret. There is no username or password; the session ULID is the credential, stored as the plain-text primary key and used as both the bearer token and the cookie value. Session tokens are deliberately not hashed.

  • Generations are serialized per chat with a session-level Postgres advisory lock, acquired before generate and released after; contention returns 409. A store at a non-default schema needs a schema-qualified variant before cleanup_expired will sweep that schema.

  • Regenerate/edit appends rather than truncates. ChatStore has no range-delete primitive, so POST /v1/chats/{id}/messages/{mid} validates the request fully and returns the same SSE stream, but the chat stays linear and grows — see cli.py’s module docstring. Restoring true truncate needs a ChatStore primitive first.

  • Rate limiting is a dependency, not a middleware. _guards.rate_limit(...) is a FastAPI dependency (dependencies=[Depends(rate_limit("generation"))] on the gated routes), not a BaseHTTPMiddleware like RequestGuardMiddleware/MetricsMiddleware. It has to be: a dependency runs inside the router, after FastAPI’s ExceptionMiddleware is already wrapping the call, so raising ApiError there is caught by the same registered handler every other request-time error goes through. A BaseHTTPMiddleware sits outside ExceptionMiddleware in Starlette’s stack — an exception raised directly from its dispatch (as opposed to one raised inside the call_next it awaits) would bypass those handlers entirely and fall through to a generic 500, which is exactly why RequestGuardMiddleware above builds its JSONResponse by hand instead of raising. The one limit that isn’t a route-level dependency, rate_limit_new_session, is checked inline in _auth.get_or_create_session for the same reason enforce_content_length isn’t middleware either: it only applies to one branch of one function, not to a route as a whole.

See also#

Serving for the user’s door; Admission: rate limiting and client IP/country for the rate limiter and IP/country resolution this section describes the placement of; Architecture for where the package sits in the tree.