Runbook¶

Operating the Varagity stack: from clean clone to answered question — in the terminal or the browser — plus the operational gotchas collected while building it.

Prerequisites¶

Docker + Docker Compose, with the nvidia-container-toolkit runtime configured (the README has step-by-step Debian instructions).
An Nvidia GPU setup matching the compose GPU pinning (see GPU & VRAM — this repo's config assumes two GPUs).
≥ 12 GB free disk for images/volumes — and note Elasticsearch's disk watermarks (below): a host disk > 90% full will break indexing outright.
Model files on the host (never copied into images):
- ${models_volume}/<BASE_MODEL> — the llama.cpp .gguf chat model.
- ${embeddings_volume}/multilingual-e5-large-instruct/ — the intfloat/multilingual-e5-large-instruct snapshot (its ONNX weights ship upstream; the infinity optimum engine serves them).
- ${embeddings_volume}/bge-reranker-v2-m3/ — the BAAI/bge-reranker-v2-m3 snapshot plus a pre-exported ONNX model under onnx/ (e.g. via optimum-cli export onnx): the optimum engine does not auto-export from safetensors, and the container serves both models (why the reranker is here).

First-time setup¶

git clone https://github.com/luisegarduno/varagity && cd varagity
cp .env.example .env

Edit .env:

Point models_volume / embeddings_volume at your host model directories and set secret_infinity_key / POSTGRES_PASSWORD.
Set BASE_MODEL to your .gguf filename.
Leave the service URLs at their in-container values (http://llamacpp:8080/v1 etc.) — the checked-in convention is container values with host overrides in comments (host-vs-container).
Browsing from another machine? Put this host's LAN address in NEXT_PUBLIC_API_URL and add that origin to API_CORS_ORIGINS (web: dev vs prod).

Put some documents in ./docs/ (the gitignored ingest corpus — not golden-docs/, which is this documentation) — or upload them from the browser in a minute — then:

docker compose up -d --wait     # exits 0 once every healthcheck is green
bash scripts/smoke.sh           # 10 sequenced checks across the default stack

Either front-end works from here:

Browser — open http://localhost:3000: upload documents on the corpus page, run the ingest with live per-stage progress, and ask; answers arrive with citation chips and the evidence panel.
Terminal — docker compose logs -f varagity shows the app container ingest, then prompt. It runs chat non-interactively (ingests, prints the prompt, exits on stdin EOF); for an interactive session run the CLI attached, or from the host:

docker compose run --rm app uv run main.py chat

Ask something answerable from your corpus; you should get a grounded answer citing its [SOURCE]. :quit exits the CLI loop.

Bring-up order & healthchecks¶

docker compose up -d --wait handles ordering across the ten default services (two more are profile-gated, below): app and api declare depends_on: condition: service_healthy on all five backing services, web waits on api, grafana on prometheus, and --wait blocks until every healthcheck is green. What "healthy" means per service:

flowchart LR
    llamacpp["llamacpp<br/>curl /health"] --> gate
    infinity["infinity-embeddings<br/>curl /health"] --> gate
    postgres["postgres<br/>pg_isready"] --> gate
    elasticsearch["elasticsearch<br/>curl /_cluster/health"] --> gate
    prefect["prefect<br/>urllib /api/health"] --> gate
    gate{{"all 5 backing healthchecks green"}} --> app["app (CLI)<br/>no probe of its own"]
    gate --> api["api<br/>venv-python urllib /api/health"]
    api --> web["web<br/>node fetch :3000"]
    prometheus["prometheus<br/>wget /-/healthy"] --> grafana["grafana<br/>wget /api/health"]

Service	Probe	Semantics to know
`llamacpp`	`curl /health`	Returns 503 while the model loads (~30 s for the current 9B `.gguf`); the generous `retries: 20` covers it. Also unprioritized under load — a busy server can look slow to probe.
`infinity-embeddings`	`curl /health`	`/health` is not under the `/v1` prefix (API routes are). First boot may re-optimize ONNX graphs, which takes a minute.
`postgres`	`pg_isready`	`schema.sql` runs on first boot only (empty data dir); everything newer arrives via the API's migration runner (below).
`elasticsearch`	`curl /_cluster/health`	Reachability only — single-node clusters are `yellow` by design (replicas can never assign). Never gate on `green`.
`prefect`	python-urllib `/api/health`	The image ships no `curl`/`wget`, hence the stdlib probe.
`api`	venv-python urllib `/api/health`	Same no-curl posture. `/api/health` answers 200 whenever the process is alive — per-dependency state rides in the body, and each probe refreshes the `varagity_dependency_up` gauge. Startup applies pending SQL migrations first.
`web`	node `fetch(':3000')`	Alpine image, no curl — node's global `fetch`. `NEXT_PUBLIC_API_URL` was baked at build time (below).
`prometheus`	busybox `wget /-/healthy`	Busybox-based image — `wget --spider`.
`grafana`	busybox `wget /api/health`	Waits on `prometheus`; the datasource + dashboards are provisioned from read-only mounts at boot.

Two optional services are profile-gated — off by default, and enabling one needs no config edits (observability):

docker compose --profile prefect-exporter up -d   # flow/task-run metrics polled from the Prefect API
docker compose --profile gpu-metrics up -d        # dcgm-exporter: GPU VRAM/utilization

prefect-exporter exposes no host port (Prometheus scrapes it in-network at prefect-exporter:8000); dcgm-exporter reserves all GPUs and runs with cap_add: SYS_ADMIN (DCGM's documented posture). Profiles are not sticky across plain compose invocations — see the optional exporter profiles.

scripts/smoke.sh then verifies substance, not just liveness — ten sequenced checks over the default stack: (1) every container's health state (nine healthchecked services; app has no probe), (2) llama.cpp serves /health and /v1/models lists BASE_MODEL, (3) infinity answers on its interface-specific bound port and returns a 1024-dim embedding, (4) the pg schema — vector extension, documents/chunks, all three chunk indexes, and the five migration-runner tables (conversations, messages, message_sources, app_settings, schema_migrations — proof the API's startup migrations ran), (5) ES cluster health yellow/green, (6) Prefect /api/health, (7) the API reports every dependency reachable and serves /openapi.json plus the varagity_* metric families at /metrics, (8) the web app answers on :3000, (9) Prometheus is healthy with the varagity-api scrape target up, (10) Grafana is healthy and the anonymous search API lists the three provisioned dashboards.

Day-to-day usage¶

uv run main.py ingest              # ingest DOCS_PATH into both stores
uv run main.py ingest --reingest   # delete + re-process every discovered doc
uv run main.py chat                # ingest, then the Q&A loop (the default)
uv run main.py -v 2 chat           # verbose: full chunk/retrieval panels
uv run --group eval main.py eval       # 7-config retrieval matrix + chunker sweep
uv run --group eval main.py eval ocr   # OCR engine benchmark
uv run --group eval main.py eval chat  # multi-turn chat-engine eval (ADR-011)
uv run main.py eval graph              # GraphRAG regression harness (ADR-017) — see below

The browser flow is the same pipeline without a terminal: at http://localhost:3000, upload files (POST /api/documents, per-file cap UPLOAD_MAX_MB, landing in the same ./docs corpus), run the ingest with live per-stage/per-chunk progress (POST /api/ingest — the same tracked flow on a background thread), and ask. Prefer it for ingests you want on the Grafana Ingestion dashboard (per-process metrics). The Message graph tab beside it is the second corpus — message archives in, a resumable extraction build out (below); it has no CLI equivalent by design.

Since v3 the chat composer's 📎 attach is the one-click door (ADR-012): files and folders upload (folder structure preserved under DOCS_PATH — it is part of each document's identity), auto-ingest with a progress chip, and queue client-side behind an already-running ingest instead of erroring. Batch caps: UPLOAD_MAX_FILES (500) and UPLOAD_MAX_TOTAL_MB (2048). A composer upload runs reingest=false, so it never clears the stale-corpus flag (below). /corpus remains the full management surface with its explicit two-step.

Every UI in one place:

Surface	URL
Web GUI (chat · evidence · corpus · settings)	`http://localhost:3000`
Graph corpus tab · the graph view	`http://localhost:3000/corpus?tab=graph` · `/graph`
HTTP API (OpenAPI at `/openapi.json`)	`http://localhost:8000`
Prefect UI (flow/task runs)	`http://localhost:4200`
Prometheus	`http://localhost:9090`
Grafana dashboards	`http://localhost:3001`

--reingest semantics: idempotency keys on file bytes (content_hash), so pipeline-setting changes (CONTEXTUALIZE, chunk params, OCR_ENGINE) do not mark files as changed — unchanged files are skipped until you --reingest. It clears each document from both stores first, keeping them consistent. Re-ingesting from the GUI additionally clears the stale flag (below); the CLI variant does not.

Retrieval method is an env toggle: RETRIEVAL_METHOD=semantic|bm25|hybrid|reranked|hyde (default hybrid) — or flip it live from the GUI settings drawer, no restart. Handy for A/B-ing a query that one method struggles with.

HyDE (post-v3, ADR-016) rides that toggle: RETRIEVAL_METHOD=hyde has the chat LLM write a hypothetical answer passage per question and steers the dense arm with its passage-mode embedding — BM25 still sees your words. Pair it with the cross-encoder as RETRIEVAL_METHOD=reranked + RERANK_BASE_METHOD=hyde (rerank outside, so it judges the real query). Budget for the extra LLM round-trip before retrieval on every question — the same llama.cpp server that answers. HYDE_ENABLED=false is the kill switch, not a method: hyde then degrades to its HYDE_BASE_METHOD and logs it, exactly the RERANK_ENABLED shape — and any generation failure degrades the same way per-query, so a flaky LLM slows retrieval rather than breaking it.

Chat engine is the same shape of toggle (v3, ADR-011): CHAT_ENGINE=simple|condense_context (default simple — the benchmark-decided call; condensing wins pronoun follow-ups decisively but costs a mean ~8.6 s LLM round-trip before retrieval on this stack's reasoning model). Under condense_context, follow-ups are rewritten into standalone search queries against the last CONDENSE_HISTORY_TURNS turns — the evidence panel shows the rewrite as "Searched for: …", and the answer prompt always keeps the user's original words. CONDENSE_ENABLED=false is the kill switch, not an engine: condense_context then degrades to simple behavior and logs it — selection and the toggle are deliberately orthogonal, exactly like RERANK_ENABLED (below).

Runtime settings & the stale-corpus flag¶

The GUI's settings drawer writes through PATCH /api/settings (spec_v2 §4.7): overrides persist in the app_settings table, are merged over the env-loaded Settings, and are re-loaded on API startup — they survive restarts and take precedence over .env. GET /api/settings reports the effective values plus the stale flag.

Query-time knobs — RETRIEVAL_METHOD, TOP_K, the fusion weights, the rerank toggles, LLM_TEMPERATURE, MAX_TOKENS, CHAT_MODEL_TYPE, and v3's CHAT_ENGINE + CONDENSE_* family — take effect on the next question. No restart, no reingest (deliberately: none of the chat-engine settings are reingest-affecting).
Ingest-time knobs — CHUNKING_STRATEGY, CHUNK_SIZE/CHUNK_OVERLAP, CONTEXTUALIZE, OCR_ENGINE — don't change content hashes, so changing one while documents are ingested flags the corpus stale, and the GUI shows a persistent "re-ingest to apply" affordance.
Only a completed API-driven reingest=true run clears the flag. A CLI uv run main.py ingest --reingest re-processes the corpus but runs in another process and cannot clear it; patching a setting back to its old value doesn't clear it either — the flag records "the corpus may not match the current settings", not a diff. When in doubt, run the GUI reingest once.
The settings routes need postgres (structured 503 postgres_unreachable when it's down); unknown or invalid keys are a structured 422.

Host-vs-container `.env` usage¶

The single .env is consumed twice — Docker Compose interpolates ${lowercase} vars into service definitions, and the app loads the same file via pydantic-settings. The checked-in values are the in-container ones (service names: http://llamacpp:8080/v1). When running the app on the host (dev loops, tests, eval), override the endpoints per run:

BASE_MODEL_API_URL=http://localhost:8080/v1 \
POSTGRES_HOST=localhost \
ELASTICSEARCH_URL=http://localhost:9200 \
PREFECT_API_URL=http://localhost:4200/api \
EMBEDDING_API_URL="http://$(docker compose port infinity-embeddings 8081)/v1" \
RERANK_API_URL="http://$(docker compose port infinity-embeddings 8081)/v1" \
DOCS_PATH=./docs \
uv run main.py chat

(RERANK_API_URL is only consulted by the reranked method — same infinity address as the embeddings.)

The API runs on the host the same way — the uvicorn factory plus the same overrides:

BASE_MODEL_API_URL=http://localhost:8080/v1 \
POSTGRES_HOST=localhost \
ELASTICSEARCH_URL=http://localhost:9200 \
PREFECT_API_URL=http://localhost:4200/api \
EMBEDDING_API_URL="http://$(docker compose port infinity-embeddings 8081)/v1" \
RERANK_API_URL="http://$(docker compose port infinity-embeddings 8081)/v1" \
DOCS_PATH=./docs \
uv run uvicorn varagity.api.main:create_app --factory --port 8000

A host API applies the startup migrations and loads persisted setting overrides exactly like the container. Mind the ports: with the compose api service up, host :8000 is already taken — docker compose stop web api first (stopping web too keeps :3000 free for the dev server, below), or pick another --port.

infinity's host binding is interface-specific

The compose maps infinity to 192.168.86.21:8081, not 0.0.0.0 — plain localhost:8081 does not reach it from the host. Resolve the bound address with docker compose port infinity-embeddings 8081 (as above and in scripts/smoke.sh).

Web app: dev vs prod¶

NEXT_PUBLIC_API_URL is a build-time constant. Next.js inlines NEXT_PUBLIC_* values into the compiled bundle; compose passes it as a build arg (default http://localhost:8000). Consequences:

Editing it in .env does nothing to an already-built image — run docker compose build web && docker compose up -d web.
It is the browser's route to the API, not the container's: browsers on other machines need this host's LAN address in NEXT_PUBLIC_API_URL (rebuild) and that origin added to API_CORS_ORIGINS (API restart) — two knobs, both required.

The dev loop skips the image entirely (web/ is bun-only — never npm/yarn):

cd web
bun install
NEXT_PUBLIC_API_URL=http://localhost:8000 bun run dev   # http://localhost:3000

pointed at the compose api or a host-run API. Free the port first (docker compose stop web): the dev server must own :3000, which is also the only origin API_CORS_ORIGINS allows by default. bun run test (Vitest unit), bun run lint, bun run build, bun run gen:types (regenerates lib/types.ts from the running API's OpenAPI schema — generated, never hand-edited), and bun run build:docs-map (rebuilds golden-docs/assets/map/, the docs site's committed copy of /map — ADR-018; a Vitest guard fails on a stale bundle) round out the toolchain.

bun is the package manager, not the runtime. Node executes everything — next, vitest, and playwright run under Node via their shebang scripts (bun run <script> merely launches them; no --bun, no bun test). The bun version pin lives in two places: the web/Dockerfile base stage (oven/bun:1.3.14-alpine) and .github/workflows/ci.yml (bun-version: 1.3.14) — bump both together. Installs are fast even cold: a fresh-cache bun install --frozen-lockfile measured ~3.2 s for 779 packages (bun 1.3.14, 2026-07-15), versus tens of seconds under the previous package manager (ADR-005 amendment).

The opt-in e2e harness¶

web/e2e/ holds Playwright specs (smoke, keyboard, chat flow, groups, page previews, the codebase map, the corpus tabs, graph chat, the graph view, and axe accessibility) that exercise the real stack — deliberately no webServer block in playwright.config.ts, so nothing is started for you:

docker compose up -d --wait
bun --cwd web run e2e         # or: cd web && bun run e2e

Browser binaries are an explicit step: run bunx playwright install --with-deps once per machine — no lifecycle script installs them (playwright@1.61.1 ships no scripts field at all), so this is a documented setup step, not an install side-effect.
Targets http://localhost:3000 (which must reach the API on :8000); PLAYWRIGHT_BASE_URL overrides the base URL.
Serial on purpose (fullyParallel: false, workers: 1): the backend is single-user — one conversation store, one GPU — so parallel specs would race each other's state. Generation on the local GPUs runs 30–90 s per answer; the per-test timeout is 120 s.
Read-only toward both corpora: the specs chat and drive the UI; they never upload, delete, or reingest, and the graph specs never press Build — starting a multi-day extraction is not a test's business.
The graph specs skip cleanly on an empty graph (corpus-tabs, graph-chat, graph-view): they read GET /api/graph/status first and skip with a reason when the subsystem is off or nothing has been extracted, so a fresh stack reports skips rather than failures.
Testing a dev build: the dev server must own :3000 (stop the compose web first) — API_CORS_ORIGINS allows only that origin, so any other port fails CORS before it fails a test.
Each spec runs twice: desktop chromium (1280×800) and mobile chromium (390×844, touch).

Graph corpus operations¶

The message graph is the second corpus: message archives in, a resumable extraction build, and a graph the chat composer can target. Everything below happens in the GUI at http://localhost:3000/corpus?tab=graph (or against the API directly) — there is no CLI path, deliberately.

The API process is the graph's single writer

The engine's storages are single-writer per working directory and the api container is that writer: builds, queries, exports, and status polls all go through its one in-process session. Nothing else may write the graphdata volume — no second API process, no host-mode API against the same directory, no CLI build (there is deliberately no such command), no hand-editing of the workdir files. Two writers do not conflict loudly; they corrupt the graph quietly. Reads are safe from that same process, which is why a chat question is answered while a multi-day backfill is still extracting.

1 · Upload the archive. Drop the file(s) on the Message graph tab (POST /api/graph/documents, per-file cap GRAPH_UPLOAD_MAX_MB, default 4 GB — a decade of iMessage is one enormous file). Uploads are sniffed, not extension-filtered: the server decides what a file is by reading its bytes, so a renamed chat.db is fine and an unreadable file is deleted again and reported as unsupported_graph_source.

Copy the WAL sidecars too

A live iMessage database keeps its most recent messages in chat.db-wal. Copying chat.db alone silently loses them — the build succeeds, the graph is just missing the newest conversations. Copy all three (chat.db, chat.db-wal, chat.db-shm) and upload them together; sidecars are stored beside their database and never sniffed on their own. GRAPH_HANDLE_NAMES_FILE (a contacts export) may live in the same directory — it is read as configuration, not as a source.

2 · Build. Press Build. The run streams progress (scan → parse → index → per-status ticks) and the stream replays from frame one, so opening the tab six hours in shows the same picture. Budget for it:

Corpus	Wall-clock (this host, one llama.cpp slot)
per message	7.17 s measured
10,000 messages	≈ 20 h
+10 new messages into an existing graph	≈ 140 s

3 · Kill and resume. A build is safe to kill (docker compose restart api, a reboot, a crash): document statuses are flushed to disk at every stage boundary, and every in-flight or failed document is re-selected on the next pass. Press Build again to resume — nothing already extracted is re-done, and the run reports what it inherited as resumed. There is no separate resume button because there is no separate operation.

4 · Bounded backfills — the escape hatch for spot-checking an archive whose full build runs for a day:

# newest 500 messages only (the GUI's "limit" field does this)
curl -sX POST localhost:8000/api/graph/build \
  -H 'content-type: application/json' -d '{"message_limit": 500}'

# only messages from 2024 onward (API-only — no GUI field)
curl -sX POST localhost:8000/api/graph/build \
  -H 'content-type: application/json' -d '{"since": "2024-01-01"}'

Both keep what they cannot see: a bounded render is partial by construction, so out-of-window documents are kept rather than deleted. The one exception is a re-span casualty — growing the window backward shifts the greedy day-span packing and re-keys downstream documents, and a removed key whose every recorded message this render re-indexes under new keys is a duplicate transcript, not archive content. Those are deleted (the build log reads pruning N re-spanned, keeping M); anything short of full guid coverage is kept. Run an unbounded build afterwards to fill in the rest — the manifest diff means the bounded slice is already indexed and costs nothing the second time.

5 · Re-upload a grown chat.db. Upload the newer copy over the old one and build again. Thread-days whose content is byte-identical are skipped; changed days are deleted from the graph and re-extracted (the engine's own dedup would otherwise keep the stale transcript forever); genuinely new days are extracted. Only the difference costs GPU time. reingest=true is the sledgehammer — it wipes the working directory and re-indexes everything — and it is the only thing that clears the graph-stale flag that a deleted source file sets.

6 · Back it up. A graph that took days to build is worth a backup: Export on the tab writes the whole working directory into graph-docs/varagity-exports/, and Restore brings it back in seconds — see Backups.

7 · Reset the graph (keeping the source archives, which are a host bind):

docker compose down
docker volume rm varagity_graphdata     # the extracted graph only
docker compose up -d --wait             # next build starts from zero

docker compose down -v now wipes the graph too, along with everything else. ./graph-docs survives both — delete files there (or from the tab) to retire a source.

Host-mode runs need the two graph paths overridden alongside the usual endpoints (above) — the checked-in values are container paths:

Setting	Container (`.env`)	Host override
`GRAPH_DOCS_PATH`	`/app/graph-docs`	`./graph-docs`
`GRAPH_STORAGE_PATH`	`/app/graph-data`	`./graph-data`

Point a host API at a scratch GRAPH_STORAGE_PATH unless you mean to take over the real graph — the single-writer rule is not enforced by a lock file across processes.

Turning it off. GRAPH_ENABLED=false is a kill switch in the shape RERANK_ENABLED established: uploads, builds, and the graph view answer a structured 403 graph_disabled, and a graph-targeted question degrades to a document answer, labelled graph → documents on the turn. Listing, deleting, and the status poll keep working, because a disabled graph still has a corpus directory to manage. Nothing already extracted is touched.

When the graph answers badly, the two levers are GRAPH_QUERY_MODE (mix default; local/global/hybrid/naive are the engine's own vocabulary) and GRAPH_QUERY_PREFIX (whether queries get e5's Instruct: wrapper — passages are never prefixed). Both are env-only and both are re-measurable in minutes with eval graph --skip-build (below) — change them against a number, not a hunch.

The GraphRAG bake-off (`eval graph`)¶

eval graph is the harness that decided ADR-017 and now lives on as the GraphRAG regression guard (the eval chat shape): it builds a deterministic synthetic iMessage corpus, indexes it with every registered graph engine sequentially, and scores their answers against a golden QA set by fact recall and provenance recall.

uv run main.py eval graph                          # smoke profile, every registered engine
uv run main.py eval graph --profile full --engine lightrag

No extra dependency group. The shipped engine (lightrag-hku) is a main dependency since ADR-017 — a plain uv sync is enough. The bake-off's bakeoff group, which carried cognee and graphiti-core, went out with the losing adapters; git history has both if a candidate ever re-enters the seam.
Needs the live GPU services, not Docker stores. llama.cpp and infinity must be up; postgres/Elasticsearch are irrelevant because graph engines self-store. Running from the host means the usual endpoint overrides — see Host-vs-container .env usage — and that includes PREFECT_API_URL=http://localhost:4200/api: the harness is a Prefect flow, so the two model URLs alone die at flow start on the in-container prefect hostname.
Artifacts (all gitignored): corpora and per-engine working directories under data/eval/graph/, results at data/eval/results/<timestamp>-graph.json.
While a run is in flight, use uv run --no-sync for everything else. A plain uv run in another shell re-syncs the virtualenv, and a sync that changes a package under a live process kills a multi-hour index for an unrelated command.

Flags

Flag	Effect
`--profile smoke` (default)	226 hand-scripted messages, 10 threads — plus an automatic incremental re-index check
`--profile full`	10,001 messages across 50 threads and 12 years (226 scripted + blocklist-clean filler)
`--engine NAME` (repeatable)	run a subset; one engine per session is the sane default at full profile
`--mode NAME`	override an engine's primary query mode for a recorded extra pass (e.g. LightRAG `global`)
`--skip-build`	reuse the corpus and each engine's already-built graph — re-scores in seconds instead of hours
`--message-target N`	cap the corpus at N messages: how an engine too slow for the uncapped corpus still gets a full-profile seat

--skip-build has one honest caveat: it re-scores fact recall and latency, but not provenance for engines whose citation→guid index is session state built during the index pass — those read null provenance on a re-score (the CLI warns). Re-index if provenance is what you are iterating on.

--message-target makes a different corpus, on purpose. The database, manifest, and per-engine working directories are named for the cap (full-mt500), so a capped run can never clobber an uncapped one that another engine is still indexing against, and --skip-build reuse is keyed the same way. The scripted messages ride in every corpus (filler tops up to N), so golden anchors always resolve. The CLI prints a "comparable only to runs over the same corpus" note, and so must any table you build from it.

Expected wall-clock (measured on this host — one llama.cpp slot, a reasoning model; budget a night per full-profile engine):

Run	Wall-clock
`--profile smoke`, all three engines	4 h 51 m
`--profile full`, LightRAG (10,001 messages)	19.9 h index + 13 min queries
`--profile full`, cognee (10,001 messages)	14.2 h index + 7.3 h queries
`--profile full --message-target 500`, Graphiti	5.6 h index + 3 min queries
`--skip-build`, one engine, 17 queries	~2 min

Per-message index cost is the number that scales: 7.2 s (LightRAG), 5.1 s (cognee), 40.0 s (Graphiti — which is why it ran capped; an uncapped 10,001-message build extrapolates to ~4.6 days).

The -m bakeoff pytest marker (uv run pytest -m bakeoff --no-cov) is the tiny live smoke test — build ~30 messages, ask one question, re-build unchanged, and check the workdir sidecars. The marker keeps its bake-off name but now just means "runs the real engine against the live GPU services". --no-cov because any -m run trips the repo-wide coverage floor.

Backups¶

docker compose down -v wipes both processed corpora along with the volumes (below): the chunk rows, their contextual blurbs and embeddings, the BM25 index, and the extracted message graph. The source folders survive because they are host binds — and so does a backup bundle, which is written into a reserved folder inside one (docs/varagity-exports/ or graph-docs/varagity-exports/, Data model; the design is ADR-019). Restoring a bundle brings the corpus back with no parsing, chunking, contextualizing, embedding or graph extraction. Both live on each /corpus tab's Export & restore panel, or against the API.

The document corpus¶

# export: writes docs/varagity-exports/varagity-rag-<UTC yyyymmdd-HHMM>.zip
curl -sX POST localhost:8000/api/backups/rag/export
curl -sN localhost:8000/api/backups/rag/status   # follow it (SSE, replayed from the start)

# what's in the folder, with each bundle's restore verdict and settings drift
curl -s localhost:8000/api/backups/rag | jq '.[] | {name, restorable, reason, drift}'

# restore one — this REPLACES the corpus in both stores
curl -sX POST localhost:8000/api/backups/rag/restore \
  -H 'content-type: application/json' -d '{"name": "varagity-rag-20261108-1312.zip"}'
curl -sN localhost:8000/api/backups/rag/status

Restore replaces. The corpus's rows in both stores become the bundle's; nothing is merged. The GUI asks first whenever the corpus isn't empty (after a down -v there is nothing to replace, so it doesn't). Conversations keep their evidence snapshots either way.
One operation per corpus. Ingest, export and restore exclude each other (409 corpus_busy, naming the holder). While a restore runs, a chat question against the documents answers 503 corpus_restoring (the composer says "being restored"; ask again when it ends), document deletes answer 409, and a composer 📎 upload waits in "queued" and ingests once the restore finishes.
The corpus lock reaches the CLI. A restore holds the RAG corpus's Postgres advisory lock exclusively; every ingest holds it shared — including uv run main.py ingest and the startup ingest of main.py chat in their own process, which the API's in-process gate can't see. So during a restore a CLI ingest exits at once with one line (Not ingesting: the corpus is being restored — …, exit 1), and while a CLI ingest runs a restore answers 409 corpus_busy naming "an ingest in another process". Two ingests still run side by side as they always have, and the lock dies with its database session, so a killed process never strands the corpus.
Strict compatibility. A bundle made with another embedding model or dimension is refused (422 bundle_incompatible), never restored: neither store would notice, and retrieval would quietly return the wrong passages. A damaged bundle never restores: the restore checks every member's sha256 as it loads it, and nothing takes effect until all of them checked out. Neither the list nor the restore button hashes, so a bundle damaged inside an intact zip lists as restorable until its first restore run fails on it — with the corpus untouched; from then on (until the file changes or the API restarts) the list shows it as damaged and a restore of it is refused (422 bundle_invalid).
The stale flag and drift. Restoring a bundle made under other reingest-affecting settings (CHUNKING_STRATEGY, CHUNK_SIZE, CHUNK_OVERLAP, CONTEXTUALIZE, OCR_ENGINE) flags the corpus stale, as does a bundle that was stale when exported. The list shows that drift up front; drift never refuses a restore.
Restore on the stack mode it came from. Each document's source is re-rooted onto the restoring API's DOCS_PATH (/app/docs in compose, ./docs in host mode). A host-mode restore is therefore correct for host mode — and the compose API would then list every document flat until it restores the bundle itself.
If Elasticsearch fails after the commit, the run fails saying the rows are restored; restore the same bundle again to rebuild the BM25 index.
Not in a bundle: chat history, conversation folders and settings-drawer overrides (they live in pgdata, which down -v wipes).

The message graph¶

# export: writes graph-docs/varagity-exports/varagity-graph-<UTC yyyymmdd-HHMM>.zip
curl -sX POST localhost:8000/api/backups/graph/export
curl -sN localhost:8000/api/backups/graph/status  # snapshot → pack (bytes) → finalize

# the folder, with each bundle's verdict and the engine version that wrote it
curl -s localhost:8000/api/backups/graph | jq '.[] | {name, restorable, reason, engine: .manifest.engine}'

# restore one — this REPLACES the graph's working directory
curl -sX POST localhost:8000/api/backups/graph/restore \
  -H 'content-type: application/json' -d '{"name": "varagity-graph-20261108-1315.zip"}'
curl -sN localhost:8000/api/backups/graph/status  # validate → extract (bytes) → swap → open

What it holds. The engine's working directory verbatim — the graph, its vector indexes, the doc-status resume ledger, the LLM response cache a later incremental build rebuilds shared entities from, the build manifest (so the next Build stays a diffing upsert) and the size summary. A restored graph builds on exactly where the exported one left off.
Export while it answers. The export copies the files under the graph's write lock: Build and Re-build wait (409 corpus_busy), graph chat and /graph keep answering. It needs postgres (for the stale flag) and no model service. Deleting a source mid-export is fine — the bundle records the stale flag as it stands once the copy ends.
Restore is extract, swap, open. The bundle is extracted beside the live workdir, on the graphdata volume, then swapped in: the session closes, the two folders are renamed, and the restored graph is opened. If it won't open, the previous graph goes back and reopens, and the run fails saying so. While it runs, a graph question answers 503 corpus_restoring (the composer says "being restored"), Build waits, and deleting a graph source answers 409 corpus_busy (the restore ends by writing the bundle's own stale flag). A graph turn already streaming at the moment of the swap may end in an error — ask again. The graph-stale flag comes back exactly as it was exported.
graphdata needs room for two graphs during a restore: the extracted copy sits beside the live one until the swap commits (the live graph here is about 270 MB on disk). A full volume fails the restore cleanly — the staging copy is removed and the live graph is untouched.
Engine versions are strict. A graph bundle restores only on the same engine at the same major.minor version (lightrag-hku 1.5.x here); a 1.4.9 bundle lists with that reason and its Restore is refused (422 bundle_incompatible). After upgrading the engine across a minor version, export again.
A crash mid-swap heals itself. The swap parks folders beside the workdir (lightrag.restore-<run>, lightrag.previous-<run>); if the API dies mid-swap, the next start finishes the job — the previous graph moves back unless the swap had already committed (the moment the restored graph opened). Leave those folders alone; and as always, never point a second process at graphdata.
GRAPH_ENABLED=false answers 403 graph_disabled on the graph's list, export, restore and download (the status stream keeps reporting).

The wiped-stack drill¶

The acceptance test for the whole feature, and the thing to rehearse before you need it:

Export both corpora from /corpus (Documents, then Message graph) — or with the two POST …/export calls above — and copy both bundles off the machine. They sit in the bind mounts and survive down -v, but a backup only on the disk it backs up is not a backup.

Wipe and restart:

docker compose down -v && docker compose up -d --wait

Restore each corpus. Each /corpus tab lists its bundle (they never left the folders); with nothing to replace, Restore starts at once, without a confirmation. Time the graph restore — seconds to minutes, where its original build took days.
Ask each corpus a question. A document question and a graph question both answer with citations from the restored data.
What doesn't come back: chat history, conversation folders and settings-drawer overrides lived only in pgdata. Re-apply any overrides you depend on — and note that until you do, a bundle made under other ingest settings restores flagged stale.

Volumes and resets¶

Volume	Holds	Reset effect
`pgdata`	pgvector data — the ingested corpus + metadata, conversations, runtime setting overrides	re-runs `schema.sql` on next boot; the API re-applies all migrations on its next startup
`esdata`	the BM25 index	index recreated on next ingest
`model_cache`	Docling layout/table models + EasyOCR weights (`~/.cache`, shared by the `api` and `app` containers)	re-downloads on next PDF parse
`prefect`	the Prefect server's SQLite backing store	run history lost
`prometheus_data`	scraped time-series history	metric history lost — dashboards start empty
`grafana_data`	Grafana's internal state (sessions, preferences)	harmless — the datasource + dashboards re-provision at boot
`graphdata`	the extracted message graph (graphml + vector indexes + document statuses + our two sidecars), api-only	the graph is gone — a full rebuild from `./graph-docs` costs hours, a restore from a backup seconds

docker compose down -v drops all of them — the full factory reset, and since stage 2 that includes the message graph. The two host binds (./docs, ./graph-docs) are not volumes and always survive — and with them any backup bundles in docs/varagity-exports/ and graph-docs/varagity-exports/ (above), which bring both corpora back without a rebuild (the drill).

Schema migrations run on API startup¶

schema.sql is the fresh-install fast path: postgres runs it on first boot only (empty data dir). Everything newer — conversations, messages, message_sources, app_settings — arrives as ordered, idempotent SQL in varagity/stores/migrations/NNN_*.sql, tracked in schema_migrations and applied by the API on startup, one transaction per file (a failing migration rolls back atomically and is retried next boot). Consequences:

Upgrading a live v1 pgdata volume needs no down -v: the next api boot applies the v2 tables; the ingested corpus survives in place.
The CLI never runs migrations. CLI ingest/chat only touch the schema.sql tables, so they work regardless — but the conversation and settings routes need the migrated tables, so bring api up at least once.
If postgres is unreachable at API startup, the runner logs and skips (boot proceeds; postgres-backed routes answer structured 503s until it's back); an actual SQL failure fails startup loudly.
Keep schema.sql and the migrations in sync: fresh installs take the fast path, live volumes take the runner.

Postgres credentials freeze at first boot

The pgdata volume keeps the password from first boot. Editing POSTGRES_PASSWORD in .env later changes what clients send but not what the server expects — host TCP connections start failing auth while docker compose exec postgres psql (trust-based local socket) still works, which is confusing to debug. Fix: ALTER USER varagity WITH PASSWORD '…' inside the container, or docker compose down -v to re-initialize.

GPU & VRAM¶

There is no VRAM isolation between containers — both GPU services see real device memory, so the total must fit. Current topology on this host:

GPU	Card	Service	Steady-state VRAM
0	RTX 2080 Ti (22.5 GB)	`llamacpp`	~9.0 GB (MoE experts offloaded to CPU)
1	RTX 5060 (8 GB)	`infinity-embeddings`	e5 + reranker ONNX, batch-capped to fit

Pinning gotchas learned the hard way:

infinity's optimum engine ignores INFINITY_DEVICE_ID — pin at the Docker layer with device_ids: ["1"] in the compose reservation. Inside the container the visible device is then 0, which is what INFINITY_DEVICE_ID is set to.
llamacpp uses count: 1, which grabs the first GPU (device 0).
The llama.cpp command keeps -ot ".ffn_(up|down)_exps.=CPU" — the configured BASE_MODEL is a MoE model, and this offloads expert FFN weights to CPU. It is why an ~9B Q8 model fits in ~9 GB, and also why prompt evaluation is noticeably slower than decode (~54 tok/s decode; slow prefill is the CPU-offload signature, not a bug).

The optional gpu-metrics profile turns this table into a live dashboard — per-card VRAM/utilization in Grafana (below).

The reranker rides the embedding container¶

bge-reranker-v2-m3 is served by the same infinity instance (semicolon-separated multi-model syntax) and exposed at /v1/rerank — and since v2 it is wired into the query path (ADR-006, spec_v2 §5): RETRIEVAL_METHOD=reranked over-fetches RERANK_CANDIDATES candidates from RERANK_BASE_METHOD, cross-encodes them here, and keeps RERANK_TOP_N. RERANK_ENABLED=false is a kill switch, not a method: reranked then degrades to its base method's ranking (and logs it) — the toggle and method selection are deliberately orthogonal. Operational facts:

The engine must stay optimum on this GPU: the image's torch build has no CUDA kernels for the 5060's Blackwell sm_120 and crashes at warmup; onnxruntime works.
The ONNX must be pre-exported into the model dir (onnx/model.onnx); optimum does not export from safetensors at serve time.
INFINITY_BATCH_SIZE: '32;4' caps the reranker's batch at 4 — the default 32 OOMs an 8 GB card on a single ~2.2 GB attention buffer at warmup. Order matches INFINITY_MODEL_ID (e5 first).
GPU pinning happens at the Docker layer (device_ids: ["1"]) because the optimum engine ignores INFINITY_DEVICE_ID (above).
Only a served cross-encoder is valid at /rerank — infinity structurally rejects bi-encoders (e5, jina) there, and the rerank client surfaces that as a clear, non-retryable error instead of backing off into it. Never point RERANK_MODEL at an embedding model.
Measured cost: ≈0.66 s per query for the default 40-candidate pool on this card — charted as the rerank stage on Grafana's Query dashboard.

First-run model downloads¶

Beyond the bind-mounted LLM/embedding models, three things download on first use and are cached afterwards:

Docling layout/table models → ~/.cache/huggingface (first PDF parse).
EasyOCR weights → pinned to ~/.cache/docling/models/EasyOcr by our engine factory (first OCR fallback).
tiktoken's cl100k_base ranks file (first token count; if it can't download, counting degrades to a chars/4 estimate with a warning — ingest never fails on it).

In-container, the model_cache volume (mounted over /home/user/.cache in both the app and api containers — CLI- and GUI-triggered ingests share one cache) covers the first two, so they survive rebuilds. Expect the first PDF ingest to be minutes slower than steady state.

Containers & file permissions¶

Both Python images (Dockerfile, Dockerfile.api) run as a non-root user whose uid/gid come from build args defaulting to 1000 — the typical single-user host account. This matters twice:

./docs is a host-owned bind mount the API must write (uploads, per-document delete): a mismatched uid turns every upload into EACCES. On a host where id -u isn't 1000, rebuild with
```
docker compose build --build-arg APP_UID=$(id -u) --build-arg APP_GID=$(id -g) api app
```
api and app share the model_cache volume, so their uids must agree — always override both images together.

Both images also ship the tesseract binary (EasyOCR arrives as a Python dependency of Docling), so either OCR engine works from either front-end.

The api container runs a single uvicorn worker by design: process-local state (the in-memory ingest runner, the Prometheus registry) stays coherent, and /metrics needs a single-process registry. Scale with container replicas, not --workers.

OCR fallback operations¶

PDFs take a fast text-layer pass first; OCR (pass 2) triggers automatically when a document yields < PDF_OCR_MIN_CHARS non-whitespace chars, has ≥ PDF_OCR_TEXTLESS_PAGE_RATIO textless pages, or pass 1 raised. Chunks recovered this way carry extraction: "ocr_fallback" provenance (SELECT ... WHERE metadata->>'extraction' = 'ocr_fallback'). Bitmap images (.png/.jpg/…) skip the trigger logic entirely: they have no text layer, so the image parser always runs the same configured engine in full-page mode and its chunks carry extraction: "ocr".

flowchart TB
    pdf[("PDF")] --> force{"PDF_OCR_FORCE_FULL_PAGE?"}
    force -->|"yes · corrupt text layer"| p2
    force -->|no| p1["Pass 1 · Docling do_ocr=false<br/>(fast text-layer extraction)"]
    p1 --> trig{"trigger OCR?<br/>chars &lt; PDF_OCR_MIN_CHARS<br/>OR textless ratio ≥ PDF_OCR_TEXTLESS_PAGE_RATIO<br/>OR pass 1 raised"}
    trig -->|no| ok["chunks<br/>extraction: text"]
    trig -->|yes| p2["Pass 2 · Docling do_ocr=true<br/>OCR_ENGINE: easyocr | tesseract"]
    p2 --> got{"recovered any text?"}
    got -->|yes| okocr["chunks<br/>extraction: ocr_fallback"]
    got -->|no| guard["empty-extraction guard<br/>0-chunk documents row + warning<br/>(never a silent drop)"]

Engines: OCR_ENGINE=easyocr (default — ADR-004) or tesseract. CPU-only by design; throughput measured at ~0.10 pages/s (EasyOCR) vs ~0.55 (Tesseract) — acceptable because only textless documents pay it, in offline batch ingestion.
Both images install both engines (containers), so switching OCR_ENGINE from the GUI settings drawer works too — like any ingest-time knob, it flags the corpus stale (above).
Only PDFs ever OCR: the office/web parsers reuse the same Docling core but their text is digital by construction (spec_v2 §8).
PDF_OCR_FORCE_FULL_PAGE=true is the escape hatch for corrupt-text-layer PDFs (garbage embedded text passes the content triggers by definition): it skips pass 1 entirely and OCRs every page.
A PDF where even OCR recovers nothing ends in the empty-extraction guard: a 0-chunk documents row, a warning, and a summary count — never a silent drop.

Page previews (evidence panel)¶

Chunks from digital PDFs and PPTX decks offer "Show preview" in the evidence panel — the source page rendered server-side with the chunk's text highlighted (ADR-010). Operational facts:

Degrades per document, never 500s: the locate route answers available:false plus a machine reason, and the GUI falls back to the full-text view with a muted reason line. preview_disabled | unsupported_type | file_missing | file_changed | conversion_unavailable | conversion_failed | no_match is the full vocabulary.
file_changed means exactly that: the file on disk no longer matches the ingested content_hash (edited without reingesting). The preview refuses to render bytes that didn't produce the chunk — reingest to clear.
Host-mode API runs lose only PPTX previews: without LibreOffice on the host PATH, decks answer conversion_unavailable (expected degrade); PDF previews are unaffected. In-container, soffice ships in the api image.
The PPTX conversion cache is container-ephemeral (/tmp/varagity-preview/ inside api): a restart re-pays one soffice run per deck (~0.5–2 s warm, up to ~12 s on the very first cold call); doc_id is content-hashed, so a cache hit can never be stale.
CJK decks render substituted glyph boxes — the image ships metric-compatible Latin fonts only (fonts-noto-cjk at ~300 MB is deliberately omitted for English corpora).
Knobs are env-only (PREVIEW_ENABLED, PREVIEW_RENDER_WIDTH, PREVIEW_MIN_COVERAGE, PREVIEW_CONVERT_TIMEOUT_S) — not in the settings drawer. PREVIEW_ENABLED=false is the kill switch: the API answers preview_disabled and every chunk shows full text.

Elasticsearch notes¶

yellow is healthy on this single-node cluster; only red is a problem.
Disk watermarks: with the host disk > 90% full, ES's default percentage-based watermarks refuse to allocate new primary shards — the cluster goes red and every write hangs until timeout. The compose service deliberately keeps the default watermarks (they protect the data volume); free host disk instead. The ephemeral testcontainers stores used by tests/eval set cluster.routing.allocation.disk.threshold_enabled=false — throwaway containers must never depend on host disk pressure.
The ES client major version must match the server major (9.x ↔ 9.x); the dependency is pinned elasticsearch>=9,<10 accordingly.

Prefect¶

UI at http://localhost:4200: one flow run per ingest/question/eval — from the CLI and the API alike (peers over the same flows, spec_v2 §4.9) — with per-stage task runs, durations, and logs.
Backing store is the default SQLite in the prefect volume (ADR-003) — fine for a single-user dev stack; the official Postgres+Redis compose is a production posture this stack doesn't need.
Flows run in-process from both front-ends; there are no workers, deployments, or schedules to manage.
PREFECT_API_URL must be in the environment before prefect is imported (Prefect captures it at import time). The app handles this itself (varagity/pipeline/__init__.py); it only bites if you script against the modules directly.
The API's chat preflight deliberately excludes prefect: with no server reachable, Prefect 3 falls back to an ephemeral in-process API and questions still answer — untracked, not failed.
Orchestration-level metrics (run/state counts) come from the optional prefect-exporter profile (below).

Observability operations¶

App metrics flow api → Prometheus → Grafana; everything is provisioned — docker compose up yields working dashboards with zero click-ops (spec_v2 §6, ADR-007).

Prometheus (http://localhost:9090, host port PROMETHEUS_PORT) scrapes api:8000/metrics every 15 s. The optional exporters' targets sit in the scrape config permanently: while a profile is disabled its target simply reports up == 0 — enabling --profile prefect-exporter (flow/task-run metrics polled from the Prefect API, 60 s interval) or --profile gpu-metrics (DCGM, 30 s) needs no config edit.
Grafana (http://localhost:3001, host port GRAFANA_PORT; the container serves 3000) is fully provisioned: one Prometheus datasource (uid prometheus — the dashboard JSON references it by uid; keep it) and three dashboards — Query, Ingestion, Infra — loaded from observability/grafana/dashboards/ (mounted :ro, folder Varagity, UI edits disabled — edit the JSON and restart grafana instead). Anonymous read-only viewing is enabled (org role Viewer), so dashboards render without a login; the image-default admin/admin stands for edits — dev posture (below).
METRICS_ENABLED gates the /metrics route only — the collectors always record. false removes the route (404; Prometheus shows the varagity-api target down) without touching any pipeline code.
Metrics are per-process. The collectors live in whatever process runs the flow: a CLI ingest records into its own short-lived process and never reaches the scrape — run ingests through the GUI/API (POST /api/ingest) to populate the Ingestion dashboard. Same for queries: CLI chat is invisible to Grafana; API chat is charted.
The store-derived gauges are the exception, and the Ingestion dashboard's size panels read them (ADR-013): varagity_corpus_* queries pgvector at scrape time (so it sees CLI ingests too), and the Graph corpus row's varagity_graph_* reads the graph workdir's summary sidecar and document-status file — never the engine, never the graphml. Both cache for 10 s, serve the last good snapshot through an outage rather than 500-ing /metrics, and emit no samples at all when there is nothing yet: an empty panel means "nothing extracted", which is deliberately not the same claim as 0. Graph builds are API-only, so unlike the corpus gauges these can never miss a writer. Cross-check them against GET /api/graph/status — same source, same numbers.
dcgm-exporter consumer-card caveat: several DCGM fields (profiling metrics especially) come back empty on GeForce parts — this host's 2080 Ti / RTX 5060 included. Blank panels there are the cards, not broken wiring.

The optional exporter profiles¶

Both exporters are compose profiles — defined in docker-compose.yml but excluded from a plain docker compose up -d:

docker compose --profile prefect-exporter up -d   # Prefect flow/task-run metrics
docker compose --profile gpu-metrics up -d        # dcgm-exporter: GPU VRAM/utilization

Profiles are not sticky. Plain docker compose commands don't include a profiled service: a down + plain up -d loses the exporter (its target goes red on the Infra dashboard), and compose config changes never reach a running one — though a plain up -d does leave it running, unmanaged. To make a profile part of every invocation, set COMPOSE_PROFILES=prefect-exporter in .env (comma-separate to add gpu-metrics).
OFFSET_MINUTES — why the compose sets 1440. The exporter windows its flow-run queries to runs started within the last OFFSET_MINUTES (image default 3). Our flows are bursty and rare, so a 3-minute window reads prefect_flow_runs_total = 0 and an empty prefect_info_flow_runs nearly always — the same failure class as increase() over the ingest counters (spec_v3 §6.1). 1440 (24 h) answers "did anything run today?". prefect_flows_total is unwindowed and unaffected; prefect_deployments_total and the work-pool counts read 0 forever by design — flows run in-process, so there are no deployments or workers to count.
PREFECT_API_URL must keep its /api suffix: the exporter's health probe concatenates /health onto it and SystemExits on failure — a wrong URL is a crash-loop, not an empty panel.
A pre-start scrape error reads "server misbehaving". While the exporter container is down, Docker's embedded DNS reports the target's failed lookup as lookup prefect-exporter ... server misbehaving rather than NXDOMAIN — that is the container being absent, not a real network fault.
The exporter image ships no wget/curl — probe it via Prometheus's /api/v1/targets (or the Infra dashboard's scrape-targets panel), not from a sibling container.

Security posture (dev-only)¶

This stack is a single-user development posture — do not expose it:

Elasticsearch runs with xpack.security.enabled=false (no auth, no TLS).
PostgreSQL uses a static password from .env.
llama.cpp is unauthenticated (BASE_MODEL_API_KEY is a placeholder the SDK requires).
infinity has an API key (secret_infinity_key) but no TLS; its host binding is at least interface-specific.
The API is plain HTTP, unauthenticated, and single-user by design; CORS restricts browsers to API_CORS_ORIGINS (the local web app), but any LAN client can call :8000 directly.
Grafana keeps the image-default admin/admin with anonymous read-only viewing enabled; Prometheus has no auth at all.

Performance expectations¶

Numbers from this host (see ADRs and eval results for context):

Operation	Observed
llama.cpp model load	~30 s
Chat decode	~54 tok/s (slow prefill = MoE CPU-offload signature)
Contextualization	~8 s/chunk (one LLM call per chunk, per-document prompt-cache grouping)
Fixtures corpus ingest (16 chunks)	~42 s non-contextual; ~3 min contextual
OCR fallback	~0.10 pages/s EasyOCR / ~0.55 pages/s Tesseract (CPU)
Cross-encoder rerank	≈0.66 s per query for the 40-candidate pool (RTX 5060, ONNX)
Query (hybrid, top-10)	~7.5 s, LLM generation dominated; Prefect overhead ≈0.06 s

Troubleshooting quick reference¶

Symptom	Likely cause → fix
`llamacpp` unhealthy for ~5 min after up	Model still loading (503 is normal); check `docker compose logs llamacpp` for the `.gguf` load line
Host psql auth failures, but `compose exec postgres psql` works	Password drift — `pgdata` keeps the first-boot password (above)
ES `red`, ingest hangs/times out	Host disk > 90% (watermarks) — free disk; testcontainers are immune by config
`localhost:8081` unreachable	infinity binds a specific interface — `docker compose port infinity-embeddings 8081`
infinity crashes at warmup after GPU/engine changes	torch has no `sm_120` kernels — keep `INFINITY_ENGINE=optimum`; reranker OOM → keep the `32;4` batch cap
Config change didn't take effect on ingest	Content hashes unchanged — run `ingest --reingest` (or the GUI's stale-banner action)
Ingest warns `no served token count — cutting the document preamble…`	llama.cpp's `/tokenize` (at the server root, not under `/v1`) didn't answer, or `BASE_MODEL_API_URL` names a server that isn't llama.cpp. The file still ingests, situated in a shorter byte-safe head. Check with `curl -sX POST <server root>/tokenize -d '{"content":"x"}'`
A file fails contextualization with `exceed_context_size_error` (HTTP 400)	The prompt alone overflows `LLM_CONTEXT_TOKENS`. The document preamble is fitted with the served tokenizer, so this means a chunk far past its 2048-token allowance — a token-based `CHUNK_SIZE` set very high; lower it
GUI upload fails; `EACCES` in `api` logs	uid mismatch on the `./docs` mount — rebuild with the `APP_UID`/`APP_GID` build args (above)
Ingestion dashboard stays empty after a CLI ingest	Metrics are per-process — run the ingest through the GUI/API (above)
Changed `NEXT_PUBLIC_API_URL`, browser still calls the old origin	Build-time constant — `docker compose build web && docker compose up -d web`
Browser on another machine gets CORS errors	That origin must be in `API_CORS_ORIGINS` and baked into `NEXT_PUBLIC_API_URL` (above)
`POST /api/ingest` answers 409	One run at a time — watch `GET /api/ingest/status`, retry once it's terminal; `corpus_busy` means a backup export or restore holds the corpus (watch `GET /api/backups/rag/status`)
`main.py ingest` / `chat` prints "Not ingesting: the corpus is being restored"	A GUI/API restore holds the corpus lock — wait for it to finish (above)
Restore answers `409 corpus_busy` "an ingest in another process"	A CLI `ingest` (or `chat`'s startup ingest) holds the corpus lock shared — let it finish, then restore (above)
Restore answers `422 bundle_incompatible`	The bundle was embedded with another `EMBEDDING_MODEL` / `EMBEDDING_DIM` — restore it on a stack configured like the one that exported it (above)
First PDF ingest very slow	One-time Docling/EasyOCR model downloads (above)
Evidence panel says "preview unavailable — showing text"	Reason-dependent degrade — `file_changed`: file edited since ingest, reingest to clear; `conversion_unavailable`: host-mode API without LibreOffice, pptx-only and expected (above)
First PPTX preview slow after an `api` restart	The conversion cache is container-ephemeral — one `soffice` run per deck re-pays it (above)
PPTX preview renders □□ glyph boxes	CJK fonts deliberately omitted from the api image (above)
Graph build finished but the newest messages are missing	The `-wal` sidecar wasn't copied — re-copy all three files, re-upload, rebuild (above)
Re-uploaded a grown `chat.db`, the graph still shows the old conversation	The workdir manifest was edited or lost its record — never hand-edit it; `reingest=true` rebuilds from scratch (above)
`POST /api/graph/build` answers 409 / 403	One build at a time (`409 graph_build_running`), or `GRAPH_ENABLED=false` (`403 graph_disabled`)
A graph question answered from documents (`graph → documents` badge)	The graph degraded — kill switch off, or the session wouldn't open; check the `api` logs for `GraphUnavailable`
`/graph` says "Nothing extracted yet" after a build	The build ran bounded, or against a different `GRAPH_STORAGE_PATH` (host-mode overrides — above)
Grafana's Graph corpus row is empty	Nothing extracted yet (no samples ≠ 0), or the api is reading another workdir — compare with `GET /api/graph/status`
Flow runs missing from the Prefect UI (host run)	`PREFECT_API_URL` not set for the process — pass the localhost override
Prefect flow-run panels read 0 with the exporter up	The exporter only reports runs started within `OFFSET_MINUTES` (compose sets 1440; the image default of 3 reads 0 between runs) — check the container's env (above)
Exporter gone after a plain `docker compose up -d`/`down`	Profiles are not sticky — set `COMPOSE_PROFILES` in `.env` (above)