Pipelines

Every pipeline is a Prefect flow (varagity/pipeline/) whose stages run as tracked task runs — state, duration, retries, and logs per stage at the Prefect UI (http://localhost:4200). Business logic lives in the plain modules; the flow layer is thin @task adapters, so the same code runs with or without Prefect. Two peer front-ends invoke the same flows in-process — the CLI (main.py) and the HTTP API (varagity/api/, spec_v2 §4.9) — the API adding streaming around the tasks, never forking them.

Prefect posture

• Flows run in-process from the CLI and the API — no worker, deployment, or schedule (spec §21 #8). Every ingest/question/eval, terminal or browser, creates flow runs against the compose prefect server (SQLite-backed, ADR-003).
• PREFECT_API_URL is exported before prefect is imported (varagity/pipeline/__init__.py): Prefect captures its environment at import time, so a later export would be silently ignored. With no server reachable, Prefect 3 falls back to an ephemeral in-process API — host runs without the stack still work, just untracked.
• Result caching is disabled on every task (NO_CACHE): the stages are side-effecting calls against live services with unhashable inputs (store clients, progress displays); Prefect's default input-hash policy would log an error per run and a cache hit could never be correct.
• Flow parameter validation is off (validate_parameters=False): tests and the eval harness inject duck-typed store/client fakes that pydantic's is-instance validation would reject.
• Flow bodies double as the Prometheus probe points (spec_v2 §6.2): the query flows time each stage around its task call, record retrieval scores, and count outcomes; two ingest tasks carry the blurb-latency histogram and the corpus-growth counters (below). Recording is unconditional — METRICS_ENABLED gates only the API's /metrics route — but collectors are per-process, so only flows run inside the API process ever reach the scrape (see the runbook).

Ingestion flow (ingest)

flowchart TB
    start(["ingest_flow(docs_path)"]) --> discover["discover_documents<br/>bucket files by parser family — §9.1, spec_v2 §8.1"]
    discover --> loop{{"for each file"}}
    loop --> skip{"unchanged?<br/>(doc_id, content_hash)<br/>already in documents"}
    skip -->|"yes · skip before parse"| loop
    skip -->|no| parse["parse_document<br/>text | pdf (two-pass OCR) | office | web | image (OCR) — §9.2, spec_v2 §8"]
    parse --> chunk["chunk_document<br/>CHUNKING_STRATEGY registry, 5 strategies — §9.3, spec_v2 §7"]
    chunk --> ctx["contextualize_chunks<br/>LLM situating blurb per chunk — §9.4"]
    ctx --> embed["embed_chunks<br/>e5 passage mode, batched — §9.5"]
    embed --> store["store_chunks<br/>ES bulk first, then pg txn — §9.6"]
    store --> loop

    classDef retry stroke-width:3px,stroke-dasharray:4 3;
    class ctx,embed,store retry;

The three model/store stages (dashed border) carry Prefect retries=2; discovery/parse/chunk are local and deterministic, so they carry none (below).

• One orchestration loop. The loader (varagity/ingest/loader.py) owns the loop — idempotency skip, empty-extraction guard, per-file failure containment, original_index allocation, file-clock provenance capture (file_timestamps → file_created_at/file_modified_at on every record) — and invokes each stage through an IngestStages seam. The flow passes @task-wrapped equivalents of the same stage functions through the same loop, so plain and tracked execution cannot drift. The task bundle is public (TASK_STAGES) — the API's ingest runner wraps these very tasks (below).
• Five parser families, one loop (spec_v2 §8, ADR-009). Discovery buckets each file: .txt/.md/.rst → text; .pdf → pdf (fast text-layer pass with the automatic OCR fallback pass — runbook); the OOXML families, .csv, and OpenDocument → office; .html/.htm/.xhtml → web; bitmap images → image (no text layer exists, so OCR always runs and chunks carry extraction: "ocr"). Office and web share the PDF parser's Docling core but never OCR — their text is digital by construction, so every chunk carries extraction: "text".
• Five chunking strategies (spec_v2 §7). chunk_document dispatches on CHUNKING_STRATEGY: recursive_character (the benchmark-kept default — ADR-008), token_based, markdown_aware, semantic, docling_hybrid. CHUNK_SIZE's unit is per-strategy: characters for recursive_character/markdown_aware, tokens for token_based/docling_hybrid (and for semantic's re-split ceiling — it splits on embedding-similarity boundaries, not a size budget). markdown_aware additionally stamps each chunk's heading_path breadcrumb (e.g. Harbor Operations > Dredging) into its metadata, carried by ChunkRecord into both stores.
• One preamble per document. contextualize_chunks fits the document into the situating prompt's budget once, before its first chunk (fit_document_preamble in varagity/context/contextual.py), and hands that one preamble to every chunk: byte-identical prompt prefixes for llama.cpp's cache, and the served token count is paid per document, not per chunk. The budget is held in the served model's tokens. A document of at most budget-many UTF-8 bytes fits as-is, uncounted. Otherwise cl100k cuts first, then llama.cpp's /tokenize (server root, not /v1) counts the cut and re-cuts it in proportion while it is over — typically one re-cut for digit-dense text, which the served model splits one digit per token. With no served count (server unreachable, not llama.cpp) the preamble is the document's first budget-many UTF-8 bytes, which fits any tokenizer: a shorter context, never a failed file.
• Retries in two layers, different scopes. The model/store clients retry transient HTTP failures within one call (tenacity); the contextualize/embed/store tasks carry retries=2 with exponential backoff to re-run the whole stage after the client gives up — e.g. an Elasticsearch restart mid-ingest. Both store writes are idempotent, so a stage re-run is safe. Discovery/parse/chunk are local and deterministic; retrying them cannot help, so they carry none.
• Failure containment. One file raising is logged, counted (failed in the summary table), and visible as a failed task run — it never aborts the corpus. A file with no extractable text gets a 0-chunk documents row and a dedicated summary count (never silently dropped).
• Ordering inside store_chunks: Elasticsearch bulk first, pgvector transaction last — the pg documents row is the idempotency marker, so a failure in between leaves the file re-attemptable (see Data model).
• The corpus lock, shared. Every run holds the RAG corpus's Postgres advisory lock shared, from before its first write (the BM25 index create) until it returns. Ingests overlap each other exactly as before; a backup restore holds the lock exclusive (below), so an ingest in any process — the CLI's ingest, or chat's startup ingest — and a restore exclude each other. A refused run raises CorpusLocked before it writes anything; the CLI prints one line and exits 1. The lock belongs to the run's database session, so a killed process releases it with its connection.
• Prometheus probe points: contextualize_chunks feeds varagity_contextualize_latency_seconds per document (real blurb generation only — the CONTEXTUALIZE=false identity path doesn't count), and store_chunks bumps the document/chunk counters after both writes land, so corpus-growth metrics count documents that stored, not attempts.

API-triggered ingest

POST /api/ingest runs the same flow: the API starts ingest_flow on a daemon thread inside its own process and returns a run handle immediately (202); GET /api/ingest/status streams the run as SSE (spec_v2 §4.2). The runner (varagity/api/ingest_runner.py) wraps the flow's public TASK_STAGES seam with event emitters — the Prefect tasks (tracking, retries, metrics) are untouched; each stage transition just also emits a frame. Tracking and streaming compose; there is no second pipeline.

• Event protocol: status (run snapshot — the first frame, and the terminal frame carrying the summary counters) · progress (one per discover/parse/chunk/contextualize/embed/store/file_done transition, plus per-chunk contextualize ticks via a proxy over the loader's rich progress handle — browser granularity equal to the terminal's on the ingest's long pole) · log (relayed varagity.ingest records: skips, no-text warnings, failures).
• Replay from frame one: events accumulate per run, and a subscriber always receives the full backlog before the live tail — connecting mid-run, or after completion, renders the same picture. An API process where nothing ever ran answers a single idle status frame and closes.
• One run at a time: a second POST while one is in flight is a structured 409 ingest_already_running. A preflight rejects with a structured 503 when a required backing service is down (llamacpp is required only while CONTEXTUALIZE is on).
• The stale-flag hook: a completed reingest=true API run clears the persisted corpus-stale flag (runbook); a CLI ingest --reingest runs in another process and does not.
• Because the flow runs in the API process, the ingest Prometheus counters actually reach the scrape — a CLI ingest records into a registry that dies with the process.

Query flows (query / query-stream)

flowchart TB
    start(["query_flow(query, history) · query_stream_flow(query, history)"]) --> condense["condense_query<br/>chat engine: simple | condense_context — spec_v3 §4.2<br/>PreparedQuery: search_query / original_query"]
    condense --> embed["embed_query<br/>retriever.encode_query(search_query) — None for bm25<br/>(hyde: LLM passage → passage-mode embed — ADR-016)"]
    embed --> retrieve["retrieve<br/>top-k via semantic | bm25 | hybrid | reranked | hyde<br/>(reranked: over-fetch → cross-encode → cut — spec_v2 §5)"]
    retrieve -.->|on_retrieved| hook["CLI matches table ·<br/>SSE retrieval event"]
    retrieve --> generate["generate_answer<br/>grounded, cited answer — §10.2 prompt"]
    retrieve --> generate_s["generate_answer_stream<br/>on_delta per fragment · should_abort<br/>polled between deltas — spec_v2 §4.3"]
    generate --> state(["QueryState — §10.1<br/>{query, query_vector, retrieved,<br/>formatted_context, answer}"])
    generate_s --> state_s(["StreamedQueryState<br/>… + {reasoning, aborted, usage}"])

    classDef sidecar stroke-dasharray:4 3;
    class hook sidecar;

Both flows share the staging — condense → embed → retrieve → generate, with on_retrieved firing between retrieve and generate — and return the spec §10.1 state dict (which since v3 carries prepared: PreparedQuery); query_stream_flow extends it as StreamedQueryState with reasoning, aborted, and usage.

• The condense stage is always in the graph (v3, spec_v3 §4.2; ADR-011): condense_query runs the configured chat engine over the question and its conversation history and returns the PreparedQuery two-string split — everything downstream retrieves with search_query while the answer prompt always gets original_query, the user's verbatim words. Under simple (the default) it is the identity split in ~3 ms with no LLM call; condense_context makes one non-streaming LLM call to resolve a follow-up against history — and falls back to the raw query on any failure (empty, over-length, or a raised call — logged at WARNING, the turn never dies there). The first turn never condenses: empty history is the identity path.
• Query embedding is its own tracked stage via the retrievers' encode_query() seam; the vector is handed to retrieve so nothing is encoded twice. For bm25 it is None (nothing to encode).
• reranked composes inside retrieve (spec_v2 §5, ADR-006) rather than adding a stage: it over-fetches a pool of max(RERANK_CANDIDATES, k) from RERANK_BASE_METHOD (default hybrid), cross-encodes every candidate's original content against the query at infinity's /v1/rerank, and keeps top_n = min(k, RERANK_TOP_N) — re-ranking narrows; it never invents candidates. Each survivor's RetrievalTrace gains rerank_score, rerank_delta (+ moved up / − moved down), and final_rank — the evidence panel's rerank +2 badges — and its score becomes the cross-encoder relevance. RERANK_ENABLED=false is a kill switch, not a method: reranked then passes the base ranking through the same cut and logs the degradation. The rerank sub-stage is timed into Prometheus on its own (stage="rerank"), and the flow's retrieve observation deliberately includes it — total retrieval latency is what the user feels; rerank's share is subtractable.
• hyde composes inside encode_query (ADR-016): the embed stage's retriever.encode_query(search_query) is where the hypothetical answer passage is generated (one non-streaming LLM call, <think>-stripped via clean_response — the condense lesson) and embedded in e5 passage mode; retrieve then hands HYDE_BASE_METHOD the original query text plus that vector — dense-arm-only substitution, so a hybrid base's BM25 arm keeps exact keyword recall and traces pass through untouched. The generation sub-stage is timed into Prometheus on its own (stage="hyde") and the flow's embed observation deliberately includes it, mirroring the rerank sub-stage. Every unusable generation (raised call, empty, overlong) and HYDE_ENABLED=false degrade to the base method's raw-query retrieval at WARNING — a degraded search still answers; a failed turn answers nothing. Pairing with the cross-encoder stacks rerank outside (RETRIEVAL_METHOD=reranked + RERANK_BASE_METHOD=hyde), so the reranker judges the user's real query.
• query_stream_flow is the streaming twin backing POST /api/chat (spec_v2 §4.3): identical embed/retrieve staging, then generate_answer_stream_task hands each delta to on_delta while it runs. on_retrieved firing before generation is what puts the SSE retrieval event (chunks with traces, method, top_k, reranked_to) on the wire before any answer token — evidence before prose. The generate stage stays a tracked Prefect task while tokens flow: the task boundary, run log, and metrics are preserved; streaming is a callback out of the task, not a bypass of it.
• Deltas are classified, not raw: the LLM client's ThinkStreamSplitter labels each fragment reasoning (inside <think>…</think>) or answer, and the API maps them 1:1 onto SSE reasoning/token events. The wire order is retrieval → reasoning/token (interleaved, stream order) → done | error.
• Client disconnect frees the GPU: should_abort is polled between deltas, so tearing down the SSE stream stops generation at the next token. An abort is a deliberate stop, not a failure — the task completes normally with aborted=True (counted as outcome="aborted"), and the API persists nothing for the turn.
• No Prefect-level retries on the query path: it is interactive, the clients already retry transient HTTP failures internally, and stacked backoff would multiply the wait before a hard failure surfaces at the prompt.
• No Prefect-level retries on condense_query either — the same convention, sharpened by the fallback: a condense failure must fail fast into the raw-query fallback, not retry into the user's latency budget.
• Prometheus probe points: both flow bodies time condense/embed/retrieve/generate around the task calls (varagity_query_latency_seconds{stage, method}), record the returned scores (varagity_retrieval_score), and count the outcome (varagity_query_total{outcome=ok|aborted|error}); the streaming flow additionally counts the server-reported token usage (varagity_llm_tokens_total).
• Measured overhead of flow tracking: ≈0.06 s against a ~7.5 s question — LLM generation dominates.

The graph flows (graph-build / graph-query-stream)

The message graph is a peer corpus, so it gets peer flows (varagity/pipeline/graph_flow.py) — one that extracts it and one that answers from it. Both are API-only: the engine's storages are single-writer per working directory and the API process owns them, so there is deliberately no CLI graph build.

flowchart TB
    start(["graph_build_flow(service, batches)"]) --> scan["scan + parse<br/>message sources sniff the corpus dir"]
    scan --> render["render thread-day transcripts<br/>pure — identical days render identical text"]
    render --> diff["manifest diff<br/>new · changed · unchanged · removed"]
    diff --> del["delete changed + removed<br/>(the engine's enqueue dedup can't update)"]
    del --> enqueue["enqueue pending<br/>durable PENDING doc-status rows"]
    enqueue --> process["process<br/>LLM extraction, batched, status-flushed"]
    process --> sidecars["rewrite manifest + summary"]

    classDef durable stroke-width:3px;
    class enqueue,process durable;

• One flow run per build attempt, not per document. The engine enqueues into its own durable status store and processes in batches inside a single call; per-document task runs would mean reaching inside its pipeline. Per-document progress is reported instead by the runner's status sampler, which reads the engine's own document counts — the same tracking-and-streaming composition the ingest runner uses.
• Resume is the same call. Every in-flight or failed document (pending, parsing, analyzing, processing, failed) is re-selected at the top of every batch, and statuses flush to disk at each stage boundary. A backfill killed 14 hours in resumes by pressing Build again — nothing is re-extracted, and the run reports what it inherited as resumed. That is why reingest is a separate, explicit flag: it is the one path that throws the working directory away.
• The manifest diff is the upsert. Rendering is pure, so an unchanged thread-day hashes identically no matter how often its chat.db is re-uploaded and costs nothing. A changed day is deleted from the graph and re-inserted (the engine drops a re-submitted id in any status, so without this the stale transcript would be permanent); a removed day is retracted on a full-corpus render, while a bounded build — partial by construction — retracts only its re-span casualties (below) and keeps the rest: pruning out-of-window keys on a partial render's say-so would erase the archive. A delete the engine refuses keeps its old manifest record, so the next build still sees it as stale.
• Bounded builds are the escape hatch (message_limit, since): a full backfill runs for hours (7.17 s/message measured — see the runbook), so a spot check on the newest N messages, or a dated slice, is how the corpus is smoke-tested before committing days of GPU time. Both imply prune_removed=False.
• Growing a bounded window re-keys documents, and the build cleans up after it. Day-span packing is greedy, so widening the window backward shifts a thread's pack boundaries and downstream documents re-key: their old keys turn up removed although nothing left the archive (observed on a 5k→10k grow: 109 new, 48 unchanged, 38 removed). A bounded build deletes such a key only on proof of full guid coverage — every message its manifest record holds is re-indexed under new keys by this very render — which is what separates a re-span casualty (a duplicate transcript, safe to drop) from genuinely out-of-window content (kept, per the rule above). Ghost keys left by grows that ran before this sweep existed are caught the same way by the next bounded build that covers their messages.
• No Prefect-level retries. An engine that fails mid-backfill must surface rather than silently re-run hours of extraction — and a re-called build resumes from the durable statuses anyway.

The query path is query_stream_flow with exactly one stage swapped:

flowchart LR
    q(["graph_query_stream_flow(query, service)"]) --> condense["condense_query<br/>the SAME task — spec_v3 §4.2"]
    condense --> gret["graph_retrieve<br/>entities · relations · transcript days"]
    gret -.->|on_retrieved| sse["SSE retrieval event<br/>(evidence before prose)"]
    gret --> gen["generate_answer_stream<br/>the SAME task, graph-shaped context"]
    gen --> state(["StreamedGraphState"])

    classDef sidecar stroke-dasharray:4 3;
    class sse sidecar;

• Condense and generate are the same tasks the chunk path runs, so a graph turn inherits the search/answer query split (the user's verbatim words always reach the answer prompt), the [SOURCE] citation contract, <think> splitting, client-abort handling, and token accounting — with no second implementation of any of them. Only the middle stage differs, because only the middle stage is different.
• Embed + retrieve collapse into one task. The engine owns its own keyword extraction, embedding, and fusion inside a single call (GRAPH_QUERY_MODE, default mix); splitting the wrapper would invent task boundaries the engine does not have.
• Reads never wait for a build. graph_retrieve takes no write lock, so a question is answered during a backfill — it queues only behind the in-flight extraction call for the single llama.cpp slot.
• The answer is ours, not the engine's (ADR-017): the retrieval-only decision means the engine returns evidence and varagity/graph/answer.py renders the grounding context (budgeted against LLM_CONTEXT_TOKENS) that the ordinary streaming generate task writes from.

API-triggered graph builds

POST /api/graph/build is the ingest runner's shape, one corpus over: a daemon thread inside the API process, an immediate 202 handle, and GET /api/graph/build/status streaming the run as SSE with full backlog replay — a browser opened six hours into a backfill renders the same picture one opened at the start did. progress frames walk scan → parse (per file) → bound/reset → index → sampled process ticks; log frames relay the varagity.graph logger.

• One build at a time: a second POST is a structured 409 graph_build_running; a preflight rejects with 503 when a model service is down. GRAPH_ENABLED=false refuses with 403 graph_disabled.
• One writer per corpus (ADR-019): a build claims the graph corpus's gate for its whole run, so a build during a graph backup export is 409 corpus_busy, and an export during a build is refused the same way. The reingest=true wipe runs under the service's write lock too, so no backup can ever copy a workdir being removed. The build's scan never counts the reserved graph-docs/varagity-exports/ folder.
• Killing the process is safe: the durable statuses are the run's memory, and both sidecars are written atomically.
• The graph-stale flag is cleared only by a completed API-driven reingest=true run — deleting a source file sets it, and the tab says so until a rebuild retracts those messages.

The backup flows (rag-export / rag-restore / graph-export / graph-restore)

rag-export (varagity/pipeline/backup_flow.py) writes the chunk corpus into a backup bundle (Data model), and rag-restore turns a bundle back into the corpus; graph-export writes the message graph's workdir into one (below), and graph-restore swaps a bundle's workdir back in (below). All four are thin @flow shells over the backup package (varagity.backup.rag's export_rag and restore_rag, varagity.backup.graph's export_graph and restore_graph), for the same reason graph-build is one: the work belongs to the backup package, while Prefect owns the record that a backup ran, how long it took, and whether it failed (ADR-019).

flowchart LR
    start(["rag_export_flow(dest, ingest_settings)"]) --> stale["read _corpus_stale<br/>(app_settings)"]
    stale --> snap["iter_corpus()<br/>one REPEATABLE READ snapshot"]
    snap --> docs["documents.jsonl<br/>+ relative_path"]
    docs --> chunks["chunks.jsonl<br/>vectors spooled beside it"]
    chunks --> fin["embeddings.f32 · bundle.json<br/>fsync · os.replace"]

• One flow run per export, no tasks. An export is a single consistent read. Splitting it into task runs would invent boundaries the snapshot doesn't have. Stage progress travels as the API runner's progress events instead: snapshot → documents → chunks → finalize.
• No model service, no retries. The flow reads postgres and writes one file. A failure removes the partial bundle, so the honest response is to surface it and let the user press Export again, not to re-read the corpus silently.
• One pass, aligned by construction. A zip takes one member at a time, so the vector matrix is spooled to an unnamed temp file beside the bundle while chunks.jsonl streams, then copied in as embeddings.f32 (stored, not deflated, because float32 noise doesn't compress). Row i is line i's vector because both came from the same row of the same snapshot.
• API and GUI only. There is deliberately no CLI export command. It matches the graph build's posture, and keeps one shape for both corpora.

The restore (rag-restore)

flowchart LR
    start(["rag_restore_flow(bundle, ingest_settings)"]) --> validate["validate<br/>the bundle's verdict again"]
    validate --> lock["corpus_lock(exclusive)<br/>advisory, until BM25 is done"]
    lock --> txn["load · index — one transaction<br/>TRUNCATE · DROP INDEX · binary COPY ×2<br/>CREATE INDEX · ANALYZE · COMMIT"]
    txn --> bm25["bm25<br/>recreate the index · streaming_bulk · refresh"]
    bm25 --> flag["write _corpus_stale<br/>(bundle flag, or settings drift)"]

• No model call. Nothing is parsed, chunked, contextualized or embedded: the bundle's lines stream straight into binary COPY (vectors in pgvector's own binary form, bit for bit), each document's source re-rooted onto this stack's DOCS_PATH.
• The commit is where the restore takes effect. One transaction empties documents and chunks, drops the HNSW index (its definition saved from pg_indexes), loads both tables, rebuilds the index and analyzes. Any failure before the commit — a malformed line, a constraint the bundle breaks, a member whose sha256 turns out wrong as it is read — rolls back and leaves the previous corpus untouched. TRUNCATE holds both tables until then, so readers wait for the restored rows rather than seeing half.
• Then BM25, from the restored rows. The index is deleted and created anew with its mapping, and the rows stream in through helpers.streaming_bulk. If Elasticsearch fails here the run fails, but the rows are restored — restoring the same bundle again repairs the index.
• The exclusive corpus lock is held from before the TRUNCATE until the BM25 rebuild ends. Without it, a CLI ingest could write rows the rebuild then misses, or index into the gap between the index's delete and its re-create — where Elasticsearch would auto-create it with default mappings that create_index() then keeps for good.
• The stale flag comes from the bundle, written only after everything succeeded: stale if the bundle was stale at export, or if its rows were made under other reingest-affecting settings than this stack's. The runner passes a provider of those settings, read when the flag is written, so a settings change made mid-restore is judged against the restored rows too.
• No retries, for the export's reason: a failure is either a rollback or says what it left, so the user presses Restore again.

The graph export (graph-export)

flowchart LR
    start(["graph_export_flow(dest)"]) --> lock["snapshot<br/>GraphService.writing() · list the workdir"]
    lock --> pack["pack<br/>each file → &lt;engine&gt;/name, deflated · counted in bytes"]
    pack --> fin["finalize<br/>read _graph_stale (app_settings) · bundle.json · fsync · os.replace"]

• The write lock does the hard part. The flow copies the graph's working directory (Data model) under GraphService.writing(), the lock every build, resume and delete takes, so no file changes mid-copy; reads take no lock, so graph chat and /graph keep answering. The engine's *.tmp* files are skipped, and so is anything a bundle can't carry (each named in a relayed warning).
• The engine names its own format. The manifest records GraphEngine.storage_format() — for LightRAG, lightrag-hku and its installed version, read from package metadata, so the backup code never hardcodes the library and the registry stays import-light. The counts come from the workdir's size summary, never from parsing the graphml.
• No model service, no session, no retries. The flow copies files the API itself owns and never opens the engine. It reads the stale flag from Postgres last, just before sealing, because deleting a graph source sets it without taking any lock: a source deleted mid-export is still recorded. A failure removes the partial bundle, and the user presses Export again. Progress travels as snapshot → pack (bytes copied of the files' total, about 50 frames) → finalize.

The graph restore (graph-restore)

flowchart LR
    start(["graph_restore_flow(bundle, run_id)"]) --> validate["validate<br/>the bundle's verdict again"]
    validate --> extract["extract<br/>&lt;engine&gt;/* → lightrag.restore-RUN · sha256 re-checked · synced"]
    extract --> swap["swap<br/>write + open locks: close · lightrag → .previous-RUN · staged → lightrag"]
    swap --> open["open<br/>session().stats() · commit: .previous → .replaced · delete it"]
    open --> flag["write _graph_stale<br/>(the bundle's own)"]

• No model call, no re-extraction. The bundle is a workdir: its files are extracted into a staging folder beside the live one, on the graphdata volume, so the renames that follow are atomic. Each member's sha256 is checked again as it is read and each file is synced, so a bundle that changed since the route verified it can't go live. Progress counts the bytes extracted.
• The swap holds both of the service's locks. GraphService.replace_workdir keeps the write lock for the whole swap (no build, resume, delete or export can start) and the open lock around the renames, so no lazy open — a status poll, a /graph read — can land on a half-swapped folder; one that arrives mid-swap waits, then opens the restored graph. Closing the session first finalizes the engine's storages into the old files and resets LightRAG's process-global state, so the open reads the new files from disk.
• The open is the check, and the commit follows it. Opening asserts the vectors' embedding_dim; if it (or reading the graph's size) fails, the swap is undone — the previous graph returns and reopens, and the run fails. Once it opened, renaming .previous-RUN to .replaced-RUN commits the restore, and only then is the replaced graph deleted. A crash before the commit is rolled back by the next process's GraphService construction; one after it just has its debris removed.
• The stale flag is the bundle's own. The graph is exactly what was exported, so _graph_stale is set to the bundle's stale — written only once the swap committed.
• No retries: a failed restore left the previous graph in place, so the user presses Restore again.

API-triggered exports and restores

POST /api/backups/{corpus}/export and POST /api/backups/{corpus}/restore are the ingest runner's shape again: a daemon thread inside the API process, an immediate 202 handle, and GET /api/backups/{corpus}/status streaming the run as SSE with full backlog replay (HTTP API). Each corpus (rag, graph) has its own runner and gate, so a graph export and a RAG restore run side by side.

• The corpus gate. BackupRunner (varagity/api/backup_runner.py) claims the corpus's CorpusGate (varagity/api/corpus_gate.py) for the whole run, and so do the ingest runner (RAG) and the graph build runner (graph). A second backup run, an ingest or build during a backup, or a backup during an ingest or build is refused with 409 corpus_busy, which names the holder (a second ingest or build keeps its own 409 ingest_already_running / graph_build_running). The gate is the one-at-a-time rule inside the API process; for RAG, the advisory corpus lock (above) is its cross-process half — the graph needs none, since the API is its only writer.
• A restore refuses what it can see before it starts. The route checks the gate, the bundle's cheap verdict (for a graph bundle the engine's format is judged too) and any damage an earlier restore of the file found, and for RAG the corpus lock, before the 202 — a busy corpus or an unsound bundle is a clean 409/422. The member hashes are the run's: it verifies every member as it loads it (one read of the bundle), so a bundle damaged inside an intact zip fails its first run with the corpus untouched — and the runner remembers it, so the list shows it damaged and the route refuses it from then on.
• Each run relays only its own lines. Both corpora's runs log to the one varagity.backup namespace and may run side by side; a context variable marks the worker's records so each feed carries its own run's, and the namespace's INFO pin is shared by count.
• What the runner hands the flow. For RAG it captures the reingest-affecting settings at start (in app_settings' env-string form) and passes them in — a provider of them to a restore — so varagity.backup never imports the API layer; the export flow reads the stale flag itself, after those settings. A graph restore gets the run's id instead, which names the folders the swap parks beside the workdir.
• The saved path. A completed run reports bundle and saved_to (docs/varagity-exports/<name> or graph-docs/varagity-exports/<name>), relative to the checkout. The API sees /app/docs in compose, but the user's file manager sees ./docs, so the corpus folder's own name is the path that reads right in both modes.
• While a restore runs, chat against that corpus answers 503 corpus_restoring (a graph turn before its session is opened), and its deletes answer 409 corpus_busy.
• The graph's kill switch. While GRAPH_ENABLED=false, the graph's list, export, restore and download answer 403 graph_disabled (before any store is touched); its status stream keeps reporting.

Evaluation flows

Thin flows (eval-matrix, eval-ocr, eval-chat, eval-graph) over the spec §16 harness (varagity/eval/); each eval ingest is a tracked subflow with per-stage task runs. All need the live GPU services; all but eval-graph also need Docker (ephemeral testcontainers stores — the graph harness stores nothing in pg or ES).

• eval-matrix (main.py eval): recall@k / pass@k (k ∈ {5, 10, 20}) across the seven-configuration ladder — (1) semantic non-contextual, (2) semantic contextual, (3) contextual BM25, (4) hybrid contextual, (5) hybrid + rerank contextual (spec_v2 §5.5 — the ≈67% tier), (6) HyDE → hybrid contextual, (7) HyDE → hybrid + rerank contextual (the ADR-016 pairing). Two ingests into ephemeral testcontainers stores cover all seven configs (ingest A non-contextual → config 1; ingest B contextual, reingest → configs 2–7); embeddings, the contextualizing LLM (which also writes the HyDE passages — one generation per query per HyDE config), and the reranker are the live GPU services (stateless). The live corpus is never touched (ADR-003).
• eval-ocr (main.py eval ocr): the OCR engine benchmark behind ADR-004 — per engine: CER/WER against the fixtures' known ground truth, pages/sec, and (supplementary) retrieval recall on the scanned-doc golden queries with the engine as the only variable.
• eval-chat (main.py eval chat, v3 — spec_v3 §4.9, ADR-011): the multi-turn chat-engine eval that decided the shipped CHAT_ENGINE default. One contextual ingest into ephemeral stores backs the run; every registered engine (enumerated from the registry, like the chunker sweep) plays the hand-built conversation fixtures (tests/fixtures/conversations/ — pronoun follow-ups, elliptical refinements, and topic shifts, each turn tagged and fact-anchored), with scripted assistant replies so both engines see byte-identical history and zero answer-generation calls are paid. Each turn's engine-produced search_query is scored fact-anchored under both hybrid and reranked at k ∈ {1, 3, 5} (shallow on purpose — the 16-chunk corpus makes recall@20 meaningless), with condense latency recorded per turn. The condense settings are pinned so a host env with the kill switch off can't silently measure condense_context as simple. The 2026-07-19 verdict: condensing wins the pronoun slice decisively (recall@1 1.000 vs 0.600 under reranked) and never drags a topic shift, but costs a mean 8.6 s pre-retrieval LLM call on follow-ups — the default stays simple, a measured "no".
• eval-graph (main.py eval graph — ADR-017): the harness that decided the graph engine and now guards it. It builds a synthetic (or real) message corpus into each registered engine's own working directory under data/eval/graph/ and scores the answers against a golden QA set — production's GRAPH_STORAGE_PATH is never touched, and neither store container is needed. --skip-build re-scores an already-built graph in seconds, which is what makes a retrieval-side change (query mode, the e5 query prefix, a rerank experiment) one run from a number. Details and profiles in the runbook.

How eval-matrix covers all seven configs with only two ingests into the throwaway stores — and where the chunker sweep rides:

flowchart LR
    gpu["live GPU services<br/>embeddings + contextualizing LLM + reranker<br/>(stateless — never touch the live corpus)"]

    subgraph tc["ephemeral testcontainers · pg + ES (ADR-003)"]
        A["ingest A<br/>CONTEXTUALIZE=false"] --> c1["config 1<br/>semantic non-contextual"]
        B["ingest B<br/>CONTEXTUALIZE=true, reingest"] --> c2["config 2 · semantic contextual"]
        B --> c3["config 3 · contextual BM25"]
        B --> c4["config 4 · hybrid contextual"]
        B --> c5["config 5 · hybrid + rerank contextual"]
        B --> c6["config 6 · HyDE → hybrid contextual — ADR-016"]
        B --> c7["config 7 · HyDE → hybrid + rerank contextual"]
        B --> sweep["chunker sweep — spec_v2 §7<br/>reingest per strategy ×<br/>{semantic, bm25, hybrid, reranked}<br/>(no HyDE re-measure: the passage depends on the query, not the boundaries)"]
    end

    gpu -.-> A
    gpu -.-> B
    gpu -.-> sweep

The chunker sweep (spec_v2 §7, ADR-008) follows the matrix in the same run: every registered strategy is ingested contextually (recursive_character reuses ingest B's index; the other four reingest over the same throwaway stores) and measured across all four retrieval methods. Golden refs are re-resolved by fact per strategy (resolve_golden_by_fact): the golden set's chunk_index anchors are authored against the default boundaries and are meaningless under a foreign chunker, so each ref's literal fact snippet is located in the strategy-true chunks, and a ref counts as retrieved when any fact-carrying chunk is in the top-k. A fact that matches no chunk is reported as unresolved and scored as a guaranteed miss — never silently dropped.

Results are rendered as rich tables and persisted as timestamped JSON under data/eval/results/ (gitignored) for regression comparison. Metric semantics: recall@k is the cookbook's headline number (per-query fraction of golden chunks in the top-k, averaged); pass@k is the strict complement (share of queries with all golden chunks in the top-k). The sweep's variants are the same two numbers with each golden ref generalized from one chunk id to its acceptable fact-carrying set.

The fixtures corpus saturates the matrix

At 16 chunks, k ≤ 20 ≥ corpus size, so every config scores 1.000 at k ∈ {10, 20} — the harness is proven, but the numbers only become a quality signal on a larger corpus (the ladder's honest deliverable is reranked ≥ hybrid, which holds). The planned follow-up is the Anthropic cookbook dataset (737 chunks / 248 queries), which needs a pre-chunked ingest path and a second golden loader (deferred — spec_v2 §5.5, §15).

CLI entry points

Command	Flow(s)	Behavior
uv run main.py ingest [--reingest]	ingest	One tracked ingest run; exit 1 if any file failed — or, with one line, if a backup restore holds the corpus
uv run main.py chat (default)	ingest, then query per question	Spec §13 startup sequence: ingest first, then the Q&A loop (:quit exits). Since v3 the loop threads an in-memory history into each turn (session-scoped, cleared by :quit, never persisted) so the configured chat engine can condense
uv run --group eval main.py eval	eval-matrix (+ ingest subflows)	7-config retrieval matrix + the chunker sweep, on ephemeral stores
uv run --group eval main.py eval ocr	eval-ocr (+ ingest subflows)	OCR engine benchmark (CER/WER, pages/s, recall)
uv run --group eval main.py eval chat	eval-chat (+ ingest subflow)	Multi-turn chat-engine eval over the conversation fixtures (above)
uv run main.py eval graph	eval-graph	GraphRAG regression harness over its own throwaway workdirs (no --group eval, no stores — runbook)

The HTTP API is a peer front-end over the same flows (spec_v2 §4.9): POST /api/chat runs query-stream — or graph-query-stream when the turn targets the graph — in a threadpool under the async edge (ADR-005), and POST /api/ingest / POST /api/graph/build run ingest / graph-build on a daemon thread (above), as the backup routes run rag-export / rag-restore / graph-export / graph-restore. Runs from either front-end land at the same Prefect UI. The graph build is one flow the CLI cannot start — single-writer storage means the API process owns it — and the backup flows are API-only by choice, so both corpora keep one shape.