Skip to content

Data model

Storing complete, queryable metadata for every chunk is a hard requirement (spec §8). This page documents the schema as built, including the two deliberate v1 amendments over the spec: the relative-path doc_id and the unique identity index (both ADR-003). v2's changes are additive (spec_v2 §9): the chunk/vector tables keep their shape; conversation persistence and runtime-settings tables arrive through the migration runner, and query-time rank provenance rides the RetrievalTrace — a record that is never stored on chunk rows.

Identity derivation

Implemented in varagity/stores/records.py:

content_hash = sha256(file_bytes)                                  # bytes, not parsed text
doc_id       = sha256(f"{relative_path}:{content_hash}")[:16]      # path relative to DOCS_PATH
chunk_id     = f"{doc_id}::{chunk_index}"
  • content_hash hashes raw bytes so unchanged files are skipped before paying the parse cost, and so doc_id is stable across OCR engines (bytes, not extracted text).
  • doc_id hashes the relative path (spec §8.1 said absolute). Absolute paths differ between host (/home/…/docs/a.md) and container (/app/docs/a.md) and across machines — hashing them would break idempotency and make golden eval sets non-portable. The absolute path is still recorded in source.
  • (doc_id, original_index) is the cross-store join/fusion identity; original_index is a global monotonic counter allocated per ingest run (SELECT COALESCE(MAX(original_index), -1) + 1, then incremented in-process).

Chunk metadata (ChunkRecord)

Every chunk persists this record (pydantic-validated; stored whole in the chunks.metadata JSONB column and mirrored into typed columns where queried):

Field Type Description
doc_id str Stable per-document id (see above)
chunk_id str f"{doc_id}::{chunk_index}" — pg primary key and ES _id
original_index int Global monotonic chunk index (fusion key)
chunk_index int Chunk position within its document
source str Absolute file path (provenance only — never identity)
file_name str Basename
file_type str File extension without the dot: pdf / txt / md / docx / pptx / xlsx / html / htm
page int? First page that contributed text (Docling-parsed formats; None otherwise)¹
content str Original chunk text
context str? LLM situating blurb (None when ingested with CONTEXTUALIZE=false)
contextualized_content str context + "\n\n" + content — the text actually embedded & BM25-indexed; equals content when context is None
chunk_size / chunk_overlap int Parameters used, in the strategy's unit — characters for recursive_character/markdown_aware, tokens for token_based/docling_hybrid/semantic (provenance)
chunking_strategy str Registry name of the chunker used, e.g. recursive_character
embedding_model str Served model name, e.g. infloat/multilingual-e5-large-instruct
n_tokens int Approximate token count of content (cl100k — a documented approximation; the e5 tokenizer differs)
content_hash str The parent document's byte hash
created_at datetime Ingestion timestamp (UTC)
file_created_at datetime? Filesystem birth time of the source file (UTC) — best-effort: os.stat exposes it on macOS/Windows; on Linux a GNU-coreutils stat -c %W fallback reads it, and None means the filesystem records none (a copy/download also resets it)
file_modified_at datetime? Filesystem mtime of the source file (UTC) — the document's clock, distinct from created_at (the ingest's). Uploads keep the original file's clock: the GUI sends File.lastModified and the upload route restamps the stored file's mtime. None only on pre-field rows (backfilled by the next reingest)
extraction str "text", "ocr" (image parser — OCR is that format's only path), or "ocr_fallback" (the PDF pass-2 recovery) — beyond spec §8.1: extraction provenance for retrieval-quality debugging (OCR noise hits BM25 keyword matching hardest)
heading_path str? v2: markdown heading breadcrumb (e.g. "Operations > Dredging") set by the heading-aware chunkers (markdown_aware, docling_hybrid — spec_v2 §7); None for strategies without structure. New ChunkRecord fields land in the metadata JSONB — no migration needed

¹ page is document-level (the first page that contributed text), not per-chunk: the shared chunker copies one source_meta per document, so per-chunk page attribution has no data path yet. A future chunker with start_index support plus a loader page-map lookup would make it per-chunk. For the v2 office formats, Docling models .pptx slides and .xlsx/.ods sheets as pages, so page carries a slide/sheet number there (.odp slides expose no item provenance, so page stays None); an OCR'd image is a one-page document, so page is 1 whenever text was recognized.

The RetrievalTrace

Where ChunkRecord is ingest-time and immutable, the RetrievalTrace (also varagity/stores/records.py; spec_v2 §9.2) is query-time only — computed per answer by the retrievers and never persisted on the chunk rows. It is the "why did this chunk rank here" record behind the web evidence panel and the CLI's rank badges:

Field Type Meaning
semantic_rank / semantic_score int? / float? Rank and cosine similarity in the semantic (pgvector) arm; None when that arm's ranked list never surfaced the chunk
bm25_rank / bm25_score int? / float? Rank and BM25 relevance in the Elasticsearch arm; None likewise
fused_score float Weighted reciprocal-rank fusion score (required)
fused_rank int Rank after fusion (required)
rerank_score float? Cross-encoder relevance; None when the rerank stage is off the path
rerank_delta int? Positions moved by reranking, pre − post (+ moved up / − moved down); None when off the path
final_rank int The rank actually returned to the caller

Semantics and population:

  • All ranks are 1-based (display-ready). Single-arm retrievers (semantic, bm25) report their arm's score/rank as the fused values — there is nothing to fuse.
  • hybrid.fuse_with_traces builds the trace during fusion (preserving the per-arm rank/score maps that plain fuse discards; raw arm scores never enter the fusion math — they ride along for display only), and hydrate attaches it to each returned chunk with final_rank == fused_rank.
  • RerankedRetriever.apply_rerank then overwrites rerank_score, rerank_delta (pre_rank − final_rank), and final_rank on the surviving candidates — the base retriever's arm and fused fields stay intact — and synthesizes a trace for any trace-less candidate.
  • Backward compatible: RetrievedChunk.trace: RetrievalTrace | None defaults to None, so pre-trace callers (the eval harness, raw store search results) are unaffected.

Consumers: the CLI matches table (-v 2 renders the badges sem #1 · bm25 #3 · fused 0.94 · rerank +2), the web evidence panel, and — for history — the message_sources.trace snapshot below.

PostgreSQL schema

varagity/stores/schema.sql, mounted into the postgres container's /docker-entrypoint-initdb.d/ — it runs only on first boot (empty data directory). Reset with docker compose down -v (see the runbook). It holds the chunk-side schema below and stays the fresh-install fast path; the v2 tables (conversations, messages, message_sources, app_settings) live in migrations/ and reach fresh and existing volumes alike through the migration runner.

erDiagram
    documents ||--o{ chunks : "doc_id · ON DELETE CASCADE"
    documents {
        text doc_id PK "sha256(rel_path:content_hash)[:16]"
        text source
        text file_type
        text content_hash
        int n_chunks "0 = seen but textless"
        timestamptz ingested_at
    }
    chunks {
        text chunk_id PK "doc_id::chunk_index"
        text doc_id FK
        int original_index UK "UNIQUE(doc_id, original_index) — fusion key"
        int chunk_index
        text content "original"
        text context "situating blurb · nullable"
        text contextualized_content "embedded + BM25-indexed text"
        vector1024 embedding "cosine HNSW"
        jsonb metadata "full ChunkRecord"
        timestamptz created_at
    }
CREATE EXTENSION IF NOT EXISTS vector;

-- one row per ingested source document (idempotency + provenance)
CREATE TABLE IF NOT EXISTS documents (
    doc_id        TEXT PRIMARY KEY,
    source        TEXT NOT NULL,
    file_type     TEXT NOT NULL,
    content_hash  TEXT NOT NULL,
    n_chunks      INT  NOT NULL,
    ingested_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- one row per chunk
CREATE TABLE IF NOT EXISTS chunks (
    chunk_id                TEXT PRIMARY KEY,
    doc_id                  TEXT NOT NULL REFERENCES documents(doc_id) ON DELETE CASCADE,
    original_index          INT  NOT NULL,
    chunk_index             INT  NOT NULL,
    content                 TEXT NOT NULL,          -- original
    context                 TEXT,                   -- LLM situating blurb
    contextualized_content  TEXT NOT NULL,          -- embedded/indexed text
    embedding               vector(1024) NOT NULL,  -- EMBEDDING_DIM
    metadata                JSONB NOT NULL,         -- full ChunkRecord (source, page, tokens, …)
    created_at              TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- cosine HNSW index (e5 embeddings are normalized)
CREATE INDEX IF NOT EXISTS chunks_embedding_hnsw
    ON chunks USING hnsw (embedding vector_cosine_ops);

CREATE INDEX IF NOT EXISTS chunks_doc_id_idx ON chunks(doc_id);

-- (doc_id, original_index) is the fusion/join identity across pgvector and
-- Elasticsearch; enforcing uniqueness catches ingest bugs early.
CREATE UNIQUE INDEX IF NOT EXISTS chunks_doc_orig_uidx ON chunks(doc_id, original_index);

Notes:

  • Cosine, not L2: e5 embeddings are L2-normalized, so search orders by embedding <=> :qvec (cosine distance) and reports score = 1 - distance (spec §11.2).
  • The unique identity index is the one schema addition over spec §8.2 (ADR-003): a duplicated (doc_id, original_index) would silently corrupt hybrid fusion, so it fails loudly at write time instead.
  • n_chunks = 0 documents are deliberate: a file with no extractable text still gets a documents row, so it is visibly "seen", cheaply re-warned on later runs without re-parsing, and re-attempted under --reingest. Files are never silently dropped.
  • Writes are upserts: ON CONFLICT (chunk_id) DO UPDATE for chunks; the per-document write (documents row + all its chunks) is one transaction, so a partial failure leaves no idempotency marker and the next run re-attempts the file.

Schema migrations

schema.sql runs only on the container's first boot, so schema added after a pgdata volume exists would never reach it — the gotcha spec_v2 §9.3 exists to close. v2 adds a lightweight, idempotent migration runner (varagity/stores/migrate.py) instead of a heavier dependency (Alembic is the recorded fallback if migrations ever get non-trivial — ADR-005):

  • Ordered files: varagity/stores/migrations/NNN_*.sql, applied in filename order. A .sql file in that directory not matching ^\d{3}_[a-z0-9_]+\.sql$ fails the run loudly (ValueError) — a silently skipped migration is exactly the drift the runner exists to end.
  • Tracked: applied filenames land in schema_migrations (name TEXT PRIMARY KEY, applied_at TIMESTAMPTZ NOT NULL DEFAULT now()), created on demand.
  • One transaction per file: each unapplied migration runs together with its bookkeeping INSERT in a single transaction — a failing migration rolls back atomically and the next run re-attempts exactly that file.
  • Convergent: every migration is written IF NOT EXISTS-safe, so the runner takes a fresh volume and a v1 volume to the same schema; re-running is a no-op.
  • Applied on API startup (the lifespan in varagity/api/main.py, in a threadpool) — not by the CLI. An existing v1 pgdata volume therefore gains the v2 tables on the next api boot, with no docker compose down -v (see Volumes and resets). An unreachable postgres at startup is tolerated (logged; the persistence routes 503 until it returns — the host-mode-without-stack case), but a SQL failure fails API startup: serving on a half-applied schema would be worse than not starting.

Current migrations: 001_conversations.sql and 002_app_settings.sql (the next two sections), v3's 003_condensed_query.sql and 004_message_engine.sql — two nullable ALTER TABLE messages ADD COLUMNs for the chat engine's provenance snapshot (below) — 005_conversation_groups.sql, the sidebar's conversation folders (also below), and 006_graph_turns.sql, the graph-targeted turn's two nullable columns (below). schema.sql deliberately needed no edit for any of them: it holds only the v1 chunk-side tables — messages exists solely in 001, so both install paths (fresh volume and existing pgdata) converge through the runner alone.

Conversation persistence

Single-user chat history (spec_v2 §9.1) lives in the same Postgres as the chunks — one datastore to operate — but in deliberately independent tables, created by migration 001_conversations.sql:

CREATE TABLE IF NOT EXISTS conversations (
    conversation_id  TEXT PRIMARY KEY,        -- app-generated id
    title            TEXT NOT NULL,           -- auto-titled from first question
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at       TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE IF NOT EXISTS messages (
    message_id       TEXT PRIMARY KEY,
    conversation_id  TEXT NOT NULL REFERENCES conversations(conversation_id) ON DELETE CASCADE,
    role             TEXT NOT NULL,           -- 'user' | 'assistant'
    content          TEXT NOT NULL,           -- question or generated answer
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    -- assistant-only provenance snapshot (null for user turns):
    retrieval_method TEXT,
    latency_ms       JSONB,                   -- per-stage timings
    reasoning        TEXT                     -- captured <think> stream, if any
);

-- v3 (migrations 003/004): the chat-engine provenance snapshot — both
-- nullable, both assistant-only (ADR-011):
ALTER TABLE messages ADD COLUMN IF NOT EXISTS condensed_query TEXT;
ALTER TABLE messages ADD COLUMN IF NOT EXISTS chat_engine     TEXT;

-- 006: the graph-targeted turn (ADR-017) — which corpus was ASKED for, and
-- what the graph retrieval found. Both nullable, both assistant-only:
ALTER TABLE messages ADD COLUMN IF NOT EXISTS target_corpus  TEXT;   -- 'rag' | 'graph' | NULL
ALTER TABLE messages ADD COLUMN IF NOT EXISTS graph_evidence JSONB;  -- entities/relations/mode

CREATE TABLE IF NOT EXISTS message_sources (
    message_id       TEXT NOT NULL REFERENCES messages(message_id) ON DELETE CASCADE,
    rank             INT  NOT NULL,           -- final rank in the answer's evidence
    chunk_id         TEXT NOT NULL,           -- soft ref (survives reingest as a snapshot)
    trace            JSONB NOT NULL,          -- RetrievalTrace (§9.2) + score + content/context/source snapshot
    PRIMARY KEY (message_id, rank)
);

-- Transcript fetches read a conversation's messages in order.
CREATE INDEX IF NOT EXISTS messages_conversation_created_idx
    ON messages(conversation_id, created_at);

-- 005: sidebar conversation groups — user-created folders. Deleting a
-- group detaches its conversations (SET NULL), never deletes them.
CREATE TABLE IF NOT EXISTS conversation_groups (
    group_id    TEXT PRIMARY KEY,        -- app-generated id
    name        TEXT NOT NULL,           -- display name (not unique — ids are the identity)
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

ALTER TABLE conversations ADD COLUMN IF NOT EXISTS group_id TEXT
    REFERENCES conversation_groups(group_id) ON DELETE SET NULL;

CREATE INDEX IF NOT EXISTS conversations_group_id_idx ON conversations(group_id);

erDiagram
    conversation_groups |o--o{ conversations : "ON DELETE SET NULL"
    conversations ||--o{ messages : "ON DELETE CASCADE"
    messages ||--o{ message_sources : "ON DELETE CASCADE"
    chunks |o..o{ message_sources : "chunk_id — soft ref, no FK"
    conversation_groups {
        text group_id PK
        text name "display name"
        timestamptz created_at
    }
    conversations {
        text conversation_id PK
        text title "auto-titled from the first question"
        timestamptz created_at
        timestamptz updated_at
        text group_id FK "sidebar group — 005, nullable"
    }
    messages {
        text message_id PK
        text conversation_id FK
        text role "user / assistant"
        text content
        timestamptz created_at
        text retrieval_method "assistant turns only"
        jsonb latency_ms "per-stage timings"
        text reasoning "captured think stream"
        text condensed_query "v3 · the search rewrite, if any"
        text chat_engine "v3 · engine that produced the turn"
        text target_corpus "006 · corpus asked for — rag / graph"
        jsonb graph_evidence "006 · entities + relations + mode"
    }
    message_sources {
        text message_id PK "composite with rank"
        int rank PK "final evidence rank"
        text chunk_id "soft reference — or a graph doc_key"
        jsonb trace "score + texts + provenance + RetrievalTrace, or a graph_transcript"
    }

Notes:

  • message_sources.trace is a snapshot, not a join (ConversationStore._source_snapshot): it persists score, content, context, source, file_name, file_type, page, extraction, file_created_at/file_modified_at, and the serialized RetrievalTrace under "trace" (null when the retriever attached none) — everything the provenance panel renders. A historical conversation therefore still explains itself after a reingest replaces the chunk rows and changes chunk_ids.
  • chunk_id is deliberately a soft reference — no FK to chunks. A hard FK would either cascade history away on reingest or block corpus maintenance; the id is kept purely as a record of which chunk backed the evidence at answer time.
  • Assistant-turn provenance rides on messages itself: retrieval_method, per-stage latency_ms (JSONB), and the captured <think> stream in reasoning — all NULL for user turns.
  • v3 adds the chat-engine snapshot (migrations 003/004; ADR-011): condensed_query is the standalone search query that actually drove retrieval — the evidence panel's "Searched for: …" line, live and on reload — and chat_engine is the registry name of the engine that produced the turn. NULL condensed_query means "not condensed" (a first turn, the simple engine, the kill switch, or the raw-query fallback); the engine name is persisted even when degraded, mirroring how retrieval_method persists reranked under RERANK_ENABLED=false (and, post-v3, hyde under HYDE_ENABLED=false or a failed passage generation — ADR-016; the hypothetical passage itself is deliberately not persisted). Same snapshot-over-reference rationale as message_sources.trace: the columns explain a historical answer, so they must outlive the settings that produced it.
  • A turn is persisted in one transaction (append_message): the message, its message_sources rows (rank 1-based, best first), and the conversation's updated_at bump — a partial failure leaves no dangling turn. An aborted stream persists nothing.
  • Groups are organization, never ownership (migration 005): the sidebar's folders live in conversation_groups, and conversations.group_id is a nullable FK with ON DELETE SET NULL — deleting a group returns its conversations to the ungrouped list with history intact. Moving a conversation (PATCH /api/conversations/{id}) deliberately does not bump updated_at, so filing a chat never re-orders the recency-sorted list.

Graph-targeted turns

Migration 006_graph_turns.sql records what a message-graph turn asked for and what the graph gave back, in the same snapshot discipline as everything else on the turn:

  • target_corpus is the request, not the outcome — 'rag' or 'graph', NULL for user turns and all pre-stage-2 history (read as chunk RAG). A graph turn that degraded to a document answer (kill switch off, engine unreachable) still persists 'graph': the ask is part of the record, and the GUI's source selector stays where the user left it instead of silently flipping back.
  • graph_evidence is the retrieval, minus the transcripts — the entities, the relations, and the engine query mode that found them ({"entities": [...], "relations": [...], "mode": "mix"}, the same shape the live SSE retrieval event carries). It is NULL on a degraded turn, which is precisely the honest record: the graph produced nothing. Together the two columns distinguish all three states — chunk turn (NULL/NULL), graph turn ('graph'/payload), degraded graph turn ('graph'/NULL).
  • The cited days ride in message_sources, not a second evidence table. A graph row stores the transcript day's doc_key in chunk_id and a trace carrying {"kind": "graph_transcript", "thread_name", "span", "excerpt", "message_guids"}. The kind discriminator is what readers branch on; the v2 chunk snapshots have no kind and were never migrated. The soft-reference rationale carries over verbatim — a graph rebuild changes doc_keys exactly as a reingest changes chunk_ids, and the snapshot is what keeps the old answer explicable.

The graph workdir

The graph itself is not in Postgres. It lives in files, in the graphdata volume (GRAPH_STORAGE_PATH/<engine>, /app/graph-data/lightrag in compose), because the engine's storages are single-writer per directory and the API is that writer (ADR-017):

File(s) Owner What it holds
graph_chunk_entity_relation.graphml engine The entity/relation graph itself
vdb_entities.json, vdb_relationships.json, vdb_chunks.json engine The embedding indexes retrieval searches
kv_store_full_docs.json, kv_store_text_chunks.json, kv_store_full_entities.json, kv_store_full_relations.json, kv_store_entity_chunks.json, kv_store_relation_chunks.json engine The extracted text and its derived records
kv_store_doc_status.json engine Per-document processing status — the resume ledger, and what the varagity_graph_documents{status} gauge counts
kv_store_llm_response_cache.json engine Extraction responses, reused by a delete-then-reinsert
varagity_manifest.json ours doc_key → {content_sha256, message_guids, thread_name, span}
varagity_graph_summary.json ours Last known {entities, relations, message_guids, docs, refreshed_at}

The two sidecars are ours and earn their keep three times over (varagity/graph/manifest.py):

  1. They are the upsert diff the engine refuses to do. A re-submitted document id is dropped by the engine's enqueue stage in any status, so a re-exported chat.db whose newest thread-day gained messages would keep the stale transcript forever. Comparing rendered content hashes against the manifest is what turns that into an explicit delete-then-reinsert. Never hand-edit the manifest: an entry whose hash does not match the graph's content makes the stale transcript permanent. (A graph backup carries both sidecars with the engine's files, so the diff is backed up with the graph it describes.)
  2. They are the durable provenance index. doc_key → message_guids survives a restart, so a citation still resolves to messages on a graph nothing has rebuilt in this process.
  3. They are the size cache. GET /api/graph/status and the Prometheus gauges read the summary rather than walking a multi-megabyte graphml.

Both are written atomically (temp file + os.replace), and both degrade to "absent" when unreadable or written by a different schema version: every rendered document then looks new, which the engine's own dedup absorbs, where trusting a mis-shaped file would corrupt the graph. Neither is authoritative over the engine — they are caches of what the last write did.

doc_key is the graph's identity thread, the counterpart of (doc_id, original_index) on the chunk side:

doc_key = f"{thread_id}::{span}"      # span is YYYY-MM-DD or first..last

It is what the engine attributes a chunk to (file_path), what message_sources.chunk_id stores on a graph turn, what the manifest keys on, and what the graph view's entity drill-down resolves into source days. It is day-grain by construction — which is why the evidence panel says so.

Runtime settings overrides

Migration 002_app_settings.sql backs the GUI's live settings panel (spec_v2 §4.7):

CREATE TABLE IF NOT EXISTS app_settings (
    key         TEXT PRIMARY KEY,
    value       TEXT NOT NULL,
    updated_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);
  • One row per overridden Settings field, keyed by the field name (e.g. RETRIEVAL_METHOD) with the value in its env-string form — exactly what pydantic-settings parses from the environment, so the override layer (varagity/api/runtime_settings.py) replays rows as process environment variables verbatim (env beats .env) and clears the settings cache. Every module that reads get_settings() — the repo-wide convention — sees the override on its next call.
  • Written by PATCH /api/settings, which validates the merged whole through every config validator (the fusion-weight pair, the rerank bounds, the vocabularies) before persisting — an invalid patch changes nothing. Replayed at API startup, after the migration runner, so overrides survive an api restart; persisted rows that no longer validate are skipped with an error log (the API must boot on env defaults rather than crash).
  • Keys beginning _ are reserved for app metadata and never surface as overrides. The one in use is _corpus_stale — "a reingest-affecting setting changed since the last completed reingest" — set by PATCH /api/settings when such a setting actually changes on a non-empty corpus, cleared only by a completed API reingest (see below).

Elasticsearch index

varagity/stores/bm25_store.py (default index name varagity_contextual_bm25). Mirrors the Anthropic cookbook's ElasticsearchBM25: analyzed text fields under the built-in english analyzer with BM25 similarity; identity fields stored but not indexed.

{
  "settings": {
    "analysis": { "analyzer": { "default": { "type": "english" } } },
    "similarity": { "default": { "type": "BM25" } }
  },
  "mappings": { "properties": {
    "content":                { "type": "text",    "analyzer": "english" },
    "contextualized_content": { "type": "text",    "analyzer": "english" },
    "doc_id":                 { "type": "keyword", "index": false },
    "chunk_id":               { "type": "keyword", "index": false },
    "original_index":         { "type": "integer", "index": false }
  }}
}

Notes:

  • The index is contextual from its first document: chunks are contextualized before they reach either store, and search is a multi_match over content and contextualized_content.
  • ES stores identity + text only. Full records (source, page, context, metadata) are hydrated from pgvector by (doc_id, original_index) — pg is the single source of truth for metadata.
  • "index": false fields keep doc values, so the term-level delete_by_query used by --reingest still works (a slower doc-values scan — fine at dev scale).
  • Documents are addressed by chunk_id (_id), so re-indexing overwrites instead of duplicating — the sparse-side counterpart of the pg upsert.

Idempotency & re-ingestion semantics

  • A file whose (doc_id, content_hash) already exists in documents is skipped before parsing.
  • Pipeline-setting changes (CONTEXTUALIZE, chunk params, OCR_ENGINE) do not change content hashes → unchanged files stay skipped. Re-process with main.py ingest --reingest (CLI) or POST /api/ingest with reingest=true (GUI) — both run the same flow, which deletes each discovered document from both stores (ES delete_by_query first, then the pg cascade delete) before ingesting fresh.
  • The GUI surfaces that footgun as a persisted flag (v2): changing a reingest-affecting setting through PATCH /api/settings (CHUNKING_STRATEGY, CHUNK_SIZE, CHUNK_OVERLAP, CONTEXTUALIZE, OCR_ENGINE) on a non-empty corpus sets _corpus_stale in app_settings, rendered as a "Re-ingest to apply" affordance. Only a completed API reingest clears it — a CLI ingest --reingest never touches the flag, and patching the setting back to its old value doesn't clear it either.
  • The dual-write order is BM25 first, pgvector last: the pg documents row is the idempotency marker, so a failure between the two writes leaves no marker and the next run re-attempts the file (the chunk_id-addressed ES bulk then overwrites rather than duplicates).
  • Removing a file from docs/ does not remove its chunks — the ingest path has no corpus GC beyond --reingest. v2's DELETE /api/documents/{doc_id} closes that gap per document: ES delete first, then the pg cascade (the marker is deleted last, so a failed ES delete leaves the document visible and retryable), with opt-in source-file removal (?remove_file=true, honored only for paths inside DOCS_PATH; folders the unlink empties are pruned too, so deleting a folder upload's documents leaves no empty shells in the corpus directory). POST /api/documents/delete is the same operation over a set (the corpus table's multi-select): one terms delete_by_query and one DELETE … WHERE doc_id = ANY(…), so the ordering — and therefore the retryability — holds set-wise, at one round trip per store rather than per document. Ids with no documents row come back in not_found instead of failing the batch, so a selection racing a concurrent delete still removes what remains.
  • GET /api/documents reports each document's relative_path under DOCS_PATH (null for sources living elsewhere), and the GUI folds the corpus table back into the folders a folder upload created — collapsible rows whose checkbox and delete act on every descendant, so "delete a directory" is the same bulk delete over that folder's doc_ids, not a second delete path.

The write ordering that makes the documents row a reliable idempotency marker — sparse store first, transactional store last:

sequenceDiagram
    autonumber
    participant L as loader.store_chunks
    participant ES as Elasticsearch
    participant PG as pgvector
    L->>ES: bulk index chunks (addressed by chunk_id)
    Note right of ES: overwrite, never duplicate
    L->>PG: BEGIN
    L->>PG: upsert documents row + chunks (ON CONFLICT DO UPDATE)
    L->>PG: COMMIT
    Note right of PG: documents row = idempotency marker
    Note over L,PG: crash before COMMIT ⇒ no marker ⇒ file re-attempted next run<br/>(the chunk_id-addressed ES bulk already overwrote, so nothing duplicates)

Backup bundles

docker compose down -v deletes every named volume, and with them the processed corpora: the chunk rows, their contextual blurbs and embeddings, the BM25 index, and the extracted message graph in graphdata. The source folders survive, because they are host binds. A backup bundle is written into a reserved folder inside one of those binds, so it survives beside the sources (varagity/backup/). Each corpus has its own folder and its own bundle layout in the same container: the RAG bundle below, and the graph bundle.

docs/varagity-exports/                          reserved: discovery skips it, uploads refuse it
├── .varagity-rag-20261108-1312.zip.partial     an export in flight; never listed, swept by the next export
└── varagity-rag-20261108-1312.zip              renamed into place once complete
    ├── documents.jsonl    documents rows + relative_path                      deflated
    ├── chunks.jsonl       every chunks column but the embedding,              deflated
    │                      ORDER BY original_index
    ├── embeddings.f32     N × EMBEDDING_DIM little-endian float32;            stored
    │                      row i belongs to chunks line i
    └── bundle.json        the manifest, written last
  • The reserved folder. varagity-exports/ at the corpus root is never corpus. Ingest discovery (and so the CLI's ingest too) skips it whole, and both upload routes refuse a path under it, or a flat file named like it, with the per-file reason reserved_path. On the graph side the build's scan never counts it, the Graph sources listing never shows it, and the graph delete answers 404 for anything in it (deleting a bundle there would flag the graph stale for a file that was never a source). Only the root's own folder is reserved, matched case-insensitively. A varagity-exports/ nested deeper is ordinary corpus.
  • One consistent snapshot. Every row comes from a single REPEATABLE READ, READ ONLY transaction (ContextualVectorDB.iter_corpus), streamed through server-side cursors. An ingest or delete committing mid-export can't tear a bundle, and the export blocks no writer.
  • Bit-exact vectors. The snapshot reads pgvector's binary format, and each vector becomes a little-endian float32 matrix row by a byte swap, not a decimal round trip. Row i of embeddings.f32 is the vector of line i of chunks.jsonl by construction: both are written in one pass over the same snapshot.
  • No BM25 member. Every field of an Elasticsearch document is a copy of a chunks column (below), so the index can always be rebuilt from the rows with no model call.
  • relative_path is each document's source relative to DOCS_PATH. It is null for a document ingested from outside the corpus directory. doc_id already hashes it, and it is the portable form of source across host and container mode. (An export run by a host-mode API against container-written rows records no relative paths, because /app/docs/… doesn't resolve under ./docs.)
  • All-or-nothing. The writer streams into a hidden .{name}.partial, hashes every member as it goes, writes bundle.json as the last member (so it can record every member's sha256), syncs the file, and renames it into place with os.replace. Any failure deletes the partial, and the next export sweeps whatever a crash left behind. A listing therefore sees a whole bundle or nothing.
  • Names are varagity-<corpus>-<UTC yyyymmdd-HHMM>.zip. A same-minute repeat gets -2, -3, …, and an export never replaces a file.

bundle.json (BundleManifest) carries what the rows themselves don't record: the export-time facts a restore must judge the bundle by.

Field Meaning
format, version "varagity-backup", 1 — what makes a zip a Varagity bundle, and its layout.
corpus rag or graph — ChatRequest.corpus's vocabulary; a bundle restores only its own corpus.
created_at The snapshot instant (UTC).
embedding_model, embedding_dim EMBEDDING_MODEL / EMBEDDING_DIM at export. pgvector checks only the vector size (and LightRAG only embedding_dim), so the model name is the only guard against vectors from another embedding space. An export whose rows record a different metadata.embedding_model logs a warning.
stale The corpus's stale flag at export: _corpus_stale for RAG, _graph_stale for the graph.
counts RAG: {"documents": …, "chunks": …}. Graph: {"thread_days": …, "messages": …}, plus entities and relations when the graph reported them (each from the workdir's size summary).
ingest_settings RAG only (null for the graph). The reingest-affecting settings at export (CHUNKING_STRATEGY, CHUNK_SIZE, CHUNK_OVERLAP, CONTEXTUALIZE, OCR_ENGINE), in the env-string form app_settings stores. They are read before the stale flag, so a settings change racing the export can only make the bundle more conservative, never less.
engine Graph only (null for RAG). {name, package, version} — the engine that wrote the files (lightrag / lightrag-hku / 1.5.4), as the engine itself reports it (GraphEngine.storage_format(), read from package metadata).
members {name, size_bytes, sha256} per data member, in write order.

The zip's central directory lets GET /api/backups/{corpus} read bundle.json alone. Listing a folder of large bundles costs a directory read, not an unpack. Chat history, conversation folders and settings-drawer overrides are deliberately not in a bundle: it backs up the corpus, not the app.

The graph bundle

The message graph's bundle is the graph workdir verbatim — every file the engine wrote, plus the repo's two sidecars — in the same container, under one folder named for the engine:

graph-docs/varagity-exports/varagity-graph-20261108-1315.zip
├── lightrag/                               the workdir verbatim, *.tmp* skipped     deflated
│   ├── kv_store_*.json ×8                  incl. the LLM cache later incremental builds rebuild from
│   ├── vdb_*.json ×3                       embedding_dim + base64 float32 matrices
│   ├── graph_chunk_entity_relation.graphml
│   ├── varagity_manifest.json              what keeps the next build a diffing upsert
│   └── varagity_graph_summary.json         the size cache — the bundle's counts
└── bundle.json                             + engine format · summary counts · stale = _graph_stale
  • Verbatim, so everything comes back. Nothing is converted or re-derived: the doc-status resume ledger, the manifest (so the next build stays a diffing upsert rather than re-extracting everything), and the LLM response cache (which a later incremental build needs to rebuild the entities a changed thread-day shares with others) all ride along. The price is that a bundle is tied to the engine version that wrote it, which is why the manifest records engine.
  • Copied under the write lock. The export holds GraphService's write lock — the one every build, resume and delete takes — for the whole copy, so no build changes the graph mid-copy. Reads take no lock, so graph chat and /graph keep answering throughout. A read does write one file: LightRAG flushes its LLM response cache after every query. That is harmless, because the engine writes each file to a temp sibling (<file>.tmp.<pid>.<thread>.<ns>) and renames it into place, so every member is a whole file, from before a query or after it. The temp files are never copied, nor is anything a bundle can't carry (a folder, a symlink, an unsafe name) — each is named in a warning instead.
  • The member rule is a pattern, not a list. A workdir holds only the namespaces its engine has written so far, so no member is individually required. Every member must be a flat file directly in <engine>/, its name made of ASCII letters, digits, _, . and - and never hidden; every file in the zip must be recorded in bundle.json with its sha256 (an unrecorded file could never be verified). The verdict tolerates the <engine>/ directory entry a hand re-zip (zip -r) adds.
  • Counts come from the size summary (varagity_graph_summary.json), never from parsing the graphml: thread_days is its docs, messages its message_guids. A graph whose first build was killed before writing a summary still exports — its files are the resume state — with empty counts.
  • No model service. An export needs only Postgres, for the _graph_stale flag the bundle records; it opens no engine session. The flag is read last, once every file is copied: deleting a graph source flags the graph stale without taking any lock, so a source deleted mid-export is still recorded.

Restoring a bundle

A restore replaces the corpus with the bundle's contents; nothing is merged, and nothing is re-parsed, re-chunked, re-contextualized, re-embedded or re-extracted (ADR-019). How a restore reads the bundle back:

  • Two depths of checking. Every listed bundle gets the cheap verdict (inspect_bundle): the zip directory and bundle.json only. The format version, the corpus, the member allowlist (exactly the three RAG members; for a graph bundle, the pattern above, with at least one file), the zip-slip rules (no .., absolute path, backslash or symlink entry), each member's recorded size against the zip's, the vector matrix's size against counts.chunks × embedding_dim × 4, then the embedding model and dimension against this stack's — and for a graph bundle the engine that wrote its files: it must be the engine GRAPH_ENGINE selects, from the same package, at the same major.minor version as this stack's (running_engine_format(), the engine's own report). A restore then checks the sha256 of every member while it reads them — into COPY, or onto disk — its one read of the bundle; nothing takes effect until every digest matched (the transaction commits, the staged graph is swapped in), so a damaged bundle can never go live. verify_bundle runs the same checks standalone, without restoring. A mismatch in the model, dimension, engine, corpus or version is refused, never converted (bundle_incompatible); anything unsound or damaged is bundle_invalid.
  • Re-rooting. Each document's source becomes (DOCS_PATH / relative_path).resolve(), exactly as ingest builds it, and every chunk's metadata.source follows its document. doc_id, chunk_id, original_index, the blurbs, the vectors and every timestamp come back as exported. A document with no relative_path (ingested from outside the corpus) keeps its absolute source. So a bundle exported in compose (/app/docs/…) restores onto a host-mode ./docs and back, and the document list, page previews and "also remove the file" keep working.
  • The stale flag is derived. _corpus_stale is set when the bundle's stale was, or when its ingest_settings differ from this stack's reingest-affecting settings — the same "the corpus may not match the current settings" the flag always means. The list reports those differing settings as drift before you restore.
  • The BM25 index is rebuilt from the restored rows — every one of its fields is a copy of a chunks column (below).

A graph bundle restores as a whole working directory:

  • Extract beside, then swap. The bundle's <engine>/ files are extracted into <workdir>.restore-<run> — a sibling of the workdir on the same graphdata volume — each synced to disk. GraphService.replace_workdir then closes the session, renames the live workdir to <workdir>.previous-<run> and the staged folder into its place, and opens it. The workdir is never half-written: it is the old folder or the new one, and readers that arrive mid-swap wait for it.
  • The commit point. Once the restored graph has opened, the previous one is renamed to <workdir>.replaced-<run> and deleted. A failure before that puts the previous graph back and reopens it; a crash before it is rolled back when the next process constructs its GraphService, and a crash after it leaves only debris, which that construction removes.
  • Everything comes back. The manifest (so the next build stays a diffing upsert — a re-exported chat.db still costs only its changed days), the size summary (so the status route and the gauges read the restored sizes at once), the doc-status ledger (a build that was resuming when exported resumes after the restore), and the LLM cache.
  • The stale flag is the bundle's own. _graph_stale takes the bundle's stale, written once the swap committed: the graph is exactly what was exported, so it is stale exactly when it was then.

Golden eval dataset

data/eval/golden_qa.jsonl — one JSON object per line:

{"query": "…", "relevant": [{"rel_source": "aurora_station.md", "chunk_index": 1, "fact": "4.2 megawatts"}]}

Relevant chunks are identified portably by rel_source (path relative to the corpus root — exactly the string doc_id hashes) plus chunk_index, so entries resolve to concrete chunk_ids from corpus files alone, on any machine. chunk_index values assume the pinned eval chunk boundaries (recursive_character, 400/50 — PINNED_EVAL_SETTINGS); re-author the golden set if those pins change.

fact (v2) is the literal answer snippet the chunker sweep keys on (spec_v2 §7.4): foreign chunk boundaries invalidate chunk_index, so the sweep re-resolves each ref by content (resolve_golden_by_fact) — any strategy-true chunk containing the fact satisfies the ref; index-anchored scoring under foreign boundaries would measure nothing but the boundary mismatch. Ground-truth transcriptions of the OCR fixture PDFs live in data/eval/ocr_truth/.