Chat engines¶

The chat-engine registry (spec_v3 §4; ADR-011): an engine decides what string the retriever searches with, given the turn and its conversation history, returning the PreparedQuery two-string split — search_query drives retrieval while original_query (always the user's words) drives generation. Registered engines: simple (the stateless identity split) and condense_context (history-resolved standalone queries; the shipped default stays simple — benchmark-decided).

`varagity.chat.base` ¶

Chat-engine protocol and registry (spec_v3 §4.2, the spec §5.1 pattern).

Each chat-engine module defines one implementation decorated with @register("name"); callers resolve the configured engine with get_chat_engine(settings.CHAT_ENGINE). Registered: simple (the stateless v2 behavior, verbatim) and condense_context (spec_v3 §4.5 — follow-ups rewritten into standalone search queries against the conversation history); adding an engine later means one new file plus its import line — no caller edits, exactly as the retrieval registry's reranked addition proved.

An engine decides what string the retriever searches with, given the turn and its conversation history. It is deliberately not a :class:~varagity.retrieval.base.Retriever: condensing needs chat history, which doesn't fit Retriever.retrieve(query, k, …) — widening that signature would force a parameter on three retrievers that will never use it.

`ChatEngine` ¶

Bases: Protocol

Decides what string the retriever searches with, given a turn and its history.

runtime_checkable for the same reason :class:~varagity.retrieval.base.Retriever is: the protocol appears in Prefect flow signatures (varagity.pipeline.query_flow), and Prefect builds a pydantic parameter schema from the annotations at decoration time, which requires types usable with isinstance.

Source code in varagity/chat/base.py

@runtime_checkable
class ChatEngine(Protocol):
    """Decides what string the retriever searches with, given a turn and its history.

    ``runtime_checkable`` for the same reason
    :class:`~varagity.retrieval.base.Retriever` is: the protocol appears in
    Prefect flow signatures (``varagity.pipeline.query_flow``), and Prefect
    builds a pydantic parameter schema from the annotations at decoration
    time, which requires types usable with ``isinstance``.
    """

    def prepare(
        self,
        query: str,
        *,
        history: Sequence[Turn],
        llm: LLMClient | None,
        verbose: int,
    ) -> PreparedQuery:
        """Prepare the retrieval query for one chat turn.

        Args:
            query: The user's question, verbatim.
            history: Prior turns, oldest first (empty on a first turn).
            llm: Chat client for engines that condense; ``None`` lets such
                engines resolve one via the model registry. Engines that
                never call a model ignore it.
            verbose: Validated console verbosity (0–2).

        Returns:
            The two-string split downstream stages retrieve and answer with.
        """
        ...

`prepare(query, *, history, llm, verbose)` ¶

Prepare the retrieval query for one chat turn.

Parameters:

Name	Type	Description	Default
`query`	`str`	The user's question, verbatim.	required
`history`	`Sequence[Turn]`	Prior turns, oldest first (empty on a first turn).	required
`llm`	`LLMClient \| None`	Chat client for engines that condense; `None` lets such engines resolve one via the model registry. Engines that never call a model ignore it.	required
`verbose`	`int`	Validated console verbosity (0–2).	required

Returns:

Type	Description
`PreparedQuery`	The two-string split downstream stages retrieve and answer with.

Source code in varagity/chat/base.py

def prepare(
    self,
    query: str,
    *,
    history: Sequence[Turn],
    llm: LLMClient | None,
    verbose: int,
) -> PreparedQuery:
    """Prepare the retrieval query for one chat turn.

    Args:
        query: The user's question, verbatim.
        history: Prior turns, oldest first (empty on a first turn).
        llm: Chat client for engines that condense; ``None`` lets such
            engines resolve one via the model registry. Engines that
            never call a model ignore it.
        verbose: Validated console verbosity (0–2).

    Returns:
        The two-string split downstream stages retrieve and answer with.
    """
    ...

`PreparedQuery` `dataclass` ¶

A chat engine's decision about what to search with (spec_v3 §4.2).

The two-string split is the design's core invariant: the retriever gets search_query while the answer prompt always gets original_query — the user's own words and emphasis are never rewritten for generation.

Attributes:

Name	Type	Description
`search_query`	`str`	What the retriever searches with (drives both the query embedding and BM25).
`original_query`	`str`	What the answer prompt gets — always the user's words, verbatim.
`condensed`	`bool`	`False` means `search_query` is `original_query` verbatim (no rewrite happened).
`condense_latency_s`	`float \| None`	Wall-clock seconds the condense LLM call took, or `None` when no call was made.

Source code in varagity/chat/base.py

@dataclass(frozen=True)
class PreparedQuery:
    """A chat engine's decision about what to search with (spec_v3 §4.2).

    The two-string split is the design's core invariant: the retriever gets
    ``search_query`` while the answer prompt always gets ``original_query``
    — the user's own words and emphasis are never rewritten for generation.

    Attributes:
        search_query: What the retriever searches with (drives both the
            query embedding and BM25).
        original_query: What the answer prompt gets — always the user's
            words, verbatim.
        condensed: ``False`` means ``search_query`` is ``original_query``
            verbatim (no rewrite happened).
        condense_latency_s: Wall-clock seconds the condense LLM call took,
            or ``None`` when no call was made.
    """

    search_query: str
    original_query: str
    condensed: bool
    condense_latency_s: float | None

`Turn` `dataclass` ¶

One prior conversation turn, as chat engines consume it.

Attributes:

Name	Type	Description
`role`	`str`	`"user"` or `"assistant"`.
`content`	`str`	The turn's text (the question or the generated answer).

Source code in varagity/chat/base.py

@dataclass(frozen=True)
class Turn:
    """One prior conversation turn, as chat engines consume it.

    Attributes:
        role: ``"user"`` or ``"assistant"``.
        content: The turn's text (the question or the generated answer).
    """

    role: str
    content: str

`get_chat_engine(name)` ¶

Look up a registered chat engine by name.

Parameters:

Name	Type	Description	Default
`name`	`str`	Registry key (e.g. `"simple"`).	required

Returns:

Type	Description
`ChatEngine`	The registered engine instance.

Raises:

Type	Description
`KeyError`	If no engine is registered under `name` (message lists the available ones).

Source code in varagity/chat/base.py

def get_chat_engine(name: str) -> ChatEngine:
    """Look up a registered chat engine by name.

    Args:
        name: Registry key (e.g. ``"simple"``).

    Returns:
        The registered engine instance.

    Raises:
        KeyError: If no engine is registered under ``name`` (message lists
            the available ones).
    """
    if name not in CHAT_ENGINE_REGISTRY:
        raise KeyError(f"Unknown chat engine {name!r}. Available: {list(CHAT_ENGINE_REGISTRY)}")
    return CHAT_ENGINE_REGISTRY[name]

`register(name)` ¶

Class decorator registering a chat engine instance under name.

Parameters:

Name	Type	Description	Default
`name`	`str`	Registry key (the `CHAT_ENGINE` env value).	required

Returns:

Type	Description
`Callable[[T], T]`	The decorator, which instantiates and registers the class unchanged.

Source code in varagity/chat/base.py

def register[T: type[Any]](name: str) -> Callable[[T], T]:
    """Class decorator registering a chat engine instance under ``name``.

    Args:
        name: Registry key (the ``CHAT_ENGINE`` env value).

    Returns:
        The decorator, which instantiates and registers the class unchanged.
    """

    def deco(cls: T) -> T:
        CHAT_ENGINE_REGISTRY[name] = cls()
        return cls

    return deco

`varagity.chat.simple` ¶

The simple chat engine: today's stateless behavior, verbatim (spec_v3 §4.2).

`SimpleChatEngine` ¶

Pass-through engine: search with the user's words, ignore history.

Registering today's behavior as an engine of its own is what proves the registry isn't a special case built for one implementation — simple is a peer of every engine that follows it, not the absence of one.

Source code in varagity/chat/simple.py

@register("simple")
class SimpleChatEngine:
    """Pass-through engine: search with the user's words, ignore history.

    Registering today's behavior as an engine of its own is what proves the
    registry isn't a special case built for one implementation — ``simple``
    is a peer of every engine that follows it, not the absence of one.
    """

    def prepare(
        self,
        query: str,
        *,
        history: Sequence[Turn],
        llm: LLMClient | None,
        verbose: int,
    ) -> PreparedQuery:
        """Return the identity split: the query passes through unchanged.

        Never calls a model — a ``simple`` turn costs no LLM round-trip.

        Args:
            query: The user's question, verbatim.
            history: Prior turns; deliberately ignored.
            llm: Chat client; deliberately ignored.
            verbose: Validated console verbosity; nothing to render.

        Returns:
            ``search_query == original_query``, ``condensed=False``.
        """
        return PreparedQuery(
            search_query=query,
            original_query=query,
            condensed=False,
            condense_latency_s=None,
        )

`prepare(query, *, history, llm, verbose)` ¶

Return the identity split: the query passes through unchanged.

Never calls a model — a simple turn costs no LLM round-trip.

Parameters:

Name	Type	Description	Default
`query`	`str`	The user's question, verbatim.	required
`history`	`Sequence[Turn]`	Prior turns; deliberately ignored.	required
`llm`	`LLMClient \| None`	Chat client; deliberately ignored.	required
`verbose`	`int`	Validated console verbosity; nothing to render.	required

Returns:

Type	Description
`PreparedQuery`	`search_query == original_query`, `condensed=False`.

Source code in varagity/chat/simple.py

def prepare(
    self,
    query: str,
    *,
    history: Sequence[Turn],
    llm: LLMClient | None,
    verbose: int,
) -> PreparedQuery:
    """Return the identity split: the query passes through unchanged.

    Never calls a model — a ``simple`` turn costs no LLM round-trip.

    Args:
        query: The user's question, verbatim.
        history: Prior turns; deliberately ignored.
        llm: Chat client; deliberately ignored.
        verbose: Validated console verbosity; nothing to render.

    Returns:
        ``search_query == original_query``, ``condensed=False``.
    """
    return PreparedQuery(
        search_query=query,
        original_query=query,
        condensed=False,
        condense_latency_s=None,
    )

`varagity.chat.condense` ¶

The condense_context chat engine (spec_v3 §4.5, §4.6).

Rewrites a follow-up question into a standalone search query against the conversation history — one non-streaming LLM call before retrieval — while the answer prompt still gets the user's original words (the :class:~varagity.chat.base.PreparedQuery two-string split).

Failure is a fallback, not an error (§4.6): a transient LLM failure (after the client's own tenacity retries), an empty result, or an absurdly long one all degrade to searching with the raw query at WARNING — a degraded search query still answers a lot of questions; a 500 answers none. CONDENSE_ENABLED=false is the kill switch, checked inside :meth:~CondenseContextEngine.prepare exactly as RERANK_ENABLED is checked inside the reranked retriever — deliberately orthogonal to engine selection.

`CondenseContextEngine` ¶

Condense + Context: search with a history-resolved standalone query.

The LlamaIndex "Condense Plus Context" pattern adapted to the registry convention: the condensed rewrite drives retrieval (both the query embedding and BM25) while generation always receives the original question — the user's own words and emphasis are never rewritten for the answer prompt.

Source code in varagity/chat/condense.py

@register("condense_context")
class CondenseContextEngine:
    """Condense + Context: search with a history-resolved standalone query.

    The LlamaIndex "Condense Plus Context" pattern adapted to the registry
    convention: the condensed rewrite drives retrieval (both the query
    embedding and BM25) while generation always receives the original
    question — the user's own words and emphasis are never rewritten for
    the answer prompt.
    """

    def prepare(
        self,
        query: str,
        *,
        history: Sequence[Turn],
        llm: LLMClient | None,
        verbose: int,
    ) -> PreparedQuery:
        """Condense the turn against its history into the retrieval query.

        The first turn never condenses: empty history means there is
        nothing to resolve a reference against, and the common single-turn
        question must not pay an LLM round-trip for nothing. The history
        fed to the prompt is bounded by ``CONDENSE_HISTORY_TURNS`` (newest
        turns kept), whatever the caller loaded.

        Args:
            query: The user's question, verbatim.
            history: Prior turns, oldest first (empty on a first turn).
            llm: Chat client; resolved via the model registry
                (``settings.CONDENSE_MODEL_TYPE``) when ``None``.
            verbose: Validated console verbosity (0–2).

        Returns:
            The condensed split, or the identity split whenever no LLM
            call was made (kill switch, first turn) or the call's outcome
            was unusable (the §4.6 fallback).
        """
        settings = get_settings()
        identity = PreparedQuery(
            search_query=query, original_query=query, condensed=False, condense_latency_s=None
        )
        if not settings.CONDENSE_ENABLED:
            logger.info(
                "CONDENSE_ENABLED=false — degrading to the 'simple' engine's identity split"
            )
            return identity
        turns = self._bounded(history, settings.CONDENSE_HISTORY_TURNS)
        if not turns:
            return identity

        prompt = CONDENSE_PROMPT.format(history=format_history(turns), query=query)
        # CONDENSE_MODEL_TYPE is validated to the LLM aliases, so the
        # registry always resolves an LLMClient here; the cast (rather than
        # an isinstance gate) keeps duck-typed test doubles usable at this
        # seam, matching the flows' injectable-fake convention.
        client = (
            llm if llm is not None else cast("LLMClient", get_model(settings.CONDENSE_MODEL_TYPE))
        )
        started = time.perf_counter()
        try:
            # verbose=0: the sub-call renders nothing; v_condensed below is
            # this stage's console output (the reranked-retriever pattern).
            raw = client.generate(
                [{"role": "user", "content": prompt}],
                max_tokens=settings.CONDENSE_MAX_TOKENS,
                verbose=0,
            )
        except Exception:  # any failure falls back — the turn must not die here (§4.6)
            logger.warning("condense LLM call failed — searching with the raw query", exc_info=True)
            return identity
        latency_s = time.perf_counter() - started

        # Mandatory: generate() returns reasoning tags verbatim, and an
        # unstripped <think> block would go straight into the embedding
        # model — the single easiest way to silently destroy retrieval
        # quality in this feature (spec_v3 §4.5).
        condensed = clean_response(raw)
        # A completion-primed model may echo the prompt's trailing label
        # into its answer (the chat-engine eval caught it live); an echoed
        # label would ride into the embedding model as noise.
        if condensed.upper().startswith(CONDENSE_QUERY_LABEL):
            condensed = condensed[len(CONDENSE_QUERY_LABEL) :].strip()
        if not condensed:
            logger.warning("condenser returned an empty query — searching with the raw query")
            return identity
        if len(condensed) > settings.CONDENSE_MAX_CHARS:
            logger.warning(
                "condensed query is %d chars (CONDENSE_MAX_CHARS=%d) — the condenser "
                "misbehaved; searching with the raw query",
                len(condensed),
                settings.CONDENSE_MAX_CHARS,
            )
            return identity
        v_condensed(query, condensed, verbose)
        return PreparedQuery(
            search_query=condensed,
            original_query=query,
            condensed=True,
            condense_latency_s=latency_s,
        )

    @staticmethod
    def _bounded(history: Sequence[Turn], limit: int) -> Sequence[Turn]:
        """Keep the newest ``limit`` turns (order preserved).

        API callers already bound their history load in SQL with the same
        setting; this bound covers history-accumulating callers like the
        CLI, whose in-memory list grows for the whole session.

        Args:
            history: Prior turns, oldest first.
            limit: Maximum turns to keep; ``0`` keeps none (a negative
                slice bound would wrap, so it is handled explicitly).

        Returns:
            The newest ``limit`` turns, oldest first.
        """
        if limit <= 0:
            return ()
        return history[-limit:]

`prepare(query, *, history, llm, verbose)` ¶

Condense the turn against its history into the retrieval query.

The first turn never condenses: empty history means there is nothing to resolve a reference against, and the common single-turn question must not pay an LLM round-trip for nothing. The history fed to the prompt is bounded by CONDENSE_HISTORY_TURNS (newest turns kept), whatever the caller loaded.

Parameters:

Name	Type	Description	Default
`query`	`str`	The user's question, verbatim.	required
`history`	`Sequence[Turn]`	Prior turns, oldest first (empty on a first turn).	required
`llm`	`LLMClient \| None`	Chat client; resolved via the model registry (`settings.CONDENSE_MODEL_TYPE`) when `None`.	required
`verbose`	`int`	Validated console verbosity (0–2).	required

Returns:

Type	Description
`PreparedQuery`	The condensed split, or the identity split whenever no LLM
`PreparedQuery`	call was made (kill switch, first turn) or the call's outcome
`PreparedQuery`	was unusable (the §4.6 fallback).

Source code in varagity/chat/condense.py

def prepare(
    self,
    query: str,
    *,
    history: Sequence[Turn],
    llm: LLMClient | None,
    verbose: int,
) -> PreparedQuery:
    """Condense the turn against its history into the retrieval query.

    The first turn never condenses: empty history means there is
    nothing to resolve a reference against, and the common single-turn
    question must not pay an LLM round-trip for nothing. The history
    fed to the prompt is bounded by ``CONDENSE_HISTORY_TURNS`` (newest
    turns kept), whatever the caller loaded.

    Args:
        query: The user's question, verbatim.
        history: Prior turns, oldest first (empty on a first turn).
        llm: Chat client; resolved via the model registry
            (``settings.CONDENSE_MODEL_TYPE``) when ``None``.
        verbose: Validated console verbosity (0–2).

    Returns:
        The condensed split, or the identity split whenever no LLM
        call was made (kill switch, first turn) or the call's outcome
        was unusable (the §4.6 fallback).
    """
    settings = get_settings()
    identity = PreparedQuery(
        search_query=query, original_query=query, condensed=False, condense_latency_s=None
    )
    if not settings.CONDENSE_ENABLED:
        logger.info(
            "CONDENSE_ENABLED=false — degrading to the 'simple' engine's identity split"
        )
        return identity
    turns = self._bounded(history, settings.CONDENSE_HISTORY_TURNS)
    if not turns:
        return identity

    prompt = CONDENSE_PROMPT.format(history=format_history(turns), query=query)
    # CONDENSE_MODEL_TYPE is validated to the LLM aliases, so the
    # registry always resolves an LLMClient here; the cast (rather than
    # an isinstance gate) keeps duck-typed test doubles usable at this
    # seam, matching the flows' injectable-fake convention.
    client = (
        llm if llm is not None else cast("LLMClient", get_model(settings.CONDENSE_MODEL_TYPE))
    )
    started = time.perf_counter()
    try:
        # verbose=0: the sub-call renders nothing; v_condensed below is
        # this stage's console output (the reranked-retriever pattern).
        raw = client.generate(
            [{"role": "user", "content": prompt}],
            max_tokens=settings.CONDENSE_MAX_TOKENS,
            verbose=0,
        )
    except Exception:  # any failure falls back — the turn must not die here (§4.6)
        logger.warning("condense LLM call failed — searching with the raw query", exc_info=True)
        return identity
    latency_s = time.perf_counter() - started

    # Mandatory: generate() returns reasoning tags verbatim, and an
    # unstripped <think> block would go straight into the embedding
    # model — the single easiest way to silently destroy retrieval
    # quality in this feature (spec_v3 §4.5).
    condensed = clean_response(raw)
    # A completion-primed model may echo the prompt's trailing label
    # into its answer (the chat-engine eval caught it live); an echoed
    # label would ride into the embedding model as noise.
    if condensed.upper().startswith(CONDENSE_QUERY_LABEL):
        condensed = condensed[len(CONDENSE_QUERY_LABEL) :].strip()
    if not condensed:
        logger.warning("condenser returned an empty query — searching with the raw query")
        return identity
    if len(condensed) > settings.CONDENSE_MAX_CHARS:
        logger.warning(
            "condensed query is %d chars (CONDENSE_MAX_CHARS=%d) — the condenser "
            "misbehaved; searching with the raw query",
            len(condensed),
            settings.CONDENSE_MAX_CHARS,
        )
        return identity
    v_condensed(query, condensed, verbose)
    return PreparedQuery(
        search_query=condensed,
        original_query=query,
        condensed=True,
        condense_latency_s=latency_s,
    )

`varagity.chat.prompts` ¶

The condense prompt (spec_v3 §4.5) and its history formatting.

A module-level constant following ANSWER_PROMPT (:mod:varagity.generation.answer) exactly — plain :meth:str.format, no templating engine — sent as a single user message via the non-streaming :meth:~varagity.models.llm.LLMClient.generate path.

The topic-shift instruction is the prompt's load-bearing line: dragging the old topic into a question that changed the subject is the failure mode that matters most for a naive condenser (spec_v3 §13.4), and the chat-engine eval fixtures test for it explicitly.

`format_history(turns)` ¶

Render conversation turns into the prompt's <conversation> block.

Parameters:

Name	Type	Description	Default
`turns`	`Sequence[Turn]`	Prior turns, oldest first.	required

Returns:

Type	Description
`str`	One `role: content` line per turn, in order.

Source code in varagity/chat/prompts.py

def format_history(turns: Sequence[Turn]) -> str:
    """Render conversation turns into the prompt's ``<conversation>`` block.

    Args:
        turns: Prior turns, oldest first.

    Returns:
        One ``role: content`` line per turn, in order.
    """
    return "\n".join(f"{turn.role}: {turn.content}" for turn in turns)

Chat engines¶

varagity.chat.base ¶

ChatEngine ¶

prepare(query, *, history, llm, verbose) ¶

PreparedQuery dataclass ¶

Turn dataclass ¶

get_chat_engine(name) ¶

register(name) ¶

varagity.chat.simple ¶

SimpleChatEngine ¶

prepare(query, *, history, llm, verbose) ¶

varagity.chat.condense ¶

CondenseContextEngine ¶

prepare(query, *, history, llm, verbose) ¶

varagity.chat.prompts ¶

format_history(turns) ¶

`varagity.chat.base` ¶

`ChatEngine` ¶

`prepare(query, *, history, llm, verbose)` ¶

`PreparedQuery` `dataclass` ¶

`Turn` `dataclass` ¶

`get_chat_engine(name)` ¶

`register(name)` ¶

`varagity.chat.simple` ¶

`SimpleChatEngine` ¶

`prepare(query, *, history, llm, verbose)` ¶

`varagity.chat.condense` ¶

`CondenseContextEngine` ¶

`prepare(query, *, history, llm, verbose)` ¶

`varagity.chat.prompts` ¶

`format_history(turns)` ¶