Answers that cite their sources

A language model will happily tell you anything. That confidence is exactly the problem when the subject matter is something people care about getting right.

For our conversational projects — chief among them the chat over Śrīla Prabhupāda’s lectures — we set a hard rule early: every answer must be traceable to a trusted source, or the assistant says it does not know. No exceptions, no graceful-sounding guesses.

This is the article where we stop hand-waving and show the machinery: how the corpus is stored, what gets discarded before generation, the exact metric that decides whether we know the answer at all, and what happens when we don’t.

The shape of the problem

Retrieval-augmented generation is the usual answer — fetch relevant passages, hand them to the model, ask it to answer from those passages. It works, until the question falls outside the corpus. Then the model quietly reverts to its training data and starts improvising.

That failure is invisible to the user. The answer looks just as fluent as a grounded one. Fluency is not truth. So the whole design question becomes: how do you measure the gap between “we have material for this” and “we are about to make something up” — cheaply, and before the expensive generation runs?

How we store the sources

You cannot cite what you did not store well. Everything searchable — a slice of a lecture, a verse, a paragraph of commentary — is one fragment, and every fragment is tagged with its kind. Those kinds are the vocabulary the rest of the system speaks: track_transcript, verse, commentary, prose_chapter, letter. Almost every later decision — reranking, the per-kind reserves, the coverage gate — keys off that tag.

What a fragment stores depends on what it is, because a citation has to point at the right thing:

• A transcript fragment belongs to a track — a single lecture recording — and keeps (track_id, start_ms, end_ms): which recording, and the exact millisecond span inside it. That triple is what a citation chip replays when you tap it; getting it right is the whole point of storing audio this way.
• A library fragment (a verse, a commentary, a letter) keeps a structural address instead — the book, the location token, the human label — so “BG 2.13” resolves to a real verse rather than a lucky phrase match.

Both carry the same essentials underneath: the text, its language, and a vector embedding for meaning-search.

Two chunkers, deliberately different. Lectures and scripture don’t chunk the same way, so they don’t. Transcripts are cut into ~45-second windows with a small overlap and an 8000-character cap — long enough to hold a complete thought, short enough to point at precisely. Library text is packed by paragraph and sentence into ~900-character chunks; a verse and its commentary are already structured, so we follow that structure instead of fighting it.

Embeddings. Every fragment is embedded with text-embedding-3-small and indexed for cosine similarity with pgvector’s HNSW. That is the “by meaning” lane.

The lexical lane has its own indexes, because meaning-similarity is blind to exact strings. A pg_trgm GIN index over addr_label || source_id || tokens catches canonical addresses; two tsvector GIN indexes cover content — one russian (Snowball stemming) and one simple (no stemming, so Sanskrit transliteration survives verbatim).

Curated memory is stored apart, in an attributions table whose kind is pinned (an editor-blessed, citable answer to a specific question), boost (a topic that nudges scores), or memory (a background note that shapes framing but is never quoted). The interesting detail: when the indexer mirrors a note into Postgres, it embeds the trigger phrases and the chunks of the note body together, so a question that echoes the note’s substance — not just its trigger — still finds it. The full mechanics of curated memory are in Defense in depth against hallucination.

What we throw out before the model sees anything

Recall first, precision later. The question is expanded into 1–4 typed sub-questions (each with up to two paraphrases) by a cheap model, and every lane runs for each. Then the discarding begins.

flowchart TD
  Q["Question + sub-questions"] --> POOL["Candidate pool<br/>(dense × 3 kinds + lexical + address)"]
  POOL --> F{"cosine pre-floor<br/>0.18 on rerank path"}
  F -->|below| X["discard as pure noise"]
  F -->|above, or forced| CAP["cap to 60 by cosine"]
  CAP --> RR["Voyage rerank-2<br/>(cross-encoder)"]
  RR --> K["keep top 16 by rerank score"]
  K --> RES["+ per-kind reserves<br/>≥2 verses, ≥2 library"]
  RES --> DD["dedup + compact to<br/>cited notes only"]
  DD --> N["Grounded notes"]

A few numbers matter, and they were all learned the hard way:

• The noise pre-floor is 0.18, not 0.45. We used to drop anything below a flat 0.45 cosine — and it silently killed relevant verses sitting around 0.30 that a cross-encoder would have rescued. So on the live path the pre-floor drops to RERANK_NOISE_PREFLOOR = 0.18 (just enough to discard garbage) and the reranker decides the rest. Lexical and address hits are forced past the floor entirely, so a rare name is never lost to a shy embedding.
• The reranker is the real chooser. The pool is capped to the top 60 by cosine, every (question, passage) pair is scored jointly by Voyage rerank-2, and the top 16 by that score survive. The cut is by rank, not an absolute score threshold — cross-encoder scores are not calibrated across different questions, so there is no fixed cutoff to set.
• Per-kind reserves stop the answer from starving. A cross-encoder trained on prose quietly favours chatty transcripts over terse verses, so the cut is followed by guaranteed reserves: at least 2 verses and 2 library passages, each above a 0.40 cosine floor of their own.
• Then we dedup and compact. Duplicate clips are collapsed, and right before generation the pool is trimmed to only the notes the plan actually cites — a 30–90-note context becomes the union of the outline’s supporting notes, renumbered. A bloated context only invites lost-in-the-middle drift and off-plan citations.

The full reranking story — and why the model is shown a bare integer instead of a forgeable track ID — lives in A citation a model cannot fake.

The coverage check: one cheap metric, no LLM

Here is the part the first draft of this article waved at. Discarding weak hits is not the same as deciding the pool as a whole is too thin to answer. That decision is the gate, and it is a handful of plain booleans — no model call:

COVERAGE_MIN_MAX_SCORE = 0.55
COVERAGE_MIN_LECTURES  = 2
_EARLY_EXIT_MAX_SCORE  = 0.65
_BAILOUT_MAX_SCORE     = 0.40

def is_coverage_sufficient(result):
    if result.max_score < COVERAGE_MIN_MAX_SCORE:   # 0.55
        return False
    return len(result.by_kind.get("lecture", [])) >= COVERAGE_MIN_LECTURES  # 2

def is_coverage_good_enough(result):
    if is_coverage_sufficient(result):
        return True
    return result.max_score >= _EARLY_EXIT_MAX_SCORE   # one confident hit: 0.65

def should_bail_out(result):
    return result.max_score < _BAILOUT_MAX_SCORE       # nothing close: 0.40

The metric is cosine, on purpose. max_score is the maximum cosine similarity of the ranked set — never the rerank score. This is load-bearing: the cross-encoder scores are great for ordering within one question but are not comparable across questions, so they cannot anchor an absolute “do we know this?” threshold. Cosine can. Internally each candidate keeps its true cosine in score and the cross-encoder result in a separate field, precisely so the gates stay calibrated against the value they were tuned on.

So the verdicts read cleanly:

• max_score ≥ 0.55 and ≥ 2 lecture chunks → a solid pool; ground the answer.
• max_score ≥ 0.65 → one confident hit is enough on its own; stop searching early and save a round.
• max_score < 0.40 → nothing is close; bail out rather than pay for another retrieval round over the same emptiness.

Between those bounds the system runs a second fanout round (capped at two), then re-checks. And there is a last gate even after coverage passes: a cheap planner (gemini-2.5-flash) ranks the surviving notes into an outline, and if none of them actually support a thesis it returns an empty outline — Outline(theses=[]) — which flips a corpus_insufficient flag. That, not a zero-result retrieval, is what “nothing found” really means here: the planner read the candidates and judged them all off-topic.

Out of corpus → an honest answer, not a refusal

When corpus_insufficient is set, we used to emit a flat “не нашёл в корпусе.” Now there is a smarter path. A single stronger model is summoned — and this is the only place in the pipeline it appears:

llm_fallback_knowledge = "openrouter/anthropic/claude-sonnet-4.6"

The fallback model answers from general knowledge, in the user’s language, and returns a structured MemoryAnswer { answer, search_queries, disclaimer, in_scope }. From its own answer it derives 3–5 corpus probes, and we re-search — but admit only chunks clearing a hard _FALLBACK_MIN_SCORE = 0.5, so we never re-admit the junk the original query already rejected. Surviving chunks become optional, opportunistic citations: cited only where one genuinely supports a point, never fabricated, and the answer never refuses.

flowchart TD
  CI["corpus_insufficient"] --> CL["Claude Sonnet 4.6<br/>answer from general knowledge"]
  CL --> SC{"in scope?"}
  SC -->|"no (cooking, sport, code)"| OOS["polite decline"]
  SC -->|yes| RS["re-search on 3–5 derived probes<br/>keep only score ≥ 0.5"]
  RS --> D["disclaimer painted server-side<br/>+ faithful answer + any real notes"]

Two safety choices are worth calling out. First, the disclaimer is painted deterministically by the server, not trusted to the model — evaluation caught the model dropping its own mandated “answering from memory” line intermittently, so we emit it ourselves and tell the model it is already shown. Second, an out-of-scope branch: if the model reports the question is about cooking, sport, code or the news, we skip the re-search entirely and decline politely. The whole behaviour is behind an enable_corpus_fallback flag, overridable per turn.

Keeping latency low when there are two passes

The honest fallback means a strong model can run — so the entire pipeline is arranged so it rarely has to, and the cheap work always happens first.

flowchart LR
  QP["query plan<br/>flash-lite"] --> FN["fanout: embed + rerank<br/>no generation LLM"]
  FN --> CG["coverage gate<br/>pure booleans, no LLM"]
  CG --> SP["outline planner<br/>flash"]
  SP --> SY["synthesizer<br/>deepseek-chat (streams)"]
  CG -.->|only on miss| CF["Claude Sonnet 4.6<br/>(the expensive pass)"]

• A sufficiency pre-check picks the effort level with no LLM at all. If curated memory already answers the question, the turn takes a lean policy — no extra fanout rounds, a small slate — instead of the wide policy’s two rounds over ~20 candidates. The legacy short/long fork that this replaced cost a full ~100-source sweep, roughly 20 seconds, on questions that didn’t need it.
• The coverage gate is free. It is a few comparisons on numbers already in hand, so it can short-circuit before any generation. The 0.65 early-exit skips a second round; the 0.40 bail-out skips a query-regeneration LLM call (~7s) plus another fanout (~2s) — about 9 seconds saved on questions that were never going to be answered from the corpus.
• The model ladder runs cheap-first. Routing and query planning are gemini-3.1-flash-lite; the note-attribution planner is full gemini-2.5-flash (Flash-Lite hallucinated broken note refs ~5% of the time on the bench, so it was rejected for that one job); the synthesizer streams on deepseek-chat. Claude Sonnet 4.6 is summoned only on a coverage miss — never on a guess that a question “looks hard.” The intro is even written concurrently with grounding, so it costs almost no wall-clock.

Why “I don’t know” is a feature

Users trust a system that admits its limits far more than one that is confidently wrong once. Every honest refusal is a small deposit in a trust account you cannot refill after a single fabrication.

But notice what “I don’t know” actually is here: not a mood, not a tone the model adopts, but a number crossing a line — max_score < 0.40, or a planner returning an empty outline. It is logged, it is measured, and because it is measured it can be tuned. A faithfulness judge runs offline on claude-opus-4.8 at temperature 0, blind and with sides assigned deterministically so it cannot favour a position, scoring every answer for faithfulness, completeness, structure, and balance. Honesty here is an engineered quantity, not a hope.

That is the whole philosophy in miniature: we would rather ship something that says less and means it — and we would rather the “less” be a threshold we can read off a dashboard than a sentiment we cross our fingers about.

Where this is going

This architecture underpins more than one project in the studio. We have now written up the pieces that used to be promises: the layered defense that fetches and discards before the model speaks, the un-fakeable citation format, and the way the same chat answers every reader in their own language while resting on the original sources.