ADR 0024: BM25 over the chunk corpus, scored in the domain

Status

Accepted. 0012 no ANN index in a multi-tenant vector store

refused the same trade for the semantic channel and is the precedent this

decision follows for the lexical one. [0022 the lexical channel is not

BM25](0022-the-lexical-channel-is-not-bm25.md) is amended a second time, in

its Status only: the sentence *"the name 'BM25' appears nowhere under

src/"* no longer holds. 0023 the chunk corpus

stands, and this document is the "later" it deferred to.

Context

0022 refused to call the entity lexical channel BM25 because there was

nothing to run it on -- no stored text, so no document collection to gather

statistics over. 0023 built the corpus and stopped there deliberately: "every

decision a search method would encode is downstream of what a stored passage

is", and shipping the wrong method costs an adapter divergence in a way

shipping none does not.

The corpus now exists. ChunkStore has two adapters, InMemoryChunkStore

and PostgresChunkStore, and both need to answer the same ranking query the

same way -- the compliance suite in tests/compliance is what would notice

if they did not.

Decision

Scoring is a pure function in `domain/`; recall and statistics are the adapter's job

domain/bm25.py::bm25_score takes term frequencies, a document length, and

CorpusStats, and returns a number. It imports nothing store-shaped and

knows nothing about a query or a tenant. domain/chunk_ranking.py::rank_chunks

calls it over a LexicalCandidates a store produced and returns

RankedChunks, ordered.

Rejected: ts_rank_cd in Postgres, with the in-memory adapter approximating it.

Postgres has a text-search ranking function built in, and using it would mean

writing less code. It is rejected for the reason 0022 already established for

tokenization and 0023 established for chunk identity: two adapters computing

a rank by two different formulas can agree on which chunks match and still

disagree on their order, and nothing would notice except a caller comparing

results across backends. Putting the arithmetic in one place the compliance

suite can call directly against both adapters' candidate sets is what makes

"the two adapters rank identically" a thing that gets asserted rather than

hoped for. The adapters' job is narrower and does not admit the same

divergence: hand back which chunks contain which terms, and how many, and

n_docs / avg_doc_length / doc_frequencies for the corpus. Those are

counts, not rankings, and two adapters counting the same rows cannot disagree.

The tokenizer is domain-owned, or the purity of the scorer buys nothing

domain/tokenize.py::tokenize is the one function that decides what a term

is, called by both adapters to build their term index and by a caller to

build a query's term list. If it were not shared -- if, say, Postgres used its

built-in english text-search configuration and the in-memory adapter

approximated it in Python -- every count CorpusStats carries would already

have diverged before bm25_score ever ran, and a pure scorer over

inconsistent inputs is not a pure system. Tokenization is upstream of every

number BM25 computes, which is why it is the one piece of this design that

had to be decided first (Task 1) rather than last.

No stemming, and the cost is stated where it is paid

tokenize does not stem. "running" and "run" are different terms to it,

and a query for one does not match a passage containing only the other. The

alternative -- a stemmer -- is a language model: English-only, and a new

dependency, and "the Porter stemmer" is not one algorithm but a family of

slightly different ones, so two implementations (a Postgres extension and a

Python library, say) would reintroduce exactly the divergence a shared

tokenizer exists to prevent, one level up. Deferred as a single domain-owned

implementation if it is ever added -- BACKLOG B91, filed by Task 1, restated

here because this is where the cost is paid in practice: a real corpus, most

of the time, means a real recall gap from every unstemmed query.

Truncation is a stated total order, and its cost is bounded recall

lexical_candidates(terms, tenant_id, limit) cannot return every chunk that

matches any term -- that is a full scan of the corpus per query, the same

cost 0012 refused for the vector store's exact search and 0022 refused by

routing entity candidates through blocking keys instead. It truncates before

rank_chunks ever sees the candidates, ordered by **distinct matched term

count, then chunk id** -- computed before BM25 weighting, because the

weighting needs corpus-wide statistics, never statistics over the candidate

set truncation leaves behind, and truncating first by distinct term count is

what bounds the candidate set without scanning the whole corpus to rank it.

The consequence is the same shape 0022 already named for blocking: a chunk

matching one rare, highly informative term can be truncated away before a

chunk matching two common ones, even though IDF weighting would have scored

the rare-term match higher once both were candidates. A passage that would

have ranked first among all stored chunks can be **absent from the ranked

results entirely**, not merely low-ranked. docs/how-to/rank-passages.md

states this in the caller's own documentation, for the reason 0022 states its

blocking limit in docs/how-to/retrieve-entities.md: a missing result reads

as a bug rather than as a declared limit, unless the caller has been told.

`ON DELETE CASCADE` maintains the term index, because content addressing makes it immutable per id

PostgresChunkStore's

(tenant_id, chunk_id) back to the chunk table, ON DELETE CASCADE. Every

path that removes a chunk -- replace_source's orphan delete, delete_by_source,

delete_by_tenant -- deletes term rows for free, without any of the three

needing to know the term table exists. This is only safe because 0023 made

chunk ids content-addressed over (source_id, text): a chunk's term counts

are a pure function of its text, and a content-addressed id fixes that text

for good, so the term index for a given id is written once and never updated

in place. If chunk ids were positional instead, an in-place re-chunk would

need to update term rows rather than let a delete clean them up, and the

cascade would not be sufficient on its own.

The name "BM25" is honest here, for the first time in this codebase

0022 said the name would be wrong for the entity lexical channel because

there was no document collection and no meaningful corpus statistics to

compute — a three-word entity name has no length to normalise and no

informative document frequency. Neither objection applies to a corpus of

chunked passages: n_docs, avg_doc_length and doc_frequencies are

measured over real documents of real, varying length, and the formula in

domain/bm25.py is unmodified Robertson/Sparck-Jones IDF with the standard

saturation and length normalisation. **This is a real term-weighted ranker

over a real document corpus, and it is the first time the name has meant what

it claims under src/.**

0022's decision about the entity channel is untouched by this. It is still

a field-weighted string similarity over names and properties, it is still not

to be called BM25, and this ranker does not replace it: Acme Corp against

ACME Corporation is a string-proximity match this ranker has no way to make,

because a corporate name that short carries almost no term-frequency signal

distinguishing it from any other three-word string. Two channels this

document did not confuse is why both remain legible; a caller reaching for

"the lexical channel" now has to say which one.

0023's "no search method" was about timing, and the condition it named is met

0023 shipped ChunkStore with no search method because "every decision a

search method would encode is downstream of what a stored passage is", and

answering that with no corpus to test against would have been guessing. The

corpus has since been built, proven against real Postgres (Task 7), and this

document is the search method built once there was something to build it

against. 0023 is not overridden here -- the condition its argument named as

the reason to wait has been satisfied, which is a different thing from the

argument having been wrong.

Consequences

A caller can rank passages without a model call. Like index_documents,

lexical_candidates and rank_chunks ask nothing of an LlmProvider or an

EmbeddingProvider -- ranking a corpus is as cheap as indexing one.

**Chunk embeddings and a fused retrieve_chunks entry point still do not

exist.** This document builds the lexical half of what BACKLOG's now-former

B89 (B2) called for; the semantic half, ScoredChunk, and fusion with RRF

the way Retriever fuses entity channels are unbuilt and unscheduled.

**Corpus statistics are recomputed per query rather than maintained

incrementally.** n_docs, avg_doc_length and document frequencies are

counted at query time (count(*), avg(), per-term counts scoped to the

requested terms) rather than cached and kept in step with writes. This is

adequate at the scale this repository has measured anything at, and building

counters that are updated on every upsert_many / replace_source /

delete_by_source would be exactly the kind of speculative cost BACKLOG

warns against building ahead of a measured need.