The chunk corpus: stored passages, and what they know about the graph

Date: 2026-08-07 Status: approved, not yet implemented Scope: B1 of a two-part plan. B2 — chunk embeddings, a term-weighted ranker, and a public chunk retrieval surface — is deliberately not in this spec.

The problem

SlidingWindowChunker splits a document, hands the pieces to the LLM, and throws them away. Everything downstream of extraction therefore knows that an entity came from a document and not which passage said so, and there is no corpus over which a term-weighted ranker could compute a statistic.

ADR 0022 named both consequences and deferred both. This spec builds the corpus. It does not build the ranker.

What changes about the library's stance

ADR 0022 argued in part from "this library stores no text". That sentence described what had been built; it was not a decision. The rules that were decided are two, and neither is weakened here:

Retaining a caller-supplied passage violates neither. The ADR for this spec records the separation, and CLAUDE.md already carries it as of the commit preceding this one.

Decomposition, and why this half is first

B1 is first because B2's every decision is downstream of what a stored chunk is. Guessing a search signature before the corpus exists is how a port acquires a method its adapters cannot implement the same way.

The port therefore ships with no search method. Adding one to our own port later costs nothing; shipping the wrong one costs an adapter divergence.

Domain types

src/redstring/domain/chunk.py:


ChunkId = str


class StoredChunk(BaseModel):
    id: ChunkId
    tenant_id: TenantId
    source_id: SourceId
    text: str
    chunk_index: int
    start_char: int
    end_char: int
    entity_ids: list[EntityId] = []
    metadata: dict[str, Any] = {}

This is not extraction.chunking.Chunk, and the two must not be merged. That one is a dataclass in the extraction layer describing a split in progress: transient, tenantless, and consumed within a single pipeline run. This one is a stored record. They share four field names and no lifetime. A shared base class would put the extraction layer's type into the domain and give the transient one a tenant it has no way to fill.

text is validated by domain/json_safety.reject_unstorable_text, as Entity, Relationship and VectorRecord already are. A NUL byte in a passage is not hypothetical — it arrives from PDF extraction — and Postgres rejects it at INSERT rather than at the boundary.

`entity_ids`, and the sentence that is the whole contract

An empty entity_ids means no entities were extracted from this passage. It does not mean extraction is pending.

This has to be stated in those words on the type, because it is legitimately empty for an entire class of chunks: everything arriving through the direct ingest path, which never calls an LLM. Any code that reads emptiness as "not yet processed" will be wrong forever and will look reasonable in review.

The link points from chunk to entities and is stored on the chunk record, not as an edge in the graph store. GraphStore and ChunkStore are separate ports; a join across them is the caller's business, and putting a chunk reference into the graph would give mapping.py a second id scheme to keep in step — the specific hazard that kept consolidation a sibling layer rather than a consumer of extraction.

Identity: content-addressed


def chunk_id(source_id: SourceId, text: str) -> ChunkId: ...

A hash over (source_id, text) using the text exactly as stored, with no normalisation applied. Two passages differing only in whitespace are two passages — their start_char/end_char differ, so collapsing them would give one id two offsets — and normalising here would create a second scheme to keep in step with the one in mapping.py.

It lives in domain/chunk.py and not in extraction/mapping.py. Both write paths need it, extraction is a sibling layer, and the domain is the only place both can reach.

The source id is part of the hash. Identical boilerplate under two documents is two chunks; otherwise one document's entity_ids would attach to the other's passage, and the two would fight over the same row on every replay.

Positional identity was rejected. (source_id, chunk_index) is simpler and makes re-chunking an in-place overwrite, which is exactly the defect: chunk 3 of a re-chunked document is a different passage wearing the same id, so its stored entity links and — in B2 — its stored embedding would silently describe text that no longer says what they claim. Content addressing makes re-chunking produce new ids and leaves the old ones wrong-but-identifiable rather than lying.

The cost is orphans, and it is paid in the port.

The port

src/redstring/ports/chunk_store.py, tenant-scoped throughout, with no cross-tenant read. Like GraphStore and VectorStore it is a projection: the log is the authority and every write is idempotent because handlers replay.

Method Contract
upsert_many(chunks) Idempotent, last-write-wins per (tenant_id, id). Records may span tenants. One statement, not a loop.
get(chunk_id, tenant_id) The record or None. Unknown id is not an error. The result is the caller's; mutating it cannot change stored state.
get_by_source(source_id, tenant_id) Every chunk of that source, ordered by chunk_index ascending, ties broken by id.
replace_source(source_id, tenant_id, chunks) Write the incoming set and delete that source's chunks absent from it. See below.
delete_by_source(source_id, tenant_id) Count removed. Idempotent.
delete_by_tenant(tenant_id) Count removed. No other tenant touched.

replace_source is one call and not an upsert followed by a delete, so that folding one DocumentChunked event is a single atomic operation. Split in two, a crash between them leaves a corpus that is neither the old chunking nor the new one, and — once B2 lands — leaves document-frequency statistics computed over a set that never existed. An empty chunks argument is legal and means "this source now has no chunks"; it is not a no-op guard.

get_by_source orders by chunk_index and then by id, because chunk_index is not unique under content addressing: a re-chunk landing mid-replay can transiently produce two chunks claiming index 3. Ordering on the index alone would let the two adapters disagree about which comes first.

Write model

A DocumentChunked event in events/document.py, carrying source_id and the full chunk payload. projections/chunk.py folds it by calling replace_source — a whole document's chunking in one event and one call, which is what makes the fold atomic and the replacement correct.

Two paths emit it, and neither bypasses the log:

index_documents belongs in composition because it joins extraction (the chunkers) and projections — the same pair build_graph names, so the layer rule is satisfied by the argument already recorded rather than a new one. The layer's requirement is that a module name the pair of mutually-forbidden layers it joins; this one does.

A SourceId arriving by both routes is last-write-wins on the whole source, which follows from replace_source and needs no special case: indexing a document and later extracting it replaces the chunk set with one carrying entity links, and the reverse order drops them. That is a real behaviour, not an accident, and it is documented on index_documents — a caller who ingests after extracting has silently discarded the graph links.

Layer placement

A new top-level chunks package in the sibling band, holding chunks/adapters/memory.py and chunks/adapters/postgres.py.

containers = ["redstring"] with exhaustive = true means a new top-level package fails the contract until it is placed deliberately. Placing it requires, in the same commit that creates it:

That last one is not optional bookkeeping. Three of the existing four rows were confined by convention alone until slice 11, each correctly placed and each one commit from not being.

chunks sits beside graph and vector rather than above or below them: it holds a projection target, it needs nothing from either, and neither needs anything from it. A caller joining a chunk to its entities holds both ports, which is the same shape as every other cross-store question in this library.

Testing

A shared compliance suite, as GraphStore and VectorStore have, run against both adapters. Plus the introspection gate that derives the read-method list from the Protocol and fails when one lacks a registered mutation-isolation test and tenant-isolation test — GraphStore has this and ChunkStore gets it in the same edit, because the written rule is what failed the first four times.

Cases written against named rows of the failure-shape table in CLAUDE.md:

Before any hypothesis property is trusted, the implementation is broken on purpose and the property watched to fail.

Public surface

New exports: StoredChunk, ChunkStore, ChunkProjection, index_documents, and whatever report type index_documents returns. ChunkId is str, so it adds no closure; StoredChunk names TenantId, SourceId and EntityId, all exported already.

The end-to-end example gains an indexing step importing nothing but redstring.

Out of scope

Documentation