redstring re-architecture plan (historical)

This is history, not a plan. Every slice below is done, and it is written

in the future tense about work that has been finished, re-scoped, or

abandoned. Do not read it as a description of the library — read

docs/plans/ring-migration.md for what actually happened

and the ADRs in docs/adr/ for the decisions that outlived it.

It is kept for two things. Global Constraints below is what all eleven

slices were held to, and it is reusable for the next campaign of this kind.

And the "What redstring is not" and "Input contract" sections record scope

boundaries that are still enforced, which is why proposals to widen them keep

arriving without knowing what was already argued.

Archived unchanged in slice 11, from docs/plans/ring-migration.md.

Turn redstring into a library that constructs knowledge graphs, with

pluggable graph and vector storage — and nothing else. Clean breaks only:

no deprecation shims, no compatibility re-exports left behind.

This supersedes the original ring-migration plan (commits edced18,

cf64765), which assumed the existing package set was worth relocating.

Slices 0 and 0b from that plan are done and still stand; everything after

them is replaced.

Global Constraints

These bind every slice. Implementers and reviewers are held to all of them.

Testing

1. Red/green TDD, no exceptions. Write the test, *run it, watch it fail

for the right reason*, then write the minimal code to pass. A test that

passed the first time it ran proves nothing — delete it and start over.

Report the observed failure message, not just "tested".

1. **Property-based tests with hypothesis wherever a property is easier to

state than a table of examples.** Mandatory, not optional, for:

• the store port-compliance suites (any valid sequence of writes then

reads must round-trip; tenant isolation must hold for all inputs)

• normalization and parsing (idempotence: f(f(x)) == f(x))
• similarity and scoring (bounds, symmetry, identity)

Prefer a stated property over five hand-picked examples.

1. **Mutation testing on new domain and port logic — with cosmic-ray, not

mutmut, wherever the logic is decorated.** mutmut 3.x refuses to mutate

decorated functions. In a pydantic codebase nearly every invariant lives

in a @field_validator, @model_validator, or @property, so mutmut

reports a clean sweep while having tested none of them. Slice 2 proved

this: 5 mutants generated, 5 killed, and all 5 came from the one

undecorated function in the package.

Use cosmic-ray.toml (already in the repo, kept for exactly this reason

per CLAUDE.md) for decorated code; mutmut is fine for plain functions.

Surviving mutants are findings: either the tests are too weak or the code

is unreachable. Report the survivor count and what each survivor revealed.

Do not chase 100% — chase "every survivor is understood". If a mutation

tool cannot be scoped in reasonable time, say so plainly in the report;

a silent skip turns the gate into theatre.

1. No mocking what you own. The in-memory adapters exist precisely so

tests can use real implementations. Mock only genuinely external I/O

(an HTTP call to an LLM). A test that asserts on a mock's call args and

nothing else is a defect.

1. Tests must fail for exactly one reason. pytest-randomly randomises

order; order-dependent tests are bugs to fix, never seeds to pin.

Process

1. The pre-commit hook is the gate. Do not run ruff, bandit,

lint-imports, or the full pytest suite as separate pre-commit steps —

write the change and commit. Targeted pytest during the TDD loop

is expected and encouraged.

1. Deferred work goes in BACKLOG.md in the same commit that defers it,

with the reasoning that made deferring right. Delete an entry in the

commit that resolves it. Hard project rule (CLAUDE.md).

1. Clean breaks. No deprecation shims, no compatibility re-exports, no

"keep the old path working just in case". Delete and move on.

1. Never edit pyproject.toml dependency tables by hand — use

uv add, uv add --optional , uv remove.

1. Coverage may not fall. The ratchet enforces it. A deliberate drop

requires editing .coverage-baseline in the same commit, with the

reason in the message.

1. Small commits. Every hook run stays fast; every commit stays reviewable.

What redstring is

Given content, produce a knowledge graph. Entity and relationship

extraction, entity consolidation, temporal enrichment, embeddings.

**The event log is the write model. The graph and vector stores are

projections.** Extraction, consolidation, and temporal enrichment emit

domain events; projections fold those events into a GraphStore and a

VectorStore. Both stores are derived and disposable — rebuildable by

replay.

This is the property that matters most: **LLM extraction is the slow,

expensive, non-deterministic step.** Once its results are events, you can

re-run consolidation with new thresholds, a new blocking strategy, or a new

merge policy by replaying the log — without paying for extraction again.

Backend swaps and graph rebuilds fall out of the same mechanism, and they

independently prove the store ports.

eventsource-py supplies the event store, bus, and projection machinery

(checkpoints, DLQ, replay). It is already a dependency: context.py

re-exports eventsource.multitenancy, events/base.py re-exports

TenantDomainEvent, and four modules call register_event.

What we take from it. The event store, the event bus, the projection

system, and aggregates — at the right granularity.

An earlier revision of this plan said to skip the aggregate pattern, on the

grounds that "a document yielding ten thousand entities is not ten thousand

transactional aggregates". That argued against Entity as the aggregate and

wrongly concluded against the pattern. With Document as the aggregate,

ten thousand entities are one aggregate. It was also self-contradictory:

the stream design had already chosen document and consolidation

categories, and those are aggregate boundaries. And append() takes

expected: ExpectedVersion as a required argument, so declining aggregates

never avoided optimistic concurrency — it only meant hand-rolling it.

Not Entity — that part of the original objection stands.

ConsolidationLog is deliberately serialised per tenant: two concurrent

merges touching the same entities must not interleave. It is also the home

for three invariants that today are enforced by nothing — an entity

cannot be merged into something that is itself an alias, cannot be merged

twice, and an undo must reference a merge that happened. Those are precisely

the rules whose violation silently corrupts a graph, and having nowhere to

live is the real argument for the pattern.

TenantAwareRepository validates tenant consistency on the write path.

Tenant isolation is the property this project treats as most important, so

having it enforced by tested library code beats re-deriving it.

What redstring is not

Three scope cuts, decided deliberately:

The library does not fetch anything. Callers hand it content they already

have.

One nuance on state. The library owns the event log — that is the whole

point of the write model. What it does not own is a bespoke relational

schema for jobs, queues, and credentials. The event store is reached through

eventsource-py's port, so the caller still chooses the backend (in-memory,

SQLite, PostgreSQL) and the library remains runnable with no infrastructure

at all. SQLAlchemy therefore returns as a transitive dependency of the

event-store adapter — not as this library's persistence layer.

Input contract

A caller-supplied value object, not a row the library owns:

SourceDocument(
    id: str,                    # caller's identifier, used for provenance
    text: str,                  # already extracted and cleaned
    uri: str | None = None,
    title: str | None = None,
    published_at: datetime | None = None,
    metadata: dict = {},
)

Provenance becomes graph structure — (:Entity)-[:EXTRACTED_FROM]->(:Source)

— built from SourceDocument, never fetched.

Ports

Two that matter. Both are defined at the level the library actually uses, so

they stay implementable by something that is not Neo4j.

GraphStore
  upsert_entity / upsert_entities
  upsert_relationship / upsert_relationships
  get_entity, find_entities(name?, type?, tenant)
  neighbors(entity_id, depth, rel_types?)
  merge_entities(canonical, others)      # alias edges + provenance
  delete_by_tenant(tenant)

VectorStore
  upsert(entity_id, vector, tenant, metadata)
  search(vector, k, tenant, filter?)     # ANN
  get(entity_id, tenant)
  delete(entity_id | tenant)

Supporting ports, each with a default that needs no infrastructure:

LlmProvider, EmbeddingProvider, Cache (in-memory default; Redis

optional), Clock.

The event store, event bus, checkpoint, and DLQ ports come from

eventsource-py — we do not redefine them. GraphStore and VectorStore

are written by projection handlers subscribed to the log, never by

extraction or consolidation directly.

SourceDocument
      |
   extraction / consolidation / temporal
      |  emit
      v
  [ event log ]  <- the write model, the only authority
      |  project
      +--> GraphStore    (derived, disposable, rebuildable)
      +--> VectorStore   (derived, disposable, rebuildable)

The port must not leak Cypher. graph/queries.py is Cypher templates

today; they become Neo4j adapter internals, never port vocabulary.

Adapters

In-memory is the reference implementation and comes first, because it keeps

the port honest and gives the test suite a real backend for the first time

(BACKLOG B10). Every adapter runs the same port-compliance test suite —

that shared suite is the deliverable, not any one adapter.

SQLAlchemy does not vanish entirely: it survives inside

vector/adapters/pgvector.py and a future graph/adapters/sql.py, as an

optional extra. It stops being the library's persistence layer.

Target layout

src/redstring/
  domain/           Entity, Relationship, Alias, SourceDocument, temporal
                    value objects, domain events. Pure; no I/O, no ORM.
  ports/            GraphStore, VectorStore, LlmProvider, EmbeddingProvider,
                    Cache, Clock.
  graph/adapters/   memory, neo4j, (sql)
  vector/adapters/  memory, pgvector, (qdrant)
  llm/              provider adapters, retry, rate limiting, circuit breaker
  extraction/       entity/relationship extraction; chunking and chunk-merge
  consolidation/    blocking, similarity, merge
  temporal/         parsing, inference, timelines
  pipelines/        the composed use cases; the public API

preprocessing/ splits: HTML preprocessors are sourcing and go; the

sliding-window chunker and the entity mergers serve extraction and move into

extraction/.

What gets deleted

Roughly 7,800 lines deleted outright, before counting what services/

sheds. Dependencies dropped from the core: scrapy, trafilatura,

extruct, beautifulsoup4, boto3, unstructured, asyncpg,

psycopg2-binary, and sqlalchemy (demoted to an optional extra).

extraction/llm_extractor.py imports unstructured — document parsing

inside extraction. Same boundary violation; extraction takes text.

LLM strategy

Ollama-specific and OpenAI-specific extraction code goes. LangChain supplies

chat and embedding models; LangGraph orchestrates multi-step extraction;

deepagents provides an agentic extraction mode. pydantic-ai leaves

entirely, which resolves BACKLOG B19 (the ==0.0.31 pin) and B2

(the Anthropic stub) by deletion.

Verified test endpoint

http://192.168.1.14:8080/v1 — OpenAI-compatible, confirmed serving:

Unlike the Ollama box — which lists gpt-oss:20b but cannot load it

(BACKLOG B12) — this endpoint actually serves, so B12 becomes fixable

and extraction accuracy can be measured for the first time.

Note: qwen3.6-27b-mtp returned 200 with an empty

choices[0].message.content on a trivial prompt. Confirm the response shape

before wiring it; do not assume .content is populated.

Ports: LangChain goes behind them, not through them

LlmProvider.extract(text, schema, *, model?) -> structured result
EmbeddingProvider.embed(texts) -> list[vector]
EmbeddingProvider.dimension -> int

LangChain's BaseChatModel and Embeddings are **adapter

implementations**, never the port. domain/ must import nothing from

redstring beyond domain/, and certainly not an LLM framework — leaking

AIMessage into a domain type would undo the layering. LangChain's

interfaces also move fast; a breaking change should touch one adapter.

Embedding dimension is configuration, not a constant

nomic-embed-text is 768-dimensional. The codebase hardcodes 1024 (bge-m3)

at config.py:228 and — worse — bakes it into the pgvector column type

at models/extracted_entity.py:385 as Vector(EMBEDDING_DIMENSION).

VectorStore therefore takes dimension as configuration and **rejects

vectors whose length disagrees with it**. A dimension mismatch is a silent

correctness catastrophe that surfaces only as poor search results, so the

in-memory adapter must enforce the same check pgvector would — otherwise the

compliance suite passes on mixed dimensions that a real store rejects.

Switching embedding models means a new collection, not an in-place change.

`ExtractionMethod` must stop naming vendors

domain/entity.py currently carries LLM_OLLAMA, LLM_OPENAI,

LLM_CLAUDE, copied from the ORM. That is the wrong abstraction once

providers are LangChain adapters: the domain cares how an entity was

derived, not which vendor answered.

ExtractionMethod: LLM | PATTERN | SCHEMA_ORG | OPEN_GRAPH | HYBRID | MANUAL

with the concrete model recorded alongside as provenance

(model: str | None, e.g. "qwen3.6-27b-mtp") — strictly more useful, since

it survives model upgrades and supports "re-extract everything the old model

touched". This must land before slice 5b freezes the event schemas: an

ExtractionMethod inside a persisted event is permanent.

LangGraph: scoped, or it competes with the event log

LangGraph fits within-run orchestration — chunk → extract → merge → resolve

is genuinely a stateful graph, and that logic is hand-rolled today. The

boundary that must hold:

• LangGraph state is ephemeral working state for one extraction run.
• The event log is durable truth.

Do not use LangGraph's checkpointer as a second durable store; that

reintroduces exactly the dual-write problem this re-architecture exists to

remove. A run that dies is re-run, not resumed from a competing checkpoint.

deepagents: a second mode, not the default path

Agentic extraction — plan, use tools, iterate — is slower, costlier, and less

reproducible than deterministic bulk extraction. Those are the right

trade-offs for some documents and the wrong ones for a million. It arrives

behind the same LlmProvider port, selected per run, after the

deterministic path works end to end. It must not become the default by

accident.

Consolidation design

Consolidation is the part with the most real design work in it. It is not a

portability problem to be worked around — it gets built properly.

Blocking

Blocking generates candidate pairs worth scoring. Today's four strategies are

Postgres implementations of two different ideas:

So the port is two capabilities, not four:

BlockingKeyStrategy          # pure: entity -> set[key]
GraphStore.find_by_blocking_key(key, tenant)

Key-based blocking becomes pure domain logic that every adapter supports by

indexing keys. Fuzzy blocking goes through VectorStore.search, which is

where approximate matching already belongs. An adapter with a native fuzzy

index may override, but nothing depends on it having one.

Merge strategies

PropertyMergeStrategy currently has five members applied per-property:

PREFER_CANONICAL, PREFER_MERGED, UNION, LATEST, DEEP_MERGE.

The abstraction ports now; most of the implementations are deliberately

deferred:

• Implement now: PREFER_CANONICAL as the default, and UNION for

aliases — merging inherently produces alias sets, so that one is

structural rather than optional.

• Defer: PREFER_MERGED, LATEST, DEEP_MERGE. LATEST needs a

reliable updated-at on every property source; DEEP_MERGE needs nested

dict semantics that are easy to get subtly wrong and hard to undo. Neither

earns its complexity before there is a caller asking for it.

The port is shaped to accept the others later without redesign:

MergeStrategy:  resolve(property, canonical_value, other_values) -> value

Deferred strategies raise NotImplementedError with a message naming the

backlog entry, rather than silently degrading to a default.

Merge undo

undo_merge and split_entity both exist today and both need to know what

a merge did.

With the event log as the write model, this stops being a storage question.

A merge is an EntitiesMerged event; undoing it is a compensating

MergeUndone event, and the projection folds both. The pre-merge state is

recoverable from the log — that is what a log is for. No history table, and

no displaced-value payload smuggled onto an alias edge either.

Settled in slice 5b: Alias.displaced is deleted. What a merge displaces

is edge endpoints, and those are carried in typed form by

RelationshipRedirection on EntitiesMerged -- the edge before and after,

with after is None meaning the merge dropped it. See

docs/adr/0001-event-log-schema-and-granularity.md.

Test infrastructure

Slices 4 and 5 need backends that cannot be faked. Neither is reachable

today, and Docker is available.

redstring owns its own test backends. A docker-compose.test.yml in the

repo root provides Neo4j on host port 7688 and

pgvector/pgvector:pg16 on 5434 — deliberately off the default

ports so nothing collides with a local install, and specifically not plain

postgres, which does not ship the vector extension.

Do not borrow another project's test containers. eventsource-py runs its own

on 5433/6380; using them would couple two repos' test infrastructure so that

one suite can break the other, and a redstring failure would have a cause

outside redstring.

Adapter compliance runs are integration-marked and excluded from the

default gate (addopts = -m "not accuracy and not integration"). This is

what that long-declared, never-used marker is for (BACKLOG B10). The

in-memory adapters run the same compliance suite in the default gate,

always — if a test only passes in-memory, the suite is not doing its job.

Skip fixtures must prove the backend serves, not that a port is open.

Neo4j gets RETURN 1; pgvector gets CREATE EXTENSION IF NOT EXISTS vector

plus a round-trip row. This repo has already been burned by the weaker

check: the accuracy suite probed Ollama's model list, the model was listed

but could not load, and eight tests failed instead of skipping (BACKLOG B12).

Not using testcontainers: starting containers implicitly from inside a

pytest run lets an ordinary test invocation pull images and consume

resources without the developer asking. docker compose up -d is one

command and keeps that explicit.

Slices

Slices 0 and 0b are complete. Dispatch the rest one at a time.

Slice 1 is pure deletion and should land first — every later slice is

cheaper against a smaller tree.

Risks

• Port leakage. If the in-memory adapter turns out to be much harder than

Neo4j, the port is shaped like Cypher. That is the signal to redesign, and

it is why in-memory comes first.

• Coverage ratchet. Slice 1 deletes ~7,800 lines; total coverage will

move sharply and the baseline needs a deliberate edit with reasoning in the

commit message.

• Event schema is forever. A persisted log cannot be refactored the way

code can — a badly shaped event is permanent, or needs an upcaster. Slice

5b must not rush the event shapes. The 67 existing classes are raw

material, not a spec: they were written against the ORM, and none has ever

been emitted, so there is no compatibility to preserve and no excuse for

carrying their mistakes forward.

• Projection lag is real. Once the stores are derived, a read after a

write is not guaranteed to see it. The compliance suites must state

whether each store is read-your-writes or eventually consistent, and the

pipelines must not assume the former.

• Rebuild must be proven, not assumed. A projection that has never been

replayed from scratch does not work — it has only ever been fed live. The

replay-equivalence test in slice 5b is the guard, and it must run in CI,

not by hand.

• tenant_id scoping moves into the store ports (namespace, label, or

collection per tenant). Every port method takes it; the compliance suite

must prove isolation.

Backlog interaction

The scope cuts resolve several open entries by deletion rather than repair:

B6 (auth vestiges), B7 (db.py shape), B24 (no migration path — there will

be no relational schema to migrate), B3 (mark_sync_failed), and most of

B10 (no database in tests — the in-memory adapters become the test backend).

Delete each entry in the slice that actually removes the code, not before.

B2 (Anthropic provider stub), B5, B11, B13–B23, B25, B27 are unaffected.