Live Tenant Store Migration

Move a single tenant's events from one event store to another -- for example, off a shared PostgreSQL store and onto a dedicated one -- while the application keeps reading and writing. The source store stays authoritative until the moment of cutover, so at no point is there a window where events can be lost.

The work is driven by MigrationCoordinator from eventsource.application.migration, which walks a tenant through five phases (PENDING -> BULK_COPY -> DUAL_WRITE -> CUTOVER -> COMPLETED), and by TenantStoreRouter, a transparent proxy you put in front of your event store so application code never has to know a migration is running. Historical events stream across in the background, new events are written to both stores while the copy catches up, and the final switch is a sub-second write pause that rolls back automatically if it overruns its budget.

Follow the steps below in order the first time you run a migration. Each step is self-contained, so once you are familiar with the flow you can jump straight to the one you need -- monitoring sync lag, gating a cutover, verifying consistency, or aborting.

What this guide covers

By the end you will have moved one tenant's events onto a new store with the application still serving traffic, and you will know how to observe, gate, and undo each stage of that move.

The steps walk through:

This guide assumes you already have a working multi-tenant setup: tenant context is propagated on every operation, and both the source and target stores are reachable from the process running the coordinator. It does not cover choosing a tenant-isolation strategy or provisioning the target database -- see the Multi-Tenancy and Production Deployment guides for those.

Two things are deliberately out of scope. Migrating more than one tenant at a time is possible (each migration is independent and keyed by tenant), but this guide follows a single tenant end to end. And the terminal ABORTED and FAILED phases are covered only as escape hatches, not as a routine path -- a healthy migration ends in COMPLETED.

Prerequisites

Before you start, you need a source store, a target store, a PostgreSQL control-plane database, and the postgresql extra installed:


uv add "eventsource-py[postgresql]"   # pulls in asyncpg

The control plane is PostgreSQL-only. get_schema("migration") ships a PostgreSQL template only (there is no SQLite equivalent), and cutover coordination goes through PostgreSQLLockManager from eventsource.adapters.postgresql.locks. The stores you migrate between can be anything implementing FullEventStore (the combined ports surface: append, stream read, event lookup, global feed, and category query); the database that holds migration state cannot.

Required components

Source store -- the FullEventStore the tenant lives on today. It is the first positional argument to MigrationCoordinator(source_store=...), alongside the keyword-only source_store_id string (default "default") used as its key in routing records.

Target store -- a second FullEventStore instance, already provisioned with the events schema and reachable from the coordinator process. You pass it to start_migration(tenant_id, target_store, target_store_id, ...). It does not need to be the same backend as the source, but it should hold none of this tenant's events: nothing validates emptiness, and pre-existing events will skew the bulk-copy position mappings and the consistency check.

PostgreSQL advisory lock manager -- PostgreSQLLockManager(session_factory) from eventsource.adapters.postgresql.locks, built from a SQLAlchemy async_sessionmaker[AsyncSession]. It is a keyword-only, optional constructor argument, but the coordinator raises MigrationError when it needs to build its CutoverManager without one, so treat it as required for any migration you intend to finish. Each held lock occupies its own session, because PostgreSQL advisory locks are session-scoped -- size your connection pool accordingly.

The coordinator also takes two required repositories, MigrationRepository and TenantRoutingRepository, plus the TenantStoreRouter from Step 2. Three more arguments are optional: a PositionMapper wrapping a PositionMappingRepository and a CheckpointRepository (both needed for Step 10), and a MigrationAuditLogRepository (read in Step 9). The PostgreSQL repository implementations each take an AsyncConnection or AsyncEngine. Steps 2 and 3 wire all of these up.

Passing a PositionMapper is not by itself enough to get position mappings recorded: mappings are only written when the coordinator was constructed with one and MigrationConfig.position_mapping_enabled is True, which is the default. That gate has a cost -- with a mapper attached, the bulk copier appends one event at a time so each target position can be recorded, where it otherwise batches. Set position_mapping_enabled=False to keep the batched path if you do not need checkpoint translation (migrate_subscriptions will then have nothing to translate against).

Two application-side assumptions carry through the whole guide. Tenant context must be propagated on every store operation, since routing decisions are keyed by tenant. And application code must reach the event store through TenantStoreRouter rather than holding a direct reference to the source store -- otherwise writes will bypass dual-write and be lost at cutover.

Migration phases at a glance

MigrationPhase is a strict state machine. The happy path is five phases, and each one determines where TenantStoreRouter sends application traffic:

Phase What is happening Reads Writes
PENDING Migration record created, not started Source Source
BULK_COPY Historical events streaming to target Source Source
DUAL_WRITE New events written to both stores Source Both, source first and authoritative
CUTOVER Brief pause while routing flips Source Blocked (WritePausedError past the pause timeout)
COMPLETED Migration finished Target Target

Note that reads keep coming from the source right up until cutover completes -- the target is never read-served while it is still catching up.

Two properties on MigrationPhase are worth knowing, because later steps use them: is_active is true for BULK_COPY, DUAL_WRITE, and CUTOVER, and is_terminal is true for COMPLETED, ABORTED, and FAILED. A status loop that exits on status.phase.is_terminal stops on success and on failure alike.

The forward transitions the coordinator drives are:

MigrationRepository enforces this with a VALID_TRANSITIONS table, so an out-of-order phase update is rejected rather than corrupting state.

Sync lag is anchored to the last source position the target has actually copied; once BULK_COPY completes, that anchor only advances through the dual-write mirror, and a mirror failure during DUAL_WRITE clamps it in place rather than letting the reported lag drift wrong.

When that happens, run an in-phase resync rather than aborting:

remaining = await coordinator.run_resync_pass(migration.id) while remaining: remaining = await coordinator.run_resync_pass(migration.id)

Each call runs one bounded catch-up copy pass while the migration stays in DUAL_WRITE, and returns the number of unabsorbed mirror failures left. A return of 0 means the lag anchor is unclamped and cutover can proceed once lag drains. Bounding the retries is your policy, not the library's: a count that stops falling is a mirror problem to investigate, not a pass to repeat.

Warning — a nonzero cutover_max_lag_events accepts event loss. The lag it tolerates is not optimistic slack: safe_lag_anchor guarantees every counted event is provably absent from the target. Writes are paused for the whole cutover and nothing in the sequence copies the residue, so any lag remaining at the routing switch is events the target never receives while it becomes authoritative. The default is 0. If a cutover refuses because lag will not drain, the remedy is run_resync_pass (above), not a higher threshold.

ABORTED and FAILED are the two off-ramps. abort_migration() is available from PENDING, BULK_COPY, and DUAL_WRITE. It raises MigrationError in CUTOVER. FAILED is reachable from BULK_COPY, DUAL_WRITE, and CUTOVER. In CUTOVER, it is the outcome of an unrecoverable error or of a failed post-switch check (see Cutover safety rules). Neither off-ramp is reachable once a migration is COMPLETED -- terminal phases have no outgoing transitions at all, so a finished migration stays finished.

One transition is a rollback rather than a failure: CUTOVER -> DUAL_WRITE. A cutover that overruns cutover_timeout_ms does not land in FAILED; it returns to DUAL_WRITE with the source still authoritative, and you can retry once whatever caused the delay clears (Step 8).

Pausing is not a phase. pause_migration() sets an is_paused flag on the migration record while the phase stays BULK_COPY, so a paused copy still reports its phase normally and routing is undisturbed. It raises MigrationError if the migration has already reached a terminal phase.

Alongside MigrationPhase, each tenant's routing record carries a TenantMigrationState (NORMAL, BULK_COPY, DUAL_WRITE, CUTOVER_PAUSED, MIGRATED). That is the value the router actually reads to pick a store; the coordinator keeps it in step with the phase, so in practice you observe the phase and let routing look after itself.

Cutover safety rules

These rules hold for every migration. Read them before you call trigger_cutover() or abort_migration().

When a check fails, the outcome is visible in three places:

A clean cutover returns success=True with consistency_verified=True. Its phase becomes COMPLETED.

Step 1: Apply the migration control-plane schema

The coordinator keeps all of its state in four PostgreSQL tables. Create them before you construct anything else -- the repositories in Steps 2 and 3 assume the tables already exist and do not create them on demand.

The DDL ships with the library:


from eventsource.adapters.sql.schemas import get_schema

schema_sql = get_schema("migration")

The script is a multi-statement PostgreSQL script that also defines two plpgsql trigger functions, so most SQLAlchemy execution paths will reject it (exec_driver_sql() runs a single statement at a time). Hand the whole script to the raw asyncpg connection instead:


from sqlalchemy.ext.asyncio import create_async_engine

engine = create_async_engine("postgresql+asyncpg://.../control_plane")

async with engine.begin() as conn:
    raw_conn = await conn.get_raw_connection()
    await raw_conn.driver_connection.execute(schema_sql)

Every statement is CREATE TABLE IF NOT EXISTS / CREATE INDEX IF NOT EXISTS / CREATE OR REPLACE FUNCTION, so re-running the script is safe. If you manage schema with Alembic, paste the script into a revision's upgrade() rather than calling get_schema() at runtime.

`get_schema("migration")` and the four tables

get_schema("migration") returns the PostgreSQL template only. There is no SQLite variant -- get_schema("migration", backend="sqlite") raises ValueError, because the schema relies on JSONB, partial indexes, and plpgsql triggers. The migration tables are also not part of get_all_schemas(); that combined script covers events, outbox, checkpoints, DLQ, and snapshots. You have to apply the migration schema explicitly.

The four tables it creates:

Table Holds Read by
tenant_migrations One row per migration: phase, progress counters, per-phase timestamps, JSONB config, error and pause state MigrationRepository
tenant_routing One row per tenant: current store_id, migration_state, and a nullable active_migration_id TenantRoutingRepository, and through it TenantStoreRouter
migration_position_mappings Source position -> target position, with the event ID for correlation PositionMappingRepository / PositionMapper
migration_audit_log Append-only trail of migration events, with old_phase/new_phase, JSONB details, and operator MigrationAuditLogRepository

Three details worth knowing before you start:

updated_at on tenant_migrations and tenant_routing is maintained by BEFORE UPDATE triggers, so the timestamps are trustworthy even for writes that do not go through the repositories.

Where to apply it

Apply this schema to a control-plane database, not to the per-tenant event stores you are migrating between. The four tables describe migrations about stores; nothing in them is part of any store's event data, and no store implementation reads them. Only the repositories and the coordinator do.

Concretely:

The advisory lock manager is a separate decision. PostgreSQLLockManager takes an async_sessionmaker[AsyncSession], and pointing it at the control-plane database is the obvious choice, since that database is already PostgreSQL and already in the coordinator's dependency set. It does not have to be, though -- advisory locks are namespaced by key within whichever PostgreSQL instance holds them, so any PostgreSQL database that every coordinator process can reach will do. What matters is that all processes that might run a migration for the same tenant use the same lock database; two coordinators locking in different databases are not mutually exclusive at all.

Two placements to avoid:

Co-locating the control plane with an existing application or metadata database is fine and common. The requirements are only that it is PostgreSQL 12+, that it is reachable from every process that constructs a MigrationCoordinator, and that its lifecycle is independent of the stores being migrated.

One further consequence of the split: subscription checkpoints stay wherever your CheckpointRepository already keeps them. Step 10 rewrites checkpoint positions in place using the position mappings from the control plane; it does not move the checkpoint table itself.

Step 2: Put the router in front of the event store

TenantStoreRouter wraps your event store and sends each call to the store that the tenant uses now. Application code calls the router. It never calls the source store directly.


from eventsource.adapters.sql.migration import PostgreSQLTenantRoutingRepository
from eventsource.application.migration import TenantStoreRouter

routing_repo = PostgreSQLTenantRoutingRepository(control_engine)

router = TenantStoreRouter(
    source_store,
    routing_repo,
    stores={"default": source_store},
    default_store_id="default",
    write_pause_timeout=5.0,
)

Step 3: Create the coordinator

MigrationCoordinator runs every phase. Build it once per process.


from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker

from eventsource.adapters.postgresql.locks import PostgreSQLLockManager
from eventsource.adapters.sql.checkpoints import SQLCheckpointRepository
from eventsource.adapters.sql.migration import (
    PostgreSQLMigrationAuditLogRepository,
    PostgreSQLMigrationRepository,
    PostgreSQLPositionMappingRepository,
)
from eventsource.application.migration import MigrationCoordinator, PositionMapper

session_factory = async_sessionmaker(control_engine, class_=AsyncSession)

coordinator = MigrationCoordinator(
    source_store,
    PostgreSQLMigrationRepository(control_engine),
    routing_repo,
    router,
    source_store_id="default",
    lock_manager=PostgreSQLLockManager(session_factory),
    position_mapper=PositionMapper(PostgreSQLPositionMappingRepository(control_engine)),
    checkpoint_repo=SQLCheckpointRepository(projection_engine),
    audit_log_repo=PostgreSQLMigrationAuditLogRepository(control_engine),
)

Step 4: Start the migration and watch the bulk copy

start_migration() creates the migration record, sets routing to BULK_COPY, and starts the copy as a background task. The call returns before the copy finishes.


from eventsource.application.migration import MigrationConfig, MigrationPhase
from eventsource.adapters.postgresql.store import PostgreSQLEventStore

target_store = PostgreSQLEventStore(target_engine)  # schema already applied

migration = await coordinator.start_migration(
    tenant_id=tenant_id,
    target_store=target_store,
    target_store_id="dedicated",
    config=MigrationConfig(batch_size=1000, max_bulk_copy_rate=5000),
)

Watch the copy progress until it leaves BULK_COPY:


async for status in coordinator.stream_status(migration.id, update_interval=2.0):
    print(status.phase.value, f"{status.progress_percent:.1f}%", status.events_copied)
    if status.phase != MigrationPhase.BULK_COPY:
        break

Pause and resume the copy when you need to. A pause takes effect after the current batch:


await coordinator.pause_migration(migration.id)
await coordinator.resume_migration(migration.id)

The coordinator moves the migration to DUAL_WRITE when the copy finishes. To block until then, call wait_for_phase(migration.id, MigrationPhase.DUAL_WRITE, timeout=3600). It also returns early on a terminal phase, so check the returned phase.

start_migration() raises MigrationAlreadyExistsError when the tenant already has an active migration.

Step 5: Monitor dual-write and sync lag

In DUAL_WRITE, the router writes each new event to the source first and then to the target. The source stays authoritative. A failed target write does not fail the application write. The interceptor records the failed write, and sync lag grows until the event reaches the target.


lag = await coordinator.get_sync_lag(migration.id)
if lag is not None:
    print(lag.events)

Step 6: Gate the cutover and trigger it

Before the cutover, stop other writers for this tenant, or make sure they use the router. The write pause is held in process memory (see Cutover safety rules).


ready, reason = await coordinator.is_cutover_ready(migration.id)
if not ready:
    raise RuntimeError(f"Not ready for cutover: {reason}")

result = await coordinator.trigger_cutover(migration.id)

trigger_cutover() pauses writes, checks lag, switches routing to the target, and then runs the post-switch checks. It does not raise on a failed cutover. It returns a CutoverResult:

With the default configuration, the post-switch checks are verification (Step 7) and subscription migration (Step 10). A failed check ends the migration in FAILED, not COMPLETED.

Step 7: Verify consistency

Verify before you trigger the cutover. The target store must be registered, which start_migration() does for you.


from eventsource.application.migration import VerificationLevel

report = await coordinator.verify_consistency(
    migration.id,
    level=VerificationLevel.HASH,
    sample_percentage=100.0,
)
if not report.is_consistent:
    for violation in report.violations:
        print(violation)

Step 8: Handle a cutover timeout and retry

cutover_timeout_ms limits how long writes stay paused. Its default is 500 ms. A cutover also rolls back when sync lag is too high. In both cases the result has success=False and rolled_back=True, and the migration returns to DUAL_WRITE. The source stays authoritative. No exception reaches your code.


result = await coordinator.trigger_cutover(migration.id)
if not result.success and result.rolled_back:
    print(result.error_message)

To retry, first find the cause:

``python result = await coordinator.trigger_cutover(migration.id, timeout_ms=1000) ``

The migration keeps its DUAL_WRITE phase between attempts, so you can retry. The migration record stores the failure in last_error as Cutover failed: .... Set a limit on retries in your own code.

Step 9: Abort, read metrics, and read the audit log

abort_migration() stops a migration and restores routing to the source. It is allowed in PENDING, BULK_COPY, and DUAL_WRITE. It raises MigrationError in CUTOVER.


abort_result = await coordinator.abort_migration(
    migration.id, reason="operator stopped rollout"
)

The result has final_phase=MigrationPhase.ABORTED. The target keeps the events it already received. Nothing is deleted from it.

Metrics

Read the in-process metrics while the migration is active:


from eventsource.application.migration import get_migration_metrics

snapshot = get_migration_metrics(str(migration.id), str(tenant_id)).get_snapshot()
print(snapshot.events_copied, snapshot.sync_lag_events)
print(snapshot.cutover_durations, snapshot.cutover_failures)
print(snapshot.verification_failures, snapshot.failed_target_writes)

The OpenTelemetry instruments are:

The coordinator drops the in-process instance when a migration reaches COMPLETED, FAILED, or ABORTED. After that, get_migration_metrics() returns a new, empty instance. Read the snapshot before the migration ends, or export the OpenTelemetry instruments for long-term records.

Audit log

Read the audit trail with the repository that you passed in Step 3:


audit_repo = PostgreSQLMigrationAuditLogRepository(control_engine)

for entry in await audit_repo.get_by_migration(migration.id):
    old = entry.old_phase.value if entry.old_phase else "-"
    new = entry.new_phase.value if entry.new_phase else "-"
    print(entry.occurred_at.isoformat(), entry.event_type.value, old, "->", new)

Entries are ordered by occurred_at, oldest first. A failed post-switch check writes a MIGRATION_FAILED entry. A clean cutover writes CUTOVER_COMPLETED.

Step 10: Move subscription checkpoints

A subscription stores its position in the source store's position space. The target store uses different positions. During DUAL_WRITE, the coordinator records a position mapping for each event. Subscription migration uses those mappings to rewrite each checkpoint in place.

Subscription migration needs:

With the default configuration, trigger_cutover() runs subscription migration after the switch. Read the result with get_subscription_summary():


summary = coordinator.get_subscription_summary(migration.id)
if summary is not None:
    print(summary.successful_count, summary.failed_count, summary.skipped_count)

To run it yourself for selected subscriptions, start with a dry run. A dry run computes the new positions and writes nothing:


summary = await coordinator.migrate_subscriptions(
    migration.id,
    subscription_names=["OrderSummaryProjection"],
    dry_run=True,
)

A checkpoint that is older than every recorded mapping cannot be translated. It counts as a failed subscription. When this happens during cutover, the migration ends in FAILED.