Backend Benchmark Matrix Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: A standalone async benchmark harness (bench/) that runs the same scenario catalog across every EventStore, EventBus, and SnapshotStore backend plus the end-to-end aggregate path, emitting JSON results and a Markdown report.

Architecture: Adapters (one small class per backend, mirroring the conformance-suite factory pattern) plug backends into declarative scenarios (async callable + parameter grid). A runner expands scenarios × params × backends into cells, measures each with warmup/calibration/3 rounds inside one event loop, and writes a versioned JSON file that a report generator renders to Markdown tables.

Tech Stack: Python 3.11, stdlib only for the harness itself (argparse, asyncio, statistics, json, gc, time). Backends come from eventsource and its optional extras, imported only behind availability checks. No new dependencies.

Spec: docs/superpowers/specs/2026-07-29-backend-benchmark-matrix-design.md

Deliberate deviations from the spec (both minor, flagged for review):

  1. The spec's snapshot.save_load row is implemented as two uniform scenarios, snapshot.save and snapshot.load, so every cell has exactly one metric. Both metrics the spec asks for are preserved.
  2. The spec's metadata list includes "per-service versions"; the harness records commit/python/platform/cpu/library version but not live service versions — service identity comes from the pinned images in docker-compose.bench.yml (postgres:15, redis:7, rabbitmq:3-management, apache/kafka:3.7.0), which the report's audience can cross-reference. Wiring live version queries through every adapter was judged not worth the complexity.

Global Constraints


Task 1: Package scaffolding and results model

Files:

Interfaces:


# tests/unit/bench/test_results.py
"""Tests for the bench result data model and JSON round-trip."""

from pathlib import Path

from bench.core.results import (
    SCHEMA_VERSION,
    CellResult,
    LatencyStats,
    Round,
    RunResult,
)


def _cell(status: str = "ok") -> CellResult:
    latency = LatencyStats.from_durations([0.001, 0.002, 0.003, 0.004, 0.005])
    rounds = [
        Round(elapsed_s=0.5, operations=100, ops_per_sec=200.0, latency=latency, counters={}),
        Round(elapsed_s=0.4, operations=100, ops_per_sec=250.0, latency=latency, counters={}),
        Round(
            elapsed_s=0.6,
            operations=100,
            ops_per_sec=166.0,
            latency=latency,
            counters={"conflicts": 3},
        ),
    ]
    return CellResult(
        scenario="store.append_batch",
        interface="store",
        backend="memory",
        metric="throughput",
        params={"batch_size": 10, "payload": "small"},
        status=status,
        reason=None,
        rounds=rounds,
    )


def test_latency_stats_from_durations() -> None:
    stats = LatencyStats.from_durations([0.001, 0.002, 0.003, 0.004, 0.010])
    assert stats.min_ms == 1.0
    assert stats.p50_ms == 3.0
    assert stats.p99_ms > stats.p50_ms
    assert stats.mean_ms == 4.0


def test_cell_id_is_stable_and_param_sorted() -> None:
    cell = _cell()
    assert cell.cell_id == "store.append_batch[memory](batch_size=10,payload=small)"


def test_median_round_selected_by_ops_per_sec() -> None:
    cell = _cell()
    median = cell.median_round
    assert median is not None
    assert median.ops_per_sec == 200.0


def test_median_round_none_when_no_rounds() -> None:
    cell = _cell(status="skipped")
    cell.rounds = []
    assert cell.median_round is None


def test_json_round_trip(tmp_path: Path) -> None:
    run = RunResult(
        schema_version=SCHEMA_VERSION,
        metadata={"commit": "abc123", "python": "3.11"},
        cells=[_cell()],
    )
    path = tmp_path / "run.json"
    run.save(path)
    loaded = RunResult.from_json(path.read_text())
    assert loaded.schema_version == SCHEMA_VERSION
    assert loaded.metadata["commit"] == "abc123"
    assert loaded.cells[0].cell_id == run.cells[0].cell_id
    assert loaded.cells[0].rounds[2].counters == {"conflicts": 3}
    assert loaded.cells[0].rounds[0].latency is not None
    assert loaded.cells[0].rounds[0].latency.p50_ms == 3.0

Run: uv run pytest tests/unit/bench/test_results.py -v -p no:randomly Expected: FAIL — ModuleNotFoundError: No module named 'bench'

Create the four __init__.py files, each containing only a docstring, e.g. bench/__init__.py:


"""Internal cross-backend benchmark harness. Not part of the eventsource package."""

Create bench/results/.gitkeep (empty file). Append to .gitignore:


# Benchmark harness output
bench/results/
!bench/results/.gitkeep

# bench/core/results.py
"""Result data model for benchmark runs.

The JSON schema is versioned via SCHEMA_VERSION so future regression
tooling can diff two run files (spec: Reporting section).
"""

import json
import statistics
from dataclasses import asdict, dataclass, field
from pathlib import Path
from typing import Any

SCHEMA_VERSION = 1


def _percentile(sorted_values: list[float], pct: float) -> float:
    if not sorted_values:
        raise ValueError("no samples")
    k = (len(sorted_values) - 1) * pct / 100.0
    lo = int(k)
    hi = min(lo + 1, len(sorted_values) - 1)
    frac = k - lo
    return sorted_values[lo] * (1 - frac) + sorted_values[hi] * frac


@dataclass(frozen=True)
class LatencyStats:
    p50_ms: float
    p95_ms: float
    p99_ms: float
    mean_ms: float
    min_ms: float

    @classmethod
    def from_durations(cls, durations_s: list[float]) -> "LatencyStats":
        ms = sorted(d * 1000.0 for d in durations_s)
        return cls(
            p50_ms=_percentile(ms, 50),
            p95_ms=_percentile(ms, 95),
            p99_ms=_percentile(ms, 99),
            mean_ms=statistics.fmean(ms),
            min_ms=ms[0],
        )


@dataclass(frozen=True)
class Round:
    elapsed_s: float
    operations: int
    ops_per_sec: float
    latency: LatencyStats | None = None
    counters: dict[str, int] = field(default_factory=dict)


@dataclass
class CellResult:
    scenario: str
    interface: str
    backend: str
    metric: str  # "latency" | "throughput"
    params: dict[str, Any]
    status: str  # "ok" | "skipped" | "failed"
    reason: str | None = None
    rounds: list[Round] = field(default_factory=list)

    @property
    def cell_id(self) -> str:
        rendered = ",".join(f"{key}={self.params[key]}" for key in sorted(self.params))
        return f"{self.scenario}[{self.backend}]({rendered})"

    @property
    def median_round(self) -> Round | None:
        if not self.rounds:
            return None
        ordered = sorted(self.rounds, key=lambda r: r.ops_per_sec)
        return ordered[len(ordered) // 2]


@dataclass
class RunResult:
    schema_version: int
    metadata: dict[str, Any]
    cells: list[CellResult]

    def to_json(self) -> str:
        return json.dumps(asdict(self), indent=2, default=str)

    @classmethod
    def from_json(cls, text: str) -> "RunResult":
        raw = json.loads(text)
        cells = []
        for cell_raw in raw["cells"]:
            rounds = []
            for round_raw in cell_raw["rounds"]:
                latency_raw = round_raw.get("latency")
                latency = LatencyStats(**latency_raw) if latency_raw else None
                rounds.append(
                    Round(
                        elapsed_s=round_raw["elapsed_s"],
                        operations=round_raw["operations"],
                        ops_per_sec=round_raw["ops_per_sec"],
                        latency=latency,
                        counters=round_raw.get("counters", {}),
                    )
                )
            cells.append(
                CellResult(
                    scenario=cell_raw["scenario"],
                    interface=cell_raw["interface"],
                    backend=cell_raw["backend"],
                    metric=cell_raw["metric"],
                    params=cell_raw["params"],
                    status=cell_raw["status"],
                    reason=cell_raw.get("reason"),
                    rounds=rounds,
                )
            )
        return cls(
            schema_version=raw["schema_version"],
            metadata=raw["metadata"],
            cells=cells,
        )

    def save(self, path: Path) -> None:
        path.write_text(self.to_json())

Run: uv run pytest tests/unit/bench/test_results.py -v -p no:randomly Expected: 5 passed


uv run ruff check bench/ tests/unit/bench/ --fix && uv run ruff format bench/ tests/unit/bench/
git add bench/ tests/unit/bench/ .gitignore
git commit -m "feat: add bench package scaffolding and results model"

Task 2: Bench domain — events, payloads, aggregate, snapshot state

Files:

Interfaces:


# tests/unit/bench/test_domain.py
"""Tests for bench domain events, payload generators, and the bench aggregate."""

from uuid import uuid4

from bench.core.domain import (
    PAYLOAD_SIZES,
    SNAPSHOT_SIZES,
    BenchCounter,
    BenchEvent,
    make_events,
    make_registry,
    make_snapshot_state,
)


def test_make_events_versions_and_payload_size() -> None:
    aggregate_id = uuid4()
    events = make_events(aggregate_id, count=3, start_version=5, payload="large")
    assert [e.aggregate_version for e in events] == [5, 6, 7]
    assert all(e.aggregate_id == aggregate_id for e in events)
    assert all(len(e.payload) == PAYLOAD_SIZES["large"] for e in events)
    assert [e.seq for e in events] == [0, 1, 2]


def test_make_snapshot_state_size() -> None:
    state = make_snapshot_state(SNAPSHOT_SIZES["medium"])
    assert len(state["blob"]) == SNAPSHOT_SIZES["medium"]


def test_bench_counter_applies_increments() -> None:
    counter = BenchCounter(uuid4())
    counter.increment()
    counter.increment(2)
    assert counter.state.value == 3
    assert counter.version == 0  # not yet persisted
    assert len(counter.get_uncommitted_events()) == 2


def test_make_registry_contains_bench_events() -> None:
    registry = make_registry()
    assert registry.get_event_class("BenchEvent") is BenchEvent

Run: uv run pytest tests/unit/bench/test_domain.py -v -p no:randomly Expected: FAIL — ModuleNotFoundError / ImportError for bench.core.domain

Note: tests/fixtures/aggregates.py has DeclarativeCounterAggregate as the reference for aggregate mechanics, and tests/unit/ aggregate tests show accessor names. If counter.state / counter.version / get_uncommitted_events() differ in the real DeclarativeAggregate API, adjust the test to the real accessors (check src/eventsource/aggregates/base.py) — do not change the aggregate.


# bench/core/domain.py
"""Shared events, payload generators, and a tiny aggregate for benchmarks.

Payload sizes are defined here once so every scenario benchmarks identical
data (spec: fairness rules).
"""

from typing import Any
from uuid import UUID

from pydantic import BaseModel

from eventsource import DomainEvent, EventRegistry
from eventsource.aggregates.base import DeclarativeAggregate
from eventsource.handlers import handles

PAYLOAD_SIZES: dict[str, int] = {"small": 200, "large": 5_000}
SNAPSHOT_SIZES: dict[str, int] = {"small": 1_000, "medium": 50_000, "large": 500_000}


class BenchEvent(DomainEvent):
    """Generic benchmark event with a size-controlled payload."""

    event_type: str = "BenchEvent"
    aggregate_type: str = "Bench"
    payload: str = ""
    seq: int = 0


class BenchCounterIncremented(DomainEvent):
    event_type: str = "BenchCounterIncremented"
    aggregate_type: str = "BenchCounter"
    increment: int = 1


class BenchCounterState(BaseModel):
    counter_id: UUID
    value: int = 0


class BenchCounter(DeclarativeAggregate[BenchCounterState]):
    """Minimal aggregate for the end-to-end repository benchmark."""

    aggregate_type = "BenchCounter"

    def _get_initial_state(self) -> BenchCounterState:
        return BenchCounterState(counter_id=self.aggregate_id)

    @handles(BenchCounterIncremented)
    def _on_incremented(self, event: BenchCounterIncremented) -> None:
        if self._state is None:
            self._state = self._get_initial_state()
        self._state = self._state.model_copy(
            update={"value": self._state.value + event.increment}
        )

    def increment(self, amount: int = 1) -> None:
        event = BenchCounterIncremented(
            aggregate_id=self.aggregate_id,
            aggregate_type=self.aggregate_type,
            aggregate_version=self.get_next_version(),
            increment=amount,
        )
        self._raise_event(event)


def make_events(
    aggregate_id: UUID,
    count: int,
    start_version: int = 1,
    payload: str = "small",
) -> list[BenchEvent]:
    body = "x" * PAYLOAD_SIZES[payload]
    return [
        BenchEvent(
            aggregate_id=aggregate_id,
            aggregate_version=start_version + i,
            payload=body,
            seq=i,
        )
        for i in range(count)
    ]


def make_snapshot_state(size_bytes: int) -> dict[str, Any]:
    return {"blob": "x" * size_bytes}


def make_registry() -> EventRegistry:
    registry = EventRegistry()
    registry.register(BenchEvent)
    registry.register(BenchCounterIncremented)
    return registry

If EventRegistry.get_event_class or .register have different names, mirror what tests/integration/conftest.py does with its registry and adjust the test accordingly.

Run: uv run pytest tests/unit/bench/test_domain.py -v -p no:randomly Expected: 4 passed


uv run ruff check bench/ tests/unit/bench/ --fix && uv run ruff format bench/ tests/unit/bench/
git add bench/core/domain.py tests/unit/bench/test_domain.py
git commit -m "feat: add bench domain events and counter aggregate"

Task 3: Adapter contract and in-memory adapters

Files:

Interfaces:


# tests/unit/bench/test_adapters_memory.py
"""Tests for the adapter contract using the always-available memory adapters."""

from uuid import uuid4

from bench.adapters.buses import BUS_ADAPTERS, MemoryBusAdapter
from bench.adapters.snapshots import SNAPSHOT_ADAPTERS, MemorySnapshotAdapter
from bench.adapters.stores import STORE_ADAPTERS, MemoryStoreAdapter
from bench.core.domain import make_events


async def test_memory_store_adapter_lifecycle() -> None:
    adapter = MemoryStoreAdapter()
    assert adapter.name == "memory"
    assert await adapter.available() is None
    await adapter.setup()
    store = await adapter.create()
    aggregate_id = uuid4()
    result = await store.append_events(
        aggregate_id, "Bench", make_events(aggregate_id, 2), expected_version=0
    )
    assert result.success
    await adapter.destroy(store)
    await adapter.teardown()


async def test_memory_bus_adapter_delivers() -> None:
    adapter = MemoryBusAdapter()
    assert await adapter.available() is None
    await adapter.setup()
    bus = await adapter.create()
    received: list[object] = []

    async def handler(event: object) -> None:
        received.append(event)

    from bench.core.domain import BenchEvent

    bus.subscribe(BenchEvent, handler)
    await adapter.start_delivery(bus)
    aggregate_id = uuid4()
    await bus.publish(make_events(aggregate_id, 1))
    assert len(received) == 1
    await adapter.stop_delivery(bus)
    await adapter.destroy(bus)
    await adapter.teardown()


async def test_memory_snapshot_adapter_roundtrip() -> None:
    from datetime import UTC, datetime

    from eventsource.snapshots.interface import Snapshot

    adapter = MemorySnapshotAdapter()
    await adapter.setup()
    snapshot_store = await adapter.create()
    aggregate_id = uuid4()
    snapshot = Snapshot(
        aggregate_id=aggregate_id,
        aggregate_type="Bench",
        version=1,
        state={"blob": "x"},
        schema_version=1,
        created_at=datetime.now(UTC),
    )
    await snapshot_store.save_snapshot(snapshot)
    loaded = await snapshot_store.get_snapshot(aggregate_id, "Bench")
    assert loaded is not None and loaded.version == 1
    await adapter.destroy(snapshot_store)
    await adapter.teardown()


def test_registries_contain_memory() -> None:
    assert STORE_ADAPTERS["memory"] is MemoryStoreAdapter
    assert BUS_ADAPTERS["memory"] is MemoryBusAdapter
    assert SNAPSHOT_ADAPTERS["memory"] is MemorySnapshotAdapter

Run: uv run pytest tests/unit/bench/test_adapters_memory.py -v -p no:randomly Expected: FAIL — ModuleNotFoundError: No module named 'bench.adapters.base' (or similar)


# bench/adapters/base.py
"""Adapter contract: how a backend plugs into the harness.

Mirrors the conformance-suite factory pattern (see
src/eventsource/testing/conformance.py): scenarios never know which
backend they run on.
"""

from abc import ABC, abstractmethod
from typing import ClassVar, Generic, TypeVar

from eventsource.bus.interface import EventBus

T = TypeVar("T")


class BenchAdapter(ABC, Generic[T]):
    """Lifecycle: available? -> setup -> (create -> destroy)* -> teardown.

    create() is called once per matrix cell and must return an isolated,
    ready-to-use resource. destroy() releases it.
    """

    name: ClassVar[str] = ""

    async def available(self) -> str | None:
        """Return None if this backend can run, else a skip reason."""
        return None

    async def setup(self) -> None:
        """One-time session setup (schema creation, temp dirs)."""

    async def teardown(self) -> None:
        """One-time session cleanup."""

    @abstractmethod
    async def create(self) -> T:
        """Create a fresh resource for one cell."""

    async def destroy(self, resource: T) -> None:
        """Release a resource created by create()."""


class BusAdapter(BenchAdapter[EventBus]):
    """Bus adapters additionally manage consumer delivery.

    Scenarios subscribe handlers first, then call start_delivery();
    ordering matters for broker consumers.
    """

    async def start_delivery(self, bus: EventBus) -> None:
        """Begin delivering published events to subscribers (no-op for memory)."""

    async def stop_delivery(self, bus: EventBus) -> None:
        """Stop the consumer started by start_delivery()."""

# bench/adapters/stores.py
"""EventStore adapters. SQL backends are added by later tasks."""

from eventsource import InMemoryEventStore
from eventsource.stores.interface import EventStore

from bench.adapters.base import BenchAdapter


class MemoryStoreAdapter(BenchAdapter[EventStore]):
    name = "memory"

    async def create(self) -> EventStore:
        return InMemoryEventStore(enable_tracing=False)


STORE_ADAPTERS: dict[str, type[BenchAdapter[EventStore]]] = {
    MemoryStoreAdapter.name: MemoryStoreAdapter,
}

# bench/adapters/buses.py
"""EventBus adapters. Broker backends are added by later tasks."""

from eventsource import InMemoryEventBus
from eventsource.bus.interface import EventBus

from bench.adapters.base import BusAdapter


class MemoryBusAdapter(BusAdapter):
    name = "memory"

    async def create(self) -> EventBus:
        return InMemoryEventBus(enable_tracing=False)

    async def destroy(self, resource: EventBus) -> None:
        await resource.shutdown()


BUS_ADAPTERS: dict[str, type[BusAdapter]] = {
    MemoryBusAdapter.name: MemoryBusAdapter,
}

# bench/adapters/snapshots.py
"""SnapshotStore adapters. SQL backends are added by later tasks."""

from eventsource.snapshots.in_memory import InMemorySnapshotStore
from eventsource.snapshots.interface import SnapshotStore

from bench.adapters.base import BenchAdapter


class MemorySnapshotAdapter(BenchAdapter[SnapshotStore]):
    name = "memory"

    async def create(self) -> SnapshotStore:
        return InMemorySnapshotStore(enable_tracing=False)


SNAPSHOT_ADAPTERS: dict[str, type[BenchAdapter[SnapshotStore]]] = {
    MemorySnapshotAdapter.name: MemorySnapshotAdapter,
}

If InMemoryEventStore / InMemoryEventBus are not re-exported from the top-level eventsource package, import from their modules (eventsource.stores.in_memory, eventsource.bus.memory).

Run: uv run pytest tests/unit/bench/test_adapters_memory.py -v -p no:randomly Expected: 4 passed


uv run ruff check bench/ tests/unit/bench/ --fix && uv run ruff format bench/ tests/unit/bench/
git add bench/adapters/ tests/unit/bench/test_adapters_memory.py
git commit -m "feat: add bench adapter contract and in-memory adapters"

Task 4: Runner core — grid expansion, calibration, rounds, timeout, metadata

Files:

Interfaces:


# tests/unit/bench/test_runner.py
"""Runner tests using a fake in-process adapter and scenario."""

import asyncio
from typing import Any

from bench.adapters.base import BenchAdapter
from bench.core.runner import (
    Measurement,
    RunnerConfig,
    Scenario,
    expand_grid,
    run_cell,
    run_matrix,
)


class FakeAdapter(BenchAdapter[dict[str, Any]]):
    name = "fake"

    def __init__(self, reason: str | None = None) -> None:
        self.reason = reason
        self.setup_calls = 0
        self.teardown_calls = 0

    async def available(self) -> str | None:
        return self.reason

    async def setup(self) -> None:
        self.setup_calls += 1

    async def teardown(self) -> None:
        self.teardown_calls += 1

    async def create(self) -> dict[str, Any]:
        return {}


async def _fast_scenario_func(
    resource: dict[str, Any], params: dict[str, Any], iterations: int, prepared: Any
) -> Measurement:
    durations = [0.001] * iterations
    return Measurement(
        elapsed_s=0.001 * iterations, operations=iterations, durations_s=durations
    )


FAST = Scenario(
    name="fake.fast",
    interface="store",
    metric="latency",
    grid={"size": [1, 2]},
    func=_fast_scenario_func,
)

TINY_CONFIG = RunnerConfig(
    rounds=2,
    warmup_iterations=1,
    calibration_iterations=1,
    target_round_seconds=0.01,
    max_iterations=10,
    cell_timeout_seconds=5.0,
)


def test_expand_grid_full_and_quick() -> None:
    grid = {"a": [1, 2], "b": ["x", "y"]}
    assert len(expand_grid(grid, quick=False)) == 4
    assert expand_grid(grid, quick=True) == [{"a": 1, "b": "x"}]


async def test_run_cell_produces_rounds_and_latency() -> None:
    cell = await run_cell(FakeAdapter(), FAST, {"size": 1}, TINY_CONFIG)
    assert cell.status == "ok"
    assert len(cell.rounds) == 2
    assert cell.rounds[0].latency is not None
    assert cell.rounds[0].ops_per_sec > 0


async def test_run_cell_failure_is_captured() -> None:
    async def boom(
        resource: dict[str, Any], params: dict[str, Any], iterations: int, prepared: Any
    ) -> Measurement:
        raise RuntimeError("kaboom")

    scenario = Scenario(
        name="fake.boom", interface="store", metric="latency", grid={}, func=boom
    )
    cell = await run_cell(FakeAdapter(), scenario, {}, TINY_CONFIG)
    assert cell.status == "failed"
    assert cell.reason is not None and "kaboom" in cell.reason


async def test_run_cell_timeout_is_captured() -> None:
    async def hang(
        resource: dict[str, Any], params: dict[str, Any], iterations: int, prepared: Any
    ) -> Measurement:
        await asyncio.sleep(60)
        return Measurement(elapsed_s=0.0, operations=0)

    scenario = Scenario(
        name="fake.hang", interface="store", metric="latency", grid={}, func=hang
    )
    config = RunnerConfig(
        rounds=1,
        warmup_iterations=1,
        calibration_iterations=1,
        target_round_seconds=0.01,
        max_iterations=1,
        cell_timeout_seconds=0.2,
    )
    cell = await run_cell(FakeAdapter(), scenario, {}, config)
    assert cell.status == "failed"
    assert cell.reason is not None and "timeout" in cell.reason.lower()


class BadAdapter(FakeAdapter):
    name = "bad"


async def test_run_matrix_skips_unavailable_and_sets_metadata() -> None:
    good = FakeAdapter()
    bad = BadAdapter(reason="service down")
    result = await run_matrix([FAST], {"store": [good, bad]}, TINY_CONFIG)
    ok_cells = [c for c in result.cells if c.status == "ok"]
    skipped = [c for c in result.cells if c.status == "skipped"]
    assert len(ok_cells) == 2  # grid size 2 for the good adapter
    assert len(skipped) == 2 and skipped[0].reason == "service down"
    assert good.setup_calls == 1 and good.teardown_calls == 1
    assert bad.setup_calls == 0
    assert "python" in result.metadata and "timestamp" in result.metadata

Run: uv run pytest tests/unit/bench/test_runner.py -v -p no:randomly Expected: FAIL — ModuleNotFoundError: No module named 'bench.core.runner'


# bench/core/meta.py
"""Run metadata so reported numbers are never context-free (spec: methodology)."""

import os
import platform
import subprocess
import sys
from datetime import UTC, datetime
from typing import Any


def _git_commit() -> str:
    try:
        out = subprocess.run(
            ["git", "rev-parse", "HEAD"],
            capture_output=True,
            text=True,
            timeout=5,
            check=False,
        )
        return out.stdout.strip() or "unknown"
    except OSError:
        return "unknown"


def collect_metadata() -> dict[str, Any]:
    try:
        from importlib.metadata import version

        eventsource_version = version("eventsource-py")
    except Exception:
        eventsource_version = "unknown"
    return {
        "timestamp": datetime.now(UTC).isoformat(),
        "commit": _git_commit(),
        "python": sys.version.split()[0],
        "platform": platform.platform(),
        "cpu_count": os.cpu_count(),
        "eventsource_version": eventsource_version,
    }

(If the distribution name differs, check [project] name in pyproject.toml and use that string.)


# bench/core/runner.py
"""Matrix runner: warmup -> calibration -> timed rounds per cell.

All measurement for a cell happens inside the caller's event loop --
never a per-measurement asyncio.run() (spec: engine decision).
"""

import asyncio
import gc
import time
import traceback
from collections.abc import Awaitable, Callable
from dataclasses import dataclass, field, replace
from itertools import product
from typing import Any

from bench.adapters.base import BenchAdapter
from bench.core.meta import collect_metadata
from bench.core.results import (
    SCHEMA_VERSION,
    CellResult,
    LatencyStats,
    Round,
    RunResult,
)


@dataclass(frozen=True)
class Measurement:
    elapsed_s: float
    operations: int
    durations_s: list[float] | None = None
    counters: dict[str, int] = field(default_factory=dict)


PrepareFunc = Callable[[BenchAdapter[Any], Any, dict[str, Any]], Awaitable[Any]]
ScenarioFunc = Callable[[Any, dict[str, Any], int, Any], Awaitable[Measurement]]


@dataclass(frozen=True)
class Scenario:
    name: str
    interface: str  # "store" | "bus" | "snapshot" | "e2e"
    metric: str  # "latency" | "throughput"
    grid: dict[str, list[Any]]
    func: ScenarioFunc
    prepare: PrepareFunc | None = None


@dataclass(frozen=True)
class RunnerConfig:
    rounds: int = 3
    warmup_iterations: int = 3
    calibration_iterations: int = 5
    target_round_seconds: float = 2.0
    max_iterations: int = 10_000
    cell_timeout_seconds: float = 60.0
    quick: bool = False

    def effective(self) -> "RunnerConfig":
        if not self.quick:
            return self
        return replace(self, rounds=1, target_round_seconds=0.2)


def expand_grid(grid: dict[str, list[Any]], quick: bool) -> list[dict[str, Any]]:
    if not grid:
        return [{}]
    keys = list(grid)
    values = [[grid[k][0]] if quick else grid[k] for k in keys]
    return [dict(zip(keys, combo, strict=True)) for combo in product(*values)]


def _round_from_measurement(measurement: Measurement) -> Round:
    ops_per_sec = (
        measurement.operations / measurement.elapsed_s if measurement.elapsed_s > 0 else 0.0
    )
    latency = (
        LatencyStats.from_durations(measurement.durations_s)
        if measurement.durations_s
        else None
    )
    return Round(
        elapsed_s=measurement.elapsed_s,
        operations=measurement.operations,
        ops_per_sec=ops_per_sec,
        latency=latency,
        counters=measurement.counters,
    )


async def run_cell(
    adapter: BenchAdapter[Any],
    scenario: Scenario,
    params: dict[str, Any],
    config: RunnerConfig,
) -> CellResult:
    config = config.effective()
    cell = CellResult(
        scenario=scenario.name,
        interface=scenario.interface,
        backend=adapter.name,
        metric=scenario.metric,
        params=params,
        status="ok",
    )
    resource: Any = None
    try:
        async with asyncio.timeout(config.cell_timeout_seconds):
            resource = await adapter.create()
            prepared: Any = None
            if scenario.prepare is not None:
                prepared = await scenario.prepare(adapter, resource, params)

            await scenario.func(resource, params, config.warmup_iterations, prepared)

            calibration = await scenario.func(
                resource, params, config.calibration_iterations, prepared
            )
            per_iteration = max(
                calibration.elapsed_s / config.calibration_iterations, 1e-9
            )
            iterations = max(
                1,
                min(
                    int(config.target_round_seconds / per_iteration),
                    config.max_iterations,
                ),
            )

            for _ in range(config.rounds):
                gc.collect()
                measurement = await scenario.func(resource, params, iterations, prepared)
                cell.rounds.append(_round_from_measurement(measurement))
    except TimeoutError:
        cell.status = "failed"
        cell.reason = f"timeout after {config.cell_timeout_seconds}s"
    except Exception as exc:  # noqa: BLE001 - a failed cell must not kill the run
        cell.status = "failed"
        cell.reason = "".join(
            traceback.format_exception_only(type(exc), exc)
        ).strip()
    finally:
        if resource is not None:
            try:
                await adapter.destroy(resource)
            except Exception:  # noqa: BLE001 - teardown must not mask the cell result
                pass
    return cell


async def run_matrix(
    scenarios: list[Scenario],
    adapters: dict[str, list[BenchAdapter[Any]]],
    config: RunnerConfig,
) -> RunResult:
    cells: list[CellResult] = []
    for interface, interface_adapters in adapters.items():
        interface_scenarios = [s for s in scenarios if s.interface == interface]
        if not interface_scenarios:
            continue
        for adapter in interface_adapters:
            reason = await adapter.available()
            if reason is not None:
                for scenario in interface_scenarios:
                    for params in expand_grid(scenario.grid, config.quick):
                        cells.append(
                            CellResult(
                                scenario=scenario.name,
                                interface=scenario.interface,
                                backend=adapter.name,
                                metric=scenario.metric,
                                params=params,
                                status="skipped",
                                reason=reason,
                            )
                        )
                continue
            await adapter.setup()
            try:
                for scenario in interface_scenarios:
                    for params in expand_grid(scenario.grid, config.quick):
                        cells.append(await run_cell(adapter, scenario, params, config))
            finally:
                await adapter.teardown()
    return RunResult(
        schema_version=SCHEMA_VERSION,
        metadata=collect_metadata(),
        cells=cells,
    )

Note: asyncio.timeout() requires Python 3.11+ (the project floor).

Run: uv run pytest tests/unit/bench/test_runner.py -v -p no:randomly Expected: 5 passed


uv run ruff check bench/ tests/unit/bench/ --fix && uv run ruff format bench/ tests/unit/bench/
git add bench/core/runner.py bench/core/meta.py tests/unit/bench/test_runner.py
git commit -m "feat: add bench matrix runner with calibration and timeouts"

Task 5: EventStore scenarios

Files:

Interfaces:


# tests/unit/bench/test_scenarios_stores.py
"""Run every store scenario against the memory adapter with tiny budgets."""

import pytest

from bench.adapters.stores import MemoryStoreAdapter
from bench.core.runner import RunnerConfig, run_cell
from bench.scenarios.stores import STORE_SCENARIOS

TINY = RunnerConfig(
    rounds=1,
    warmup_iterations=1,
    calibration_iterations=1,
    target_round_seconds=0.02,
    max_iterations=5,
    cell_timeout_seconds=30.0,
)

SMALLEST_PARAMS = {
    "store.append_batch": {"batch_size": 1, "payload": "small"},
    "store.read_stream": {"stream_length": 100},
    "store.concurrent_append": {"writers": 2},
    "store.contended_append": {"writers": 2},
}


@pytest.mark.parametrize("scenario", STORE_SCENARIOS, ids=lambda s: s.name)
async def test_store_scenario_runs_on_memory(scenario) -> None:  # type: ignore[no-untyped-def]
    adapter = MemoryStoreAdapter()
    await adapter.setup()
    cell = await run_cell(adapter, scenario, SMALLEST_PARAMS[scenario.name], TINY)
    await adapter.teardown()
    assert cell.status == "ok", cell.reason
    assert cell.rounds and cell.rounds[0].operations > 0


def test_grids_match_spec() -> None:
    by_name = {s.name: s for s in STORE_SCENARIOS}
    assert by_name["store.append_batch"].grid == {
        "batch_size": [1, 10, 100, 1000],
        "payload": ["small", "large"],
    }
    assert by_name["store.read_stream"].grid == {"stream_length": [100, 1000, 10000]}
    assert by_name["store.concurrent_append"].grid == {"writers": [1, 10, 50]}
    assert by_name["store.contended_append"].grid == {"writers": [1, 10, 50]}

Run: uv run pytest tests/unit/bench/test_scenarios_stores.py -v -p no:randomly Expected: FAIL — import error on bench.scenarios.stores


# bench/scenarios/stores.py
"""EventStore scenarios: append, read, concurrency, contention."""

import asyncio
import time
from typing import Any
from uuid import UUID, uuid4

from eventsource.exceptions import OptimisticLockError
from eventsource.stores.interface import EventStore

from bench.adapters.base import BenchAdapter
from bench.core.domain import make_events
from bench.core.runner import Measurement, Scenario


async def populate_stream(
    store: EventStore,
    aggregate_id: UUID,
    count: int,
    payload: str = "small",
    chunk: int = 500,
) -> None:
    version = 0
    while version < count:
        n = min(chunk, count - version)
        events = make_events(aggregate_id, n, start_version=version + 1, payload=payload)
        await store.append_events(aggregate_id, "Bench", events, expected_version=version)
        version += n


async def _append_batch(
    store: EventStore, params: dict[str, Any], iterations: int, prepared: Any
) -> Measurement:
    batch_size: int = params["batch_size"]
    payload: str = params["payload"]
    durations: list[float] = []
    start = time.perf_counter()
    for _ in range(iterations):
        aggregate_id = uuid4()
        events = make_events(aggregate_id, batch_size, payload=payload)
        t0 = time.perf_counter()
        await store.append_events(aggregate_id, "Bench", events, expected_version=0)
        durations.append(time.perf_counter() - t0)
    return Measurement(
        elapsed_s=time.perf_counter() - start,
        operations=iterations * batch_size,
        durations_s=durations,
    )


async def _prepare_read_stream(
    adapter: BenchAdapter[Any], store: EventStore, params: dict[str, Any]
) -> UUID:
    aggregate_id = uuid4()
    await populate_stream(store, aggregate_id, params["stream_length"])
    return aggregate_id


async def _read_stream(
    store: EventStore, params: dict[str, Any], iterations: int, prepared: Any
) -> Measurement:
    aggregate_id: UUID = prepared
    stream_length: int = params["stream_length"]
    durations: list[float] = []
    start = time.perf_counter()
    for _ in range(iterations):
        t0 = time.perf_counter()
        stream = await store.get_events(aggregate_id, "Bench")
        durations.append(time.perf_counter() - t0)
        if stream.version != stream_length:
            raise RuntimeError(
                f"expected {stream_length} events, read {stream.version}"
            )
    return Measurement(
        elapsed_s=time.perf_counter() - start,
        operations=iterations * stream_length,
        durations_s=durations,
    )


async def _concurrent_append(
    store: EventStore, params: dict[str, Any], iterations: int, prepared: Any
) -> Measurement:
    writers: int = params["writers"]
    ops_per_writer = max(1, iterations // writers)

    async def writer() -> None:
        aggregate_id = uuid4()
        for version in range(ops_per_writer):
            events = make_events(aggregate_id, 1, start_version=version + 1)
            await store.append_events(
                aggregate_id, "Bench", events, expected_version=version
            )

    start = time.perf_counter()
    async with asyncio.TaskGroup() as group:
        for _ in range(writers):
            group.create_task(writer())
    return Measurement(
        elapsed_s=time.perf_counter() - start,
        operations=writers * ops_per_writer,
    )


async def _contended_append(
    store: EventStore, params: dict[str, Any], iterations: int, prepared: Any
) -> Measurement:
    writers: int = params["writers"]
    ops_per_writer = max(1, iterations // writers)
    aggregate_id = uuid4()
    conflicts = 0
    lock = asyncio.Lock()

    async def writer() -> None:
        nonlocal conflicts
        done = 0
        while done < ops_per_writer:
            version = await store.get_stream_version(aggregate_id, "Bench")
            events = make_events(aggregate_id, 1, start_version=version + 1)
            try:
                result = await store.append_events(
                    aggregate_id, "Bench", events, expected_version=version
                )
                conflicted = result.conflict
            except OptimisticLockError:
                conflicted = True
            if conflicted:
                async with lock:
                    conflicts += 1
                continue
            done += 1

    start = time.perf_counter()
    async with asyncio.TaskGroup() as group:
        for _ in range(writers):
            group.create_task(writer())
    return Measurement(
        elapsed_s=time.perf_counter() - start,
        operations=writers * ops_per_writer,
        counters={"conflicts": conflicts},
    )


STORE_SCENARIOS: list[Scenario] = [
    Scenario(
        name="store.append_batch",
        interface="store",
        metric="latency",
        grid={"batch_size": [1, 10, 100, 1000], "payload": ["small", "large"]},
        func=_append_batch,
    ),
    Scenario(
        name="store.read_stream",
        interface="store",
        metric="latency",
        grid={"stream_length": [100, 1000, 10000]},
        func=_read_stream,
        prepare=_prepare_read_stream,
    ),
    Scenario(
        name="store.concurrent_append",
        interface="store",
        metric="throughput",
        grid={"writers": [1, 10, 50]},
        func=_concurrent_append,
    ),
    Scenario(
        name="store.contended_append",
        interface="store",
        metric="throughput",
        grid={"writers": [1, 10, 50]},
        func=_contended_append,
    ),
]

Update bench/scenarios/__init__.py:


"""Scenario registry. Later tasks append bus, snapshot, and e2e scenarios."""

from bench.core.runner import Scenario
from bench.scenarios.stores import STORE_SCENARIOS


def all_scenarios() -> list[Scenario]:
    return [*STORE_SCENARIOS]

Behavior note for _contended_append: append_events signals a version conflict either by raising OptimisticLockError or by returning AppendResult(conflict=True) depending on backend — the code above handles both. Check eventsource.exceptions for the exact exception name; the conformance suite's test_optimistic_locking shows the authoritative behavior.

Run: uv run pytest tests/unit/bench/test_scenarios_stores.py -v -p no:randomly Expected: 5 passed (4 parametrized + grid check)


uv run ruff check bench/ tests/unit/bench/ --fix && uv run ruff format bench/ tests/unit/bench/
git add bench/scenarios/ tests/unit/bench/test_scenarios_stores.py
git commit -m "feat: add event store benchmark scenarios"

Task 6: EventBus scenarios

Files:

Interfaces:


# tests/unit/bench/test_scenarios_buses.py
"""Run every bus scenario against the memory adapter with tiny budgets."""

import pytest

from bench.adapters.buses import MemoryBusAdapter
from bench.core.runner import RunnerConfig, run_cell
from bench.scenarios.buses import BUS_SCENARIOS

TINY = RunnerConfig(
    rounds=1,
    warmup_iterations=1,
    calibration_iterations=1,
    target_round_seconds=0.02,
    max_iterations=5,
    cell_timeout_seconds=30.0,
)

SMALLEST_PARAMS = {
    "bus.publish_throughput": {"batch_size": 1},
    "bus.fanout": {"subscribers": 1},
    "bus.roundtrip": {},
}


@pytest.mark.parametrize("scenario", BUS_SCENARIOS, ids=lambda s: s.name)
async def test_bus_scenario_runs_on_memory(scenario) -> None:  # type: ignore[no-untyped-def]
    adapter = MemoryBusAdapter()
    await adapter.setup()
    cell = await run_cell(adapter, scenario, SMALLEST_PARAMS[scenario.name], TINY)
    await adapter.teardown()
    assert cell.status == "ok", cell.reason
    assert cell.rounds and cell.rounds[0].operations > 0


def test_grids_match_spec() -> None:
    by_name = {s.name: s for s in BUS_SCENARIOS}
    assert by_name["bus.publish_throughput"].grid == {"batch_size": [1, 10, 100]}
    assert by_name["bus.fanout"].grid == {"subscribers": [1, 10, 50]}
    assert by_name["bus.roundtrip"].grid == {}

Run: uv run pytest tests/unit/bench/test_scenarios_buses.py -v -p no:randomly Expected: FAIL — import error on bench.scenarios.buses


# bench/scenarios/buses.py
"""EventBus scenarios: publish throughput, fan-out delivery, roundtrip latency.

Publish site and handlers capture perf_counter() in the same process, so
deltas are valid monotonic-clock latencies (spec: methodology).
"""

import asyncio
import time
from dataclasses import dataclass, field
from typing import Any
from uuid import uuid4

from eventsource.bus.interface import EventBus

from bench.adapters.base import BenchAdapter, BusAdapter
from bench.core.domain import BenchEvent, make_events
from bench.core.runner import Measurement, Scenario


@dataclass
class _BusHarness:
    adapter: BusAdapter
    publish_times: dict[int, float] = field(default_factory=dict)
    deliveries: list[tuple[int, float]] = field(default_factory=list)
    delivered: asyncio.Event = field(default_factory=asyncio.Event)
    expected: int = 0
    next_seq: int = 0

    def on_delivery(self, event: BenchEvent) -> None:
        self.deliveries.append((event.seq, time.perf_counter()))
        if len(self.deliveries) >= self.expected:
            self.delivered.set()

    def begin_wave(self, expected: int) -> None:
        self.publish_times.clear()
        self.deliveries.clear()
        self.delivered = asyncio.Event()
        self.expected = expected

    def take_seq(self) -> int:
        seq = self.next_seq
        self.next_seq += 1
        return seq


def _sequenced_events(harness: _BusHarness, count: int) -> list[BenchEvent]:
    aggregate_id = uuid4()
    events = make_events(aggregate_id, count)
    stamped = []
    for event in events:
        seq = harness.take_seq()
        stamped.append(event.model_copy(update={"seq": seq}))
    return stamped


async def _prepare_publish(
    adapter: BenchAdapter[Any], bus: EventBus, params: dict[str, Any]
) -> _BusHarness:
    assert isinstance(adapter, BusAdapter)
    return _BusHarness(adapter=adapter)


async def _publish_throughput(
    bus: EventBus, params: dict[str, Any], iterations: int, prepared: Any
) -> Measurement:
    batch_size: int = params["batch_size"]
    harness: _BusHarness = prepared
    durations: list[float] = []
    start = time.perf_counter()
    for _ in range(iterations):
        events = _sequenced_events(harness, batch_size)
        t0 = time.perf_counter()
        await bus.publish(list(events))
        durations.append(time.perf_counter() - t0)
    return Measurement(
        elapsed_s=time.perf_counter() - start,
        operations=iterations * batch_size,
        durations_s=durations,
    )


async def _prepare_fanout(
    adapter: BenchAdapter[Any], bus: EventBus, params: dict[str, Any]
) -> _BusHarness:
    assert isinstance(adapter, BusAdapter)
    harness = _BusHarness(adapter=adapter)
    for _ in range(params["subscribers"]):
        async def handler(event: BenchEvent, _h: _BusHarness = harness) -> None:
            _h.on_delivery(event)

        bus.subscribe(BenchEvent, handler)
    await adapter.start_delivery(bus)
    return harness


async def _fanout(
    bus: EventBus, params: dict[str, Any], iterations: int, prepared: Any
) -> Measurement:
    subscribers: int = params["subscribers"]
    harness: _BusHarness = prepared
    harness.begin_wave(expected=iterations * subscribers)
    start = time.perf_counter()
    for _ in range(iterations):
        events = _sequenced_events(harness, 1)
        harness.publish_times[events[0].seq] = time.perf_counter()
        await bus.publish(list(events))
    await harness.delivered.wait()
    elapsed = time.perf_counter() - start
    latencies = [
        received - harness.publish_times[seq]
        for seq, received in harness.deliveries
        if seq in harness.publish_times
    ]
    return Measurement(
        elapsed_s=elapsed,
        operations=len(harness.deliveries),
        durations_s=latencies,
    )


async def _prepare_roundtrip(
    adapter: BenchAdapter[Any], bus: EventBus, params: dict[str, Any]
) -> _BusHarness:
    return await _prepare_fanout(adapter, bus, {"subscribers": 1})


async def _roundtrip(
    bus: EventBus, params: dict[str, Any], iterations: int, prepared: Any
) -> Measurement:
    harness: _BusHarness = prepared
    durations: list[float] = []
    start = time.perf_counter()
    for _ in range(iterations):
        harness.begin_wave(expected=1)
        events = _sequenced_events(harness, 1)
        t0 = time.perf_counter()
        await bus.publish(list(events))
        await harness.delivered.wait()
        durations.append(harness.deliveries[0][1] - t0)
    return Measurement(
        elapsed_s=time.perf_counter() - start,
        operations=iterations,
        durations_s=durations,
    )


BUS_SCENARIOS: list[Scenario] = [
    Scenario(
        name="bus.publish_throughput",
        interface="bus",
        metric="throughput",
        grid={"batch_size": [1, 10, 100]},
        func=_publish_throughput,
        prepare=_prepare_publish,
    ),
    Scenario(
        name="bus.fanout",
        interface="bus",
        metric="throughput",
        grid={"subscribers": [1, 10, 50]},
        func=_fanout,
        prepare=_prepare_fanout,
    ),
    Scenario(
        name="bus.roundtrip",
        interface="bus",
        metric="latency",
        grid={},
        func=_roundtrip,
        prepare=_prepare_roundtrip,
    ),
]

Append to bench/scenarios/__init__.py:


from bench.scenarios.buses import BUS_SCENARIOS


def all_scenarios() -> list[Scenario]:
    return [*STORE_SCENARIOS, *BUS_SCENARIOS]

Caveat: the memory bus dispatches inline during publish, so delivered may already be set when awaited — that is fine. Handlers must be tolerant of duplicate delivery on broker buses (at-least-once); expected counts are minimums, which >= already handles.

Run: uv run pytest tests/unit/bench/test_scenarios_buses.py -v -p no:randomly Expected: 4 passed


uv run ruff check bench/ tests/unit/bench/ --fix && uv run ruff format bench/ tests/unit/bench/
git add bench/scenarios/ tests/unit/bench/test_scenarios_buses.py
git commit -m "feat: add event bus benchmark scenarios"

Task 7: Snapshot and end-to-end aggregate scenarios

Files:

Interfaces:


# tests/unit/bench/test_scenarios_snapshot_e2e.py
"""Snapshot and end-to-end scenarios against memory backends."""

import pytest

from bench.adapters.e2e import make_e2e_adapters
from bench.adapters.snapshots import MemorySnapshotAdapter
from bench.core.runner import RunnerConfig, run_cell
from bench.scenarios.aggregate import E2E_SCENARIOS
from bench.scenarios.snapshots import SNAPSHOT_SCENARIOS

TINY = RunnerConfig(
    rounds=1,
    warmup_iterations=1,
    calibration_iterations=1,
    target_round_seconds=0.02,
    max_iterations=5,
    cell_timeout_seconds=30.0,
)


@pytest.mark.parametrize("scenario", SNAPSHOT_SCENARIOS, ids=lambda s: s.name)
async def test_snapshot_scenario_runs_on_memory(scenario) -> None:  # type: ignore[no-untyped-def]
    adapter = MemorySnapshotAdapter()
    await adapter.setup()
    cell = await run_cell(adapter, scenario, {"size": "small"}, TINY)
    await adapter.teardown()
    assert cell.status == "ok", cell.reason


@pytest.mark.parametrize("snapshots", ["none", "threshold"])
async def test_e2e_scenario_runs_on_memory(snapshots: str) -> None:
    adapter = next(a for a in make_e2e_adapters() if a.name == "memory")
    await adapter.setup()
    scenario = E2E_SCENARIOS[0]
    params = {"stream_length": 100, "snapshots": snapshots}
    cell = await run_cell(adapter, scenario, params, TINY)
    await adapter.teardown()
    assert cell.status == "ok", cell.reason
    assert cell.rounds and cell.rounds[0].latency is not None


def test_grids_match_spec() -> None:
    by_name = {s.name: s for s in SNAPSHOT_SCENARIOS + E2E_SCENARIOS}
    assert by_name["snapshot.save"].grid == {"size": ["small", "medium", "large"]}
    assert by_name["snapshot.load"].grid == {"size": ["small", "medium", "large"]}
    assert by_name["e2e.load_mutate_save"].grid == {
        "stream_length": [100, 1000, 10000],
        "snapshots": ["none", "threshold"],
    }

Run: uv run pytest tests/unit/bench/test_scenarios_snapshot_e2e.py -v -p no:randomly Expected: FAIL — import errors


# bench/scenarios/snapshots.py
"""SnapshotStore scenarios: save latency and load latency by state size."""

import time
from datetime import UTC, datetime
from typing import Any
from uuid import uuid4

from eventsource.snapshots.interface import Snapshot, SnapshotStore

from bench.adapters.base import BenchAdapter
from bench.core.domain import SNAPSHOT_SIZES, make_snapshot_state
from bench.core.runner import Measurement, Scenario


def _make_snapshot(size: str, version: int = 1) -> Snapshot:
    return Snapshot(
        aggregate_id=uuid4(),
        aggregate_type="Bench",
        version=version,
        state=make_snapshot_state(SNAPSHOT_SIZES[size]),
        schema_version=1,
        created_at=datetime.now(UTC),
    )


async def _save(
    store: SnapshotStore, params: dict[str, Any], iterations: int, prepared: Any
) -> Measurement:
    durations: list[float] = []
    start = time.perf_counter()
    for _ in range(iterations):
        snapshot = _make_snapshot(params["size"])
        t0 = time.perf_counter()
        await store.save_snapshot(snapshot)
        durations.append(time.perf_counter() - t0)
    return Measurement(
        elapsed_s=time.perf_counter() - start,
        operations=iterations,
        durations_s=durations,
    )


async def _prepare_load(
    adapter: BenchAdapter[Any], store: SnapshotStore, params: dict[str, Any]
) -> Snapshot:
    snapshot = _make_snapshot(params["size"])
    await store.save_snapshot(snapshot)
    return snapshot


async def _load(
    store: SnapshotStore, params: dict[str, Any], iterations: int, prepared: Any
) -> Measurement:
    snapshot: Snapshot = prepared
    durations: list[float] = []
    start = time.perf_counter()
    for _ in range(iterations):
        t0 = time.perf_counter()
        loaded = await store.get_snapshot(snapshot.aggregate_id, snapshot.aggregate_type)
        durations.append(time.perf_counter() - t0)
        if loaded is None:
            raise RuntimeError("snapshot vanished during load benchmark")
    return Measurement(
        elapsed_s=time.perf_counter() - start,
        operations=iterations,
        durations_s=durations,
    )


SNAPSHOT_SCENARIOS: list[Scenario] = [
    Scenario(
        name="snapshot.save",
        interface="snapshot",
        metric="latency",
        grid={"size": ["small", "medium", "large"]},
        func=_save,
    ),
    Scenario(
        name="snapshot.load",
        interface="snapshot",
        metric="latency",
        grid={"size": ["small", "medium", "large"]},
        func=_load,
        prepare=_prepare_load,
    ),
]

# bench/adapters/e2e.py
"""Composite adapter pairing a store backend with its matching snapshot backend."""

from eventsource.snapshots.interface import SnapshotStore
from eventsource.stores.interface import EventStore

from bench.adapters.base import BenchAdapter
from bench.adapters.snapshots import SNAPSHOT_ADAPTERS
from bench.adapters.stores import STORE_ADAPTERS


class E2EAdapter(BenchAdapter[tuple[EventStore, SnapshotStore]]):
    def __init__(
        self,
        store_adapter: BenchAdapter[EventStore],
        snapshot_adapter: BenchAdapter[SnapshotStore],
    ) -> None:
        self._store = store_adapter
        self._snapshot = snapshot_adapter
        # instance attribute shadows the ClassVar on purpose: name = backend pair
        self.name = store_adapter.name

    async def available(self) -> str | None:
        return await self._store.available() or await self._snapshot.available()

    async def setup(self) -> None:
        await self._store.setup()
        await self._snapshot.setup()

    async def teardown(self) -> None:
        await self._snapshot.teardown()
        await self._store.teardown()

    async def create(self) -> tuple[EventStore, SnapshotStore]:
        return (await self._store.create(), await self._snapshot.create())

    async def destroy(self, resource: tuple[EventStore, SnapshotStore]) -> None:
        store, snapshot_store = resource
        await self._store.destroy(store)
        await self._snapshot.destroy(snapshot_store)


def make_e2e_adapters() -> list[E2EAdapter]:
    adapters = []
    for backend, store_cls in STORE_ADAPTERS.items():
        snapshot_cls = SNAPSHOT_ADAPTERS.get(backend)
        if snapshot_cls is not None:
            adapters.append(E2EAdapter(store_cls(), snapshot_cls()))
    return adapters

# bench/scenarios/aggregate.py
"""End-to-end AggregateRepository benchmark: load -> mutate -> save."""

import time
from typing import Any
from uuid import UUID, uuid4

from eventsource.aggregates.repository import AggregateRepository
from eventsource.snapshots.interface import SnapshotStore
from eventsource.stores.interface import EventStore

from bench.adapters.base import BenchAdapter
from bench.core.domain import BenchCounter
from bench.core.runner import Measurement, Scenario

SNAPSHOT_THRESHOLD = 100


def _make_repo(
    store: EventStore, snapshot_store: SnapshotStore | None, snapshots: str
) -> AggregateRepository[BenchCounter]:
    if snapshots == "threshold":
        return AggregateRepository(
            event_store=store,
            aggregate_factory=BenchCounter,
            snapshot_store=snapshot_store,
            snapshot_threshold=SNAPSHOT_THRESHOLD,
            snapshot_mode="sync",
            enable_tracing=False,
        )
    return AggregateRepository(
        event_store=store,
        aggregate_factory=BenchCounter,
        enable_tracing=False,
    )


async def _prepare_e2e(
    adapter: BenchAdapter[Any],
    resource: tuple[EventStore, SnapshotStore],
    params: dict[str, Any],
) -> UUID:
    store, snapshot_store = resource
    repo = _make_repo(store, snapshot_store, params["snapshots"])
    aggregate_id = uuid4()
    aggregate = repo.create_new(aggregate_id)
    remaining = params["stream_length"]
    chunk = 100
    while remaining > 0:
        for _ in range(min(chunk, remaining)):
            aggregate.increment()
        await repo.save(aggregate)
        remaining -= chunk
    return aggregate_id


async def _load_mutate_save(
    resource: tuple[EventStore, SnapshotStore],
    params: dict[str, Any],
    iterations: int,
    prepared: Any,
) -> Measurement:
    store, snapshot_store = resource
    aggregate_id: UUID = prepared
    repo = _make_repo(store, snapshot_store, params["snapshots"])
    durations: list[float] = []
    start = time.perf_counter()
    for _ in range(iterations):
        t0 = time.perf_counter()
        aggregate = await repo.load(aggregate_id)
        aggregate.increment()
        await repo.save(aggregate)
        durations.append(time.perf_counter() - t0)
    return Measurement(
        elapsed_s=time.perf_counter() - start,
        operations=iterations,
        durations_s=durations,
    )


E2E_SCENARIOS: list[Scenario] = [
    Scenario(
        name="e2e.load_mutate_save",
        interface="e2e",
        metric="latency",
        grid={"stream_length": [100, 1000, 10000], "snapshots": ["none", "threshold"]},
        func=_load_mutate_save,
        prepare=_prepare_e2e,
    ),
]

Update bench/scenarios/__init__.py to its final form:


"""Scenario registry."""

from bench.core.runner import Scenario
from bench.scenarios.aggregate import E2E_SCENARIOS
from bench.scenarios.buses import BUS_SCENARIOS
from bench.scenarios.snapshots import SNAPSHOT_SCENARIOS
from bench.scenarios.stores import STORE_SCENARIOS


def all_scenarios() -> list[Scenario]:
    return [*STORE_SCENARIOS, *BUS_SCENARIOS, *SNAPSHOT_SCENARIOS, *E2E_SCENARIOS]

Note: AggregateRepository.create_new(aggregate_id) and repo.load(...)/repo.save(...) signatures are confirmed against src/eventsource/aggregates/repository.py:109-677. If mypy complains about the E2EAdapter.name instance/ClassVar shadow, change BenchAdapter.name to a plain class attribute name: str = "" — it is not mutated anywhere else.

Run: uv run pytest tests/unit/bench/test_scenarios_snapshot_e2e.py -v -p no:randomly Expected: 5 passed


uv run ruff check bench/ tests/unit/bench/ --fix && uv run ruff format bench/ tests/unit/bench/
git add bench/scenarios/ bench/adapters/e2e.py tests/unit/bench/test_scenarios_snapshot_e2e.py
git commit -m "feat: add snapshot and end-to-end aggregate benchmark scenarios"

Task 8: PostgreSQL and SQLite adapters (store + snapshot)

Files:

Interfaces:

The environment always has aiosqlite (dev extra), so the SQLite adapters are tested for real. PostgreSQL adapters are tested for availability-probing behavior only — no live service in unit tests.


# tests/unit/bench/test_adapters_sql.py
"""SQLite adapters run for real; PostgreSQL adapters are probed only."""

from uuid import uuid4

from bench.adapters._postgres import asyncpg_dsn, postgres_url
from bench.adapters.snapshots import SNAPSHOT_ADAPTERS, SQLiteSnapshotAdapter
from bench.adapters.stores import (
    STORE_ADAPTERS,
    PostgresStoreAdapter,
    SQLiteStoreAdapter,
)
from bench.core.domain import make_events


def test_registries_contain_sql_backends() -> None:
    assert STORE_ADAPTERS["postgresql"] is PostgresStoreAdapter
    assert STORE_ADAPTERS["sqlite"] is SQLiteStoreAdapter
    assert "postgresql" in SNAPSHOT_ADAPTERS and "sqlite" in SNAPSHOT_ADAPTERS


def test_asyncpg_dsn_strips_driver() -> None:
    assert asyncpg_dsn("postgresql+asyncpg://u:p@h:5/db") == "postgresql://u:p@h:5/db"
    assert postgres_url().startswith("postgresql+asyncpg://")


async def test_sqlite_store_adapter_appends_and_reads() -> None:
    adapter = SQLiteStoreAdapter()
    assert await adapter.available() is None
    await adapter.setup()
    store = await adapter.create()
    aggregate_id = uuid4()
    result = await store.append_events(
        aggregate_id, "Bench", make_events(aggregate_id, 3), expected_version=0
    )
    assert result.success
    stream = await store.get_events(aggregate_id, "Bench")
    assert stream.version == 3
    await adapter.destroy(store)
    await adapter.teardown()


async def test_sqlite_snapshot_adapter_roundtrip() -> None:
    from datetime import UTC, datetime

    from eventsource.snapshots.interface import Snapshot

    adapter = SQLiteSnapshotAdapter()
    await adapter.setup()
    store = await adapter.create()
    aggregate_id = uuid4()
    await store.save_snapshot(
        Snapshot(
            aggregate_id=aggregate_id,
            aggregate_type="Bench",
            version=1,
            state={"blob": "x"},
            schema_version=1,
            created_at=datetime.now(UTC),
        )
    )
    loaded = await store.get_snapshot(aggregate_id, "Bench")
    assert loaded is not None
    await adapter.destroy(store)
    await adapter.teardown()


async def test_postgres_adapter_unavailable_without_service() -> None:
    adapter = PostgresStoreAdapter(url="postgresql+asyncpg://x:x@localhost:1/nope")
    reason = await adapter.available()
    assert reason is not None and "postgres" in reason.lower()

Run: uv run pytest tests/unit/bench/test_adapters_sql.py -v -p no:randomly Expected: FAIL — import errors on the new names


# bench/adapters/_postgres.py
"""Shared PostgreSQL helpers: DSN handling, schema setup, cleanup.

Schema is applied over a raw asyncpg connection because asyncpg's simple
query protocol accepts multi-statement scripts, so get_schema("all") does
not need fragile statement splitting (unlike SQLAlchemy text()).
"""

import os

DEFAULT_URL = "postgresql+asyncpg://bench:bench@localhost:5434/eventsource_bench"


def postgres_url() -> str:
    return os.environ.get("BENCH_POSTGRES_URL", DEFAULT_URL)


def asyncpg_dsn(url: str) -> str:
    return url.replace("postgresql+asyncpg://", "postgresql://")


async def ping(dsn: str) -> str | None:
    try:
        import asyncpg
    except ImportError:
        return "postgresql extra not installed (asyncpg missing)"
    try:
        conn = await asyncpg.connect(dsn, timeout=3)
        await conn.close()
    except Exception as exc:  # noqa: BLE001 - any failure means "not available"
        return f"postgres unreachable at {dsn}: {exc}"
    return None


async def ensure_schema(dsn: str) -> None:
    import asyncpg

    from eventsource.migrations import get_schema

    conn = await asyncpg.connect(dsn, timeout=10)
    try:
        await conn.execute(get_schema("events", backend="postgresql"))
        await conn.execute(get_schema("snapshots", backend="postgresql"))
    finally:
        await conn.close()


async def truncate(dsn: str) -> None:
    import asyncpg

    conn = await asyncpg.connect(dsn, timeout=10)
    try:
        await conn.execute("TRUNCATE TABLE events, snapshots CASCADE")
    finally:
        await conn.close()

Add to bench/adapters/stores.py:


import tempfile
from pathlib import Path
from typing import Any
from uuid import uuid4

from bench.adapters._postgres import asyncpg_dsn, ensure_schema, ping, postgres_url, truncate
from bench.core.domain import make_registry


class PostgresStoreAdapter(BenchAdapter[EventStore]):
    name = "postgresql"

    def __init__(self, url: str | None = None) -> None:
        self._url = url or postgres_url()
        self._engine: Any = None
        self._session_factory: Any = None

    async def available(self) -> str | None:
        return await ping(asyncpg_dsn(self._url))

    async def setup(self) -> None:
        from sqlalchemy.ext.asyncio import (
            AsyncSession,
            async_sessionmaker,
            create_async_engine,
        )

        await ensure_schema(asyncpg_dsn(self._url))
        self._engine = create_async_engine(
            self._url, echo=False, pool_size=10, max_overflow=20
        )
        self._session_factory = async_sessionmaker(
            self._engine, class_=AsyncSession, expire_on_commit=False
        )

    async def teardown(self) -> None:
        if self._engine is not None:
            await self._engine.dispose()

    async def create(self) -> EventStore:
        from eventsource import PostgreSQLEventStore

        await truncate(asyncpg_dsn(self._url))
        return PostgreSQLEventStore(
            self._session_factory,
            event_registry=make_registry(),
            outbox_enabled=False,
            enable_tracing=False,
        )


class SQLiteStoreAdapter(BenchAdapter[EventStore]):
    name = "sqlite"

    def __init__(self) -> None:
        self._tmpdir: tempfile.TemporaryDirectory[str] | None = None

    async def available(self) -> str | None:
        try:
            import aiosqlite  # noqa: F401
        except ImportError:
            return "sqlite extra not installed (aiosqlite missing)"
        return None

    async def setup(self) -> None:
        self._tmpdir = tempfile.TemporaryDirectory(prefix="bench-sqlite-")

    async def teardown(self) -> None:
        if self._tmpdir is not None:
            self._tmpdir.cleanup()

    async def create(self) -> EventStore:
        from eventsource.stores.sqlite import SQLiteEventStore

        assert self._tmpdir is not None
        database = str(Path(self._tmpdir.name) / f"{uuid4().hex}.db")
        store = SQLiteEventStore(
            database, event_registry=make_registry(), wal_mode=True, enable_tracing=False
        )
        await store.initialize()
        return store

    async def destroy(self, resource: EventStore) -> None:
        close = getattr(resource, "close", None)
        if close is not None:
            await close()


STORE_ADAPTERS: dict[str, type[BenchAdapter[EventStore]]] = {
    MemoryStoreAdapter.name: MemoryStoreAdapter,
    PostgresStoreAdapter.name: PostgresStoreAdapter,
    SQLiteStoreAdapter.name: SQLiteStoreAdapter,
}

Add to bench/adapters/snapshots.py (same shape):


import tempfile
from pathlib import Path
from typing import Any
from uuid import uuid4

from bench.adapters._postgres import asyncpg_dsn, ensure_schema, ping, postgres_url, truncate


class PostgresSnapshotAdapter(BenchAdapter[SnapshotStore]):
    name = "postgresql"

    def __init__(self, url: str | None = None) -> None:
        self._url = url or postgres_url()
        self._engine: Any = None
        self._session_factory: Any = None

    async def available(self) -> str | None:
        return await ping(asyncpg_dsn(self._url))

    async def setup(self) -> None:
        from sqlalchemy.ext.asyncio import (
            AsyncSession,
            async_sessionmaker,
            create_async_engine,
        )

        await ensure_schema(asyncpg_dsn(self._url))
        self._engine = create_async_engine(self._url, echo=False, pool_size=10)
        self._session_factory = async_sessionmaker(
            self._engine, class_=AsyncSession, expire_on_commit=False
        )

    async def teardown(self) -> None:
        if self._engine is not None:
            await self._engine.dispose()

    async def create(self) -> SnapshotStore:
        from eventsource.snapshots.postgresql import PostgreSQLSnapshotStore

        await truncate(asyncpg_dsn(self._url))
        return PostgreSQLSnapshotStore(self._session_factory, enable_tracing=False)


class SQLiteSnapshotAdapter(BenchAdapter[SnapshotStore]):
    name = "sqlite"

    def __init__(self) -> None:
        self._tmpdir: tempfile.TemporaryDirectory[str] | None = None

    async def available(self) -> str | None:
        try:
            import aiosqlite  # noqa: F401
        except ImportError:
            return "sqlite extra not installed (aiosqlite missing)"
        return None

    async def setup(self) -> None:
        self._tmpdir = tempfile.TemporaryDirectory(prefix="bench-sqlite-snap-")

    async def teardown(self) -> None:
        if self._tmpdir is not None:
            self._tmpdir.cleanup()

    async def create(self) -> SnapshotStore:
        from eventsource.snapshots.sqlite import SQLiteSnapshotStore

        assert self._tmpdir is not None
        database = str(Path(self._tmpdir.name) / f"{uuid4().hex}.db")
        return SQLiteSnapshotStore(database, enable_tracing=False)


SNAPSHOT_ADAPTERS: dict[str, type[BenchAdapter[SnapshotStore]]] = {
    MemorySnapshotAdapter.name: MemorySnapshotAdapter,
    PostgresSnapshotAdapter.name: PostgresSnapshotAdapter,
    SQLiteSnapshotAdapter.name: SQLiteSnapshotAdapter,
}

Notes: SQLiteSnapshotStore opens connections per-operation and creates its own table lazily — verify by reading src/eventsource/snapshots/sqlite.py; if it needs explicit table creation, mirror what tests/unit/ snapshot tests do. If two adapters (store + snapshot) truncate the same PostgreSQL database in an e2e cell, that is harmless — E2EAdapter.create() calls both, truncation is idempotent, and cells own their data.

Run: uv run pytest tests/unit/bench/test_adapters_sql.py -v -p no:randomly Expected: 5 passed


uv run ruff check bench/ tests/unit/bench/ --fix && uv run ruff format bench/ tests/unit/bench/
git add bench/adapters/ tests/unit/bench/test_adapters_sql.py
git commit -m "feat: add postgresql and sqlite bench adapters"

Task 9: Broker bus adapters (Redis, Kafka, RabbitMQ)

Files:

Interfaces:

Reference files (read before implementing): tests/integration/bus/test_redis.py, test_kafka.py, test_rabbitmq.py — they show the exact working config, connect/consume/shutdown call order, and quirks (single_connection_client=True for Redis, unique topic/exchange suffixes for isolation). Mirror them; only the endpoints change (env vars from Global Constraints instead of testcontainers).

Unit tests cover import-guard behavior and registry membership only — broker correctness is exercised by the live smoke run in task 12.


# tests/unit/bench/test_adapters_brokers.py
"""Broker adapters: registry membership and unreachable-service probing."""

from bench.adapters.buses import (
    BUS_ADAPTERS,
    KafkaBusAdapter,
    RabbitMQBusAdapter,
    RedisBusAdapter,
)


def test_registry_contains_all_buses() -> None:
    assert set(BUS_ADAPTERS) == {"memory", "redis", "kafka", "rabbitmq"}


async def test_redis_unavailable_without_service() -> None:
    adapter = RedisBusAdapter(url="redis://localhost:1")
    reason = await adapter.available()
    assert reason is not None


async def test_kafka_unavailable_without_service() -> None:
    adapter = KafkaBusAdapter(servers="localhost:1")
    reason = await adapter.available()
    assert reason is not None


async def test_rabbitmq_unavailable_without_service() -> None:
    adapter = RabbitMQBusAdapter(url="amqp://guest:guest@localhost:1/")
    reason = await adapter.available()
    assert reason is not None

Run: uv run pytest tests/unit/bench/test_adapters_brokers.py -v -p no:randomly Expected: FAIL — import errors on the adapter names

Add to bench/adapters/buses.py (Redis shown in full; Kafka and RabbitMQ follow the same skeleton with their own config classes and probe):


import asyncio
import os
from uuid import uuid4

from bench.core.domain import make_registry


async def _tcp_probe(host: str, port: int, service: str) -> str | None:
    try:
        reader, writer = await asyncio.wait_for(
            asyncio.open_connection(host, port), timeout=3
        )
        writer.close()
        await writer.wait_closed()
    except (OSError, asyncio.TimeoutError) as exc:
        return f"{service} unreachable at {host}:{port}: {exc}"
    return None


class RedisBusAdapter(BusAdapter):
    name = "redis"

    def __init__(self, url: str | None = None) -> None:
        self._url = url or os.environ.get("BENCH_REDIS_URL", "redis://localhost:6381")
        self._consume_tasks: dict[int, asyncio.Task[None]] = {}

    async def available(self) -> str | None:
        try:
            import redis  # noqa: F401
        except ImportError:
            return "redis extra not installed"
        host, _, port = self._url.removeprefix("redis://").partition(":")
        return await _tcp_probe(host or "localhost", int(port or 6379), "redis")

    async def create(self) -> EventBus:
        from eventsource.bus.redis import RedisEventBus, RedisEventBusConfig

        suffix = uuid4().hex[:8]
        config = RedisEventBusConfig(
            redis_url=self._url,
            stream_prefix=f"bench_{suffix}",
            consumer_group=f"bench_group_{suffix}",
            enable_tracing=False,
        )
        bus = RedisEventBus(config=config, event_registry=make_registry())
        await bus.connect()
        return bus

    async def start_delivery(self, bus: EventBus) -> None:
        # Mirror tests/integration/bus/test_redis.py: consumption runs as a
        # background task (use start_consuming_in_background() if the bus
        # provides it, otherwise create_task(bus.start_consuming())).
        task = asyncio.create_task(bus.start_consuming())  # type: ignore[attr-defined]
        self._consume_tasks[id(bus)] = task
        await asyncio.sleep(0.5)  # let the consumer join the group

    async def stop_delivery(self, bus: EventBus) -> None:
        await bus.stop_consuming()  # type: ignore[attr-defined]
        task = self._consume_tasks.pop(id(bus), None)
        if task is not None:
            task.cancel()

    async def destroy(self, resource: EventBus) -> None:
        await self.stop_delivery(resource)
        await resource.shutdown()

class KafkaBusAdapter(BusAdapter):
    name = "kafka"

    def __init__(self, servers: str | None = None) -> None:
        self._servers = servers or os.environ.get("BENCH_KAFKA_SERVERS", "localhost:9094")

    async def available(self) -> str | None:
        try:
            import aiokafka  # noqa: F401
        except ImportError:
            return "kafka extra not installed (aiokafka missing)"
        host, _, port = self._servers.partition(":")
        return await _tcp_probe(host or "localhost", int(port or 9092), "kafka")

    async def create(self) -> EventBus:
        from eventsource.bus.kafka import KafkaEventBus, KafkaEventBusConfig

        suffix = uuid4().hex[:8]
        config = KafkaEventBusConfig(
            bootstrap_servers=self._servers,
            topic_prefix=f"bench_{suffix}",
            consumer_group=f"bench_group_{suffix}",
            enable_dlq=True,
            enable_tracing=False,
        )
        bus = KafkaEventBus(config=config, event_registry=make_registry())
        await bus.connect()
        return bus

    async def start_delivery(self, bus: EventBus) -> None:
        from eventsource.bus.kafka import KafkaEventBus

        assert isinstance(bus, KafkaEventBus)
        await bus.start_consuming_in_background()
        await asyncio.sleep(0.5)  # let the consumer join the group

    async def stop_delivery(self, bus: EventBus) -> None:
        from eventsource.bus.kafka import KafkaEventBus

        assert isinstance(bus, KafkaEventBus)
        if bus.is_consuming:
            await bus.stop_consuming()

    async def destroy(self, resource: EventBus) -> None:
        from eventsource.bus.kafka import KafkaEventBus

        assert isinstance(resource, KafkaEventBus)
        await self.stop_delivery(resource)
        if resource.is_connected:
            await resource.disconnect()


class RabbitMQBusAdapter(BusAdapter):
    name = "rabbitmq"

    def __init__(self, url: str | None = None) -> None:
        self._url = url or os.environ.get(
            "BENCH_RABBITMQ_URL", "amqp://guest:guest@localhost:5673/"
        )

    async def available(self) -> str | None:
        try:
            import aio_pika  # noqa: F401
        except ImportError:
            return "rabbitmq extra not installed (aio-pika missing)"
        hostport = self._url.split("@")[-1].split("/")[0]
        host, _, port = hostport.partition(":")
        return await _tcp_probe(host or "localhost", int(port or 5672), "rabbitmq")

    async def create(self) -> EventBus:
        from eventsource.bus.rabbitmq import RabbitMQEventBus, RabbitMQEventBusConfig

        suffix = uuid4().hex[:8]
        config = RabbitMQEventBusConfig(
            rabbitmq_url=self._url,
            exchange_name=f"bench_{suffix}",
            consumer_group=f"bench_group_{suffix}",
            durable=False,
            auto_delete=True,
            enable_tracing=False,
        )
        bus = RabbitMQEventBus(config=config, event_registry=make_registry())
        await bus.connect()
        return bus

    async def start_delivery(self, bus: EventBus) -> None:
        from eventsource.bus.rabbitmq import RabbitMQEventBus

        assert isinstance(bus, RabbitMQEventBus)
        await bus.start_consuming_in_background()
        await asyncio.sleep(0.5)

    async def stop_delivery(self, bus: EventBus) -> None:
        from eventsource.bus.rabbitmq import RabbitMQEventBus

        assert isinstance(bus, RabbitMQEventBus)
        if bus.is_consuming:
            await bus.stop_consuming()

    async def destroy(self, resource: EventBus) -> None:
        from eventsource.bus.rabbitmq import RabbitMQEventBus

        assert isinstance(resource, RabbitMQEventBus)
        await self.stop_delivery(resource)
        if resource.is_connected:
            await resource.disconnect()

If is_consuming / is_connected properties differ on a bus class, use the names the corresponding tests/integration/bus/test_*.py fixture teardown uses — those are the authoritative call sequences.

Final registry:


BUS_ADAPTERS: dict[str, type[BusAdapter]] = {
    MemoryBusAdapter.name: MemoryBusAdapter,
    RedisBusAdapter.name: RedisBusAdapter,
    KafkaBusAdapter.name: KafkaBusAdapter,
    RabbitMQBusAdapter.name: RabbitMQBusAdapter,
}

If the type: ignore[attr-defined] comments are needed because EventBus lacks consume methods, prefer narrowing with isinstance checks against the concrete bus class inside the adapter (the adapter already knows its concrete type) — that keeps mypy strict clean without ignores.

Run: uv run pytest tests/unit/bench/test_adapters_brokers.py -v -p no:randomly Expected: 4 passed


uv run ruff check bench/ tests/unit/bench/ --fix && uv run ruff format bench/ tests/unit/bench/
git add bench/adapters/buses.py tests/unit/bench/test_adapters_brokers.py
git commit -m "feat: add redis kafka and rabbitmq bench adapters"

Task 10: Markdown report generator

Files:

Interfaces:


# tests/unit/bench/test_report.py
"""Report generator tests against a hand-built RunResult."""

from bench.core.report import render_markdown
from bench.core.results import (
    SCHEMA_VERSION,
    CellResult,
    LatencyStats,
    Round,
    RunResult,
)


def _round(ops: float = 100.0, conflicts: int = 0) -> Round:
    return Round(
        elapsed_s=1.0,
        operations=100,
        ops_per_sec=ops,
        latency=LatencyStats(p50_ms=2.0, p95_ms=4.0, p99_ms=5.0, mean_ms=2.5, min_ms=1.0),
        counters={"conflicts": conflicts} if conflicts else {},
    )


def _run() -> RunResult:
    return RunResult(
        schema_version=SCHEMA_VERSION,
        metadata={"commit": "abc1234", "platform": "linux-test", "timestamp": "t"},
        cells=[
            CellResult(
                scenario="store.append_batch",
                interface="store",
                backend="memory",
                metric="throughput",
                params={"batch_size": 1},
                status="ok",
                rounds=[_round(500.0)],
            ),
            CellResult(
                scenario="store.append_batch",
                interface="store",
                backend="postgresql",
                metric="throughput",
                params={"batch_size": 1},
                status="skipped",
                reason="postgres unreachable",
            ),
            CellResult(
                scenario="store.contended_append",
                interface="store",
                backend="memory",
                metric="throughput",
                params={"writers": 10},
                status="ok",
                rounds=[_round(200.0, conflicts=25)],
            ),
        ],
    )


def test_report_contains_metadata_and_sections() -> None:
    text = render_markdown(_run())
    assert text.index("abc1234") < text.index("## store")
    assert "### store.append_batch" in text
    assert "| batch_size=1 |" in text


def test_report_renders_throughput_skip_and_conflicts() -> None:
    text = render_markdown(_run())
    assert "500/s" in text
    assert "skipped: postgres unreachable" in text
    assert "[20% conflicts]" in text  # 25 / (100 + 25)


def test_report_latency_metric_uses_percentiles() -> None:
    run = _run()
    run.cells[0].metric = "latency"
    text = render_markdown(run)
    assert "2.00ms" in text and "p95 4.00ms" in text

Run: uv run pytest tests/unit/bench/test_report.py -v -p no:randomly Expected: FAIL — ModuleNotFoundError: No module named 'bench.core.report'


# bench/core/report.py
"""Render a RunResult as a Markdown report.

Rows are parameter combinations, columns are backends, so scaling curves
read down a column and backend comparison reads across a row.
"""

from collections import defaultdict
from typing import Any

from bench.core.results import CellResult, RunResult

_INTERFACE_ORDER = ["store", "bus", "snapshot", "e2e"]


def _params_key(cell: CellResult) -> str:
    if not cell.params:
        return "(default)"
    return ", ".join(f"{k}={cell.params[k]}" for k in sorted(cell.params))


def _backend_order(backends: set[str]) -> list[str]:
    ordered = sorted(backends)
    if "memory" in ordered:
        ordered.remove("memory")
        ordered.insert(0, "memory")
    return ordered


def _format_cell(cell: CellResult) -> str:
    if cell.status != "ok":
        return f"{cell.status}: {cell.reason}"
    median = cell.median_round
    if median is None:
        return "no data"
    if cell.metric == "latency" and median.latency is not None:
        text = f"{median.latency.p50_ms:.2f}ms (p95 {median.latency.p95_ms:.2f}ms)"
    else:
        text = f"{median.ops_per_sec:,.0f}/s"
    conflicts = median.counters.get("conflicts", 0)
    if conflicts:
        rate = conflicts / (median.operations + conflicts)
        text += f" [{rate:.0%} conflicts]"
    return text


def _render_metadata(metadata: dict[str, Any]) -> list[str]:
    lines = ["# Benchmark Report", ""]
    for key in sorted(metadata):
        lines.append(f"- **{key}**: {metadata[key]}")
    lines.append("")
    return lines


def render_markdown(run: RunResult) -> str:
    lines = _render_metadata(run.metadata)
    by_interface: dict[str, dict[str, list[CellResult]]] = defaultdict(
        lambda: defaultdict(list)
    )
    for cell in run.cells:
        by_interface[cell.interface][cell.scenario].append(cell)

    for interface in _INTERFACE_ORDER:
        if interface not in by_interface:
            continue
        lines.append(f"## {interface}")
        lines.append("")
        for scenario_name in sorted(by_interface[interface]):
            cells = by_interface[interface][scenario_name]
            lines.append(f"### {scenario_name}")
            lines.append("")
            backends = _backend_order({c.backend for c in cells})
            lines.append("| params | " + " | ".join(backends) + " |")
            lines.append("|" + "---|" * (len(backends) + 1))
            by_row: dict[str, dict[str, CellResult]] = defaultdict(dict)
            row_order: list[str] = []
            for cell in cells:
                key = _params_key(cell)
                if key not in by_row:
                    row_order.append(key)
                by_row[key][cell.backend] = cell
            for key in row_order:
                row_cells = [
                    _format_cell(by_row[key][b]) if b in by_row[key] else "—"
                    for b in backends
                ]
                lines.append(f"| {key} | " + " | ".join(row_cells) + " |")
            lines.append("")
    return "\n".join(lines)

Run: uv run pytest tests/unit/bench/test_report.py -v -p no:randomly Expected: 3 passed


uv run ruff check bench/ tests/unit/bench/ --fix && uv run ruff format bench/ tests/unit/bench/
git add bench/core/report.py tests/unit/bench/test_report.py
git commit -m "feat: add markdown report generator for bench results"

Task 11: CLI entry point

Files:

Interfaces:


# tests/unit/bench/test_cli.py
"""CLI tests: memory-only quick run end to end, then report over its output."""

from pathlib import Path

from bench.__main__ import main
from bench.core.results import RunResult


def test_run_quick_memory_only(tmp_path: Path) -> None:
    code = main(
        [
            "run",
            "--quick",
            "--backend",
            "memory",
            "--scenario",
            "store.append_batch",
            "--scenario",
            "snapshot.save",
            "--out",
            str(tmp_path),
        ]
    )
    assert code == 0
    files = list(tmp_path.glob("bench-*.json"))
    assert len(files) == 1
    run = RunResult.from_json(files[0].read_text())
    names = {c.scenario for c in run.cells}
    assert names == {"store.append_batch", "snapshot.save"}
    assert all(c.backend == "memory" for c in run.cells)
    assert all(c.status == "ok" for c in run.cells)


def test_report_command(tmp_path: Path) -> None:
    main(
        [
            "run",
            "--quick",
            "--backend",
            "memory",
            "--scenario",
            "store.append_batch",
            "--out",
            str(tmp_path),
        ]
    )
    result_file = next(tmp_path.glob("bench-*.json"))
    out_file = tmp_path / "report.md"
    code = main(["report", str(result_file), "--out", str(out_file)])
    assert code == 0
    text = out_file.read_text()
    assert "# Benchmark Report" in text
    assert "store.append_batch" in text


def test_unknown_filter_values_error() -> None:
    assert main(["run", "--backend", "nope"]) == 2
    assert main(["run", "--scenario", "nope.nope"]) == 2

Run: uv run pytest tests/unit/bench/test_cli.py -v -p no:randomly Expected: FAIL — no main in bench.__main__


# bench/__main__.py
"""CLI: `python -m bench run` and `python -m bench report`."""

import argparse
import asyncio
import sys
from datetime import UTC, datetime
from pathlib import Path
from typing import Any

from bench.adapters.base import BenchAdapter
from bench.adapters.buses import BUS_ADAPTERS
from bench.adapters.e2e import make_e2e_adapters
from bench.adapters.snapshots import SNAPSHOT_ADAPTERS
from bench.adapters.stores import STORE_ADAPTERS
from bench.core.report import render_markdown
from bench.core.results import RunResult
from bench.core.runner import RunnerConfig, run_matrix
from bench.scenarios import all_scenarios

DEFAULT_OUT = Path(__file__).parent / "results"


def _build_adapters(backends: list[str] | None) -> dict[str, list[BenchAdapter[Any]]]:
    def wanted(name: str) -> bool:
        return backends is None or name in backends

    return {
        "store": [cls() for name, cls in STORE_ADAPTERS.items() if wanted(name)],
        "bus": [cls() for name, cls in BUS_ADAPTERS.items() if wanted(name)],
        "snapshot": [cls() for name, cls in SNAPSHOT_ADAPTERS.items() if wanted(name)],
        "e2e": [a for a in make_e2e_adapters() if wanted(a.name)],
    }


def _cmd_run(args: argparse.Namespace) -> int:
    scenarios = all_scenarios()
    known_backends = (
        set(STORE_ADAPTERS) | set(BUS_ADAPTERS) | set(SNAPSHOT_ADAPTERS)
    )
    if args.backend:
        unknown = set(args.backend) - known_backends
        if unknown:
            print(f"unknown backend(s): {', '.join(sorted(unknown))}", file=sys.stderr)
            return 2
    if args.scenario:
        known = {s.name for s in scenarios}
        unknown = set(args.scenario) - known
        if unknown:
            print(f"unknown scenario(s): {', '.join(sorted(unknown))}", file=sys.stderr)
            return 2
        scenarios = [s for s in scenarios if s.name in set(args.scenario)]
    if args.interface:
        scenarios = [s for s in scenarios if s.interface in set(args.interface)]

    adapters = _build_adapters(args.backend or None)
    config = RunnerConfig(quick=args.quick)
    result = asyncio.run(run_matrix(scenarios, adapters, config))

    for cell in result.cells:
        print(f"{cell.status:>7}  {cell.cell_id}" + (f"  ({cell.reason})" if cell.reason else ""))

    out_dir = Path(args.out)
    out_dir.mkdir(parents=True, exist_ok=True)
    stamp = datetime.now(UTC).strftime("%Y%m%d-%H%M%S")
    out_path = out_dir / f"bench-{stamp}.json"
    result.save(out_path)
    print(f"\nresults written to {out_path}")

    return 1 if any(c.status == "failed" for c in result.cells) else 0


def _cmd_report(args: argparse.Namespace) -> int:
    for path in args.files:
        run = RunResult.from_json(Path(path).read_text())
        text = render_markdown(run)
        if args.out:
            Path(args.out).write_text(text)
            print(f"report written to {args.out}")
        else:
            print(text)
    return 0


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(prog="bench", description="eventsource backend benchmarks")
    sub = parser.add_subparsers(dest="command", required=True)

    run_parser = sub.add_parser("run", help="run the benchmark matrix")
    run_parser.add_argument("--interface", action="append", choices=["store", "bus", "snapshot", "e2e"])
    run_parser.add_argument("--backend", action="append")
    run_parser.add_argument("--scenario", action="append")
    run_parser.add_argument("--quick", action="store_true")
    run_parser.add_argument("--out", default=str(DEFAULT_OUT))

    report_parser = sub.add_parser("report", help="render results as markdown")
    report_parser.add_argument("files", nargs="+")
    report_parser.add_argument("--out", default=None)

    args = parser.parse_args(argv)
    if args.command == "run":
        return _cmd_run(args)
    return _cmd_report(args)


if __name__ == "__main__":
    raise SystemExit(main())

Run: uv run pytest tests/unit/bench/test_cli.py -v -p no:randomly Expected: 3 passed


uv run ruff check bench/ tests/unit/bench/ --fix && uv run ruff format bench/ tests/unit/bench/
git add bench/__main__.py tests/unit/bench/test_cli.py
git commit -m "feat: add bench cli with run and report subcommands"

Task 12: Infrastructure — compose file, Makefile targets, mypy wiring, README

Files:

Interfaces:


# Docker Compose services for the benchmark harness (bench/).
#
# Coexists with docker-compose.test.yml -- all host ports are offset.
# No volumes: benchmark data is disposable by design.
#
#   make bench-up      # start all four services
#   make bench         # run the full matrix
#   make bench-down    # stop and remove

services:
  postgres:
    image: postgres:15
    container_name: eventsource-bench-postgres
    environment:
      POSTGRES_DB: eventsource_bench
      POSTGRES_USER: bench
      POSTGRES_PASSWORD: bench
    ports:
      - "${BENCH_POSTGRES_PORT:-5434}:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U bench -d eventsource_bench"]
      interval: 5s
      timeout: 5s
      retries: 5
      start_period: 10s

  redis:
    image: redis:7
    container_name: eventsource-bench-redis
    ports:
      - "${BENCH_REDIS_PORT:-6381}:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 5s
      retries: 5

  rabbitmq:
    image: rabbitmq:3-management
    container_name: eventsource-bench-rabbitmq
    environment:
      RABBITMQ_DEFAULT_USER: guest
      RABBITMQ_DEFAULT_PASS: guest
    ports:
      - "${BENCH_RABBITMQ_PORT:-5673}:5672"
    healthcheck:
      test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"]
      interval: 10s
      timeout: 10s
      retries: 5
      start_period: 20s

  kafka:
    image: apache/kafka:3.7.0
    container_name: eventsource-bench-kafka
    ports:
      - "${BENCH_KAFKA_PORT:-9094}:9094"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093,EXTERNAL://0.0.0.0:9094
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://localhost:${BENCH_KAFKA_PORT:-9094}
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
    healthcheck:
      test: ["CMD-SHELL", "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 || exit 1"]
      interval: 10s
      timeout: 10s
      retries: 10
      start_period: 30s

networks:
  default:
    name: eventsource-bench-network

First run grep -n "types:\|COMPOSE :=\|.PHONY" Makefile to see the exact current text. Then:

  1. Add near the existing COMPOSE := line:

BENCH_COMPOSE := docker compose -f docker-compose.bench.yml
  1. Extend the existing types target so it also checks bench/ (keep the existing src invocation exactly as-is and append bench/ to its path list, e.g. uv run mypy src/ bench/ --config-file=pyproject.toml).
  2. Add the new targets and register them in .PHONY:

bench-up:  ## Start Postgres + Redis + Kafka + RabbitMQ benchmark services
	$(BENCH_COMPOSE) up -d --wait

bench-down:  ## Stop and remove the benchmark services
	$(BENCH_COMPOSE) down -v

bench:  ## Run the full benchmark matrix (services optional; unavailable backends skip)
	uv run python -m bench run

bench-quick:  ## Fast sanity pass over the matrix
	uv run python -m bench run --quick

bench-report:  ## Render a results file: make bench-report RESULTS=bench/results/bench-<ts>.json
	uv run python -m bench report $(RESULTS)
  1. In pyproject.toml, change known-first-party = ["eventsource"] to known-first-party = ["eventsource", "bench"].

# bench — cross-backend benchmark harness

Internal tooling (not part of the `eventsource` package). Runs the same
scenario catalog across every backend and reports comparable numbers.
Design: `docs/superpowers/specs/2026-07-29-backend-benchmark-matrix-design.md`.

## Usage

    make bench-up       # start postgres/redis/kafka/rabbitmq (docker-compose.bench.yml)
    make bench          # full matrix (~20 min); unavailable backends are skipped
    make bench-quick    # trimmed grids, ~2 min
    make bench-report RESULTS=bench/results/bench-<ts>.json
    make bench-down

Without Docker, `make bench` still runs the memory and SQLite backends.

Filters: `uv run python -m bench run --interface store --backend postgresql
--scenario store.append_batch --quick`

## Endpoints (env-overridable)

| Variable | Default |
|---|---|
| `BENCH_POSTGRES_URL` | `postgresql+asyncpg://bench:bench@localhost:5434/eventsource_bench` |
| `BENCH_REDIS_URL` | `redis://localhost:6381` |
| `BENCH_KAFKA_SERVERS` | `localhost:9094` |
| `BENCH_RABBITMQ_URL` | `amqp://guest:guest@localhost:5673/` |

## Interpreting results

Raw JSON lands in `bench/results/` (gitignored, schema-versioned). The
Markdown report renders one table per scenario: rows are parameter
combinations, columns are backends. Numbers are only comparable within a
single run on one machine -- the metadata header records the context.
Numbers reach docs by manual curation, never automatically.

Run: uv run mypy src/ bench/ --config-file=pyproject.toml Expected: no errors (fix any strict-mode findings in bench/ now)

Run: uv run ruff check bench/ tests/unit/bench/ && uv run ruff format --check bench/ tests/unit/bench/ Expected: clean

Run: uv run python -m bench run --quick --backend memory Expected: exit 0, all-ok cell lines for memory scenarios, a results file under bench/results/

Run: docker compose -f docker-compose.bench.yml config --quiet Expected: exit 0 (compose file is syntactically valid — services are NOT started in this task)


git add docker-compose.bench.yml bench/README.md Makefile pyproject.toml
git commit -m "feat: add bench compose services makefile targets and readme"

Task 13: Full-suite verification and live smoke (ORCHESTRATOR ONLY)

Not dispatched to a worker — the orchestrator runs this after all tasks land, per the Global Constraints test-scope rule.

Run: make check (lint, mypy, import-linter, bandit, unit suite) Expected: green. Dispatch any failures back to a fix-up worker with the failing output.


make bench-up
uv run python -m bench run --quick
make bench-down

Expected: exit 0; postgres/sqlite/redis/kafka/rabbitmq cells all ok (not skipped); a results file written. Broker adapters were only probe-tested in unit tests — this is where their consume paths are actually proven. If a broker cell fails, dispatch the failure output plus the relevant tests/integration/bus/test_*.py reference to a fix-up worker scoped to task 9's files.


uv run python -m bench report bench/results/bench-*.json --out /tmp/bench-report.md

Expected: tables render, metadata header present, no failed: cells.


Execution Notes