Backend Benchmark Matrix β€” Design

Date: 2026-07-29 Status: Approved Purpose: Comparative characterization of eventsource-py backends β€” produce scaling curves and comparable numbers per backend to inform docs/how-to/choose-an-event-store-backend.md and general understanding of how each backend reacts and scales under the same workloads.

Goals

Non-Goals

ADR Impact

Reviewed against docs/adrs/. No ADR is amended or superseded; the harness is internal tooling that observes existing decisions rather than changing them.

ADR Impact
0001 async-first design Stands. The harness is a native-async runner; it exists partly because pytest-benchmark's sync-callable model conflicts with this ADR's async-first reality.
0007 event bus delivery semantics Stands. Fairness rules follow its explicit guidance to disable tracing (NullTracer) in benchmarks.
0010 uniform event bus contract Stands. Bus scenarios exercise only the uniform contract, which is what makes cross-bus comparison valid.
0011 handler error isolation with no-ack Stands. Fan-out scenarios use non-failing handlers; error-path behavior is out of scope.
0015 optional dependency extras Stands. bench/ is repo tooling outside the package; it adds no extras and imports optional backends only behind availability checks.
0016 optional tracing no-op by default Stands. Benchmarks run with tracing off.
0017 snapshot strategy pattern Stands. The end-to-end scenario consumes the strategy pattern as designed (none vs threshold).
0008, 0009 (both), 0012, 0013, 0014, 0018 Stands / not touched. Mutation testing, subscription coordination, advisory locks, event-type derivation, handler registry, migration cutover, and tenant isolation are outside the harness's scope.

No new ADR is needed: the harness is internal, makes no public contract, and its one notable choice (standalone runner over pytest-benchmark) is recorded in this spec's Decision Summary.

Decision Summary

Decision Choice
Purpose Comparative characterization (not regression, not load testing)
Interfaces EventStore Γ—3, EventBus Γ—4, SnapshotStore Γ—3, end-to-end aggregate path
Dimensions Batch/payload size, stream length (Β± snapshots), concurrency/contention, subscriber fan-out
Execution Local CLI + make targets against docker-compose services
Output Raw JSON per run + generated Markdown report
Placement Internal repo tooling (top-level bench/), not part of the package
Engine Standalone async harness (not pytest-benchmark: its sync-callable model wraps each timed call in asyncio.run, distorting measurements, and it cannot express throughput/concurrency scenarios)

Architecture


bench/
  __main__.py          # CLI: uv run python -m bench [run|report]
                       #   run: [--interface store|bus|snapshot|e2e]
                       #        [--backend NAME] [--scenario NAME] [--quick]
                       #   report: results/<run>.json [...]
  core/
    runner.py          # matrix expansion, warmup, timed measurement, stats
    results.py         # result dataclasses + versioned JSON schema
    report.py          # Markdown report generator
  adapters/
    stores.py          # memory / postgresql / sqlite
    buses.py           # memory / redis / kafka / rabbitmq
    snapshots.py       # memory / postgresql / sqlite
  scenarios/
    stores.py, buses.py, snapshots.py, aggregate.py
  results/             # gitignored raw JSON output

Infrastructure

Fairness rules (enforced by adapters)

Scenario Catalog

Scenario Γ— parameter grid Γ— backend = matrix (~106 cells at full grid: 51 store + 28 bus + 9 snapshot + 18 end-to-end).

EventStore (memory, PostgreSQL, SQLite)

Scenario Grid Metrics
append_batch batch [1, 10, 100, 1000] Γ— payload [small ~200B, large ~5KB] events/s, per-call latency
read_stream stream length [100, 1k, 10k] read latency, events/s
concurrent_append writers [1, 10, 50], distinct aggregates throughput scaling
contended_append writers [1, 10, 50], one hot aggregate effective throughput, OptimisticLockError conflict rate

EventBus (memory, Redis, Kafka, RabbitMQ)

Scenario Grid Metrics
publish_throughput batch [1, 10, 100] publish events/s
fanout subscribers [1, 10, 50] delivery throughput, publish→handler latency
roundtrip single event p50/p95/p99 latency

SnapshotStore (memory, PostgreSQL, SQLite)

Scenario Grid Metrics
save_load state size [small, ~50KB, ~500KB] save latency, load latency

End-to-end aggregate path (Γ— 3 stores)

Scenario Grid Metrics
load_mutate_save stream length [100, 1k, 10k] Γ— snapshot strategy [none, threshold] full AggregateRepository round-trip latency

This is the "when does backend X need snapshotting" answer and the number users actually feel.

--quick trims each grid to its smallest value and cuts iterations (full run target < ~20 min; quick < ~2 min).

Measurement Methodology

Per cell: setup β†’ warmup (untimed, fixed iterations) β†’ timed phase β†’ teardown.

Error Handling

Reporting

uv run python -m bench report results/.json renders Markdown: one section per interface; each scenario is a table with the swept parameter as rows, backends as columns, and the measured metric in cells (plus conflict rate columns where relevant). The metadata header leads the document.

The JSON carries a schema_version field and stable scenario/cell IDs so future regression tracking (BACKLOG P3 entry) can diff two run files without a redesign.

Testing

In tests/unit/bench/ so make check covers it: