Sequential Stream Payload Delta Compression
Reference documentation for dictionary-based delta compression of sequential event stream payloads below the snapshot boundary.
Governed by:
- ADR-0007 (Domain-Driven Design and Bounded Contexts)
- ADR-0003 (Blackbox Frontdoor Verification)
- TASK-0002 (Delta-Compress Stored Event Payloads)
1. Overview & Problem Statement
Event-sourced aggregates that record whole document revisions (such as file contents, rendered artifacts, or state carried whole rather than as explicit domain diffs) cause storage growth proportional to $\mathcal{O}(\text{size} \times \text{revisions})$.
DeltaCodec provides opt-in, dictionary-based delta compression using RFC 1950 preset dictionaries with multi-tier integrity checks:
- Per-Frame CRC32: Detects transmission or disk bit-flips prior to decompressing.
- SHA-256 Content Hash: Verifies the reconstructed payload against the original uncompressed state.
The CRC32 and SHA-256 checks detect accidental corruption only. They do not detect tampering. An attacker who can write to the store can change a frame and recompute both checks. Use access controls on the store to prevent tampering.
- Adaptive Re-Baselining: Automatically forces self-contained baseline frames when:
- The delta chain length exceeds
max_chain_length(default: 32). - The compressed delta size exceeds
ratio_thresholdof the uncompressed payload (default: 80%). - The cumulative size of the delta chain exceeds
cumulative_ratio_thresholdtimes the fulltext size (default: 2.0x). - The uncompressed payload is below
min_payload_size(default: 128 bytes).
2. Architecture & Data Model
DeltaPayload wraps the compressed byte frame and integrity metadata into a JSON-compatible structure that can be stored transparently inside existing JSONB, TEXT, or JSON columns:
{
"__delta__": true,
"v": 1,
"is_delta": true,
"base_version": 1,
"data": "<base64-encoded compressed frame>",
"hash": "<sha256-content-hash>",
"crc": 4109098394,
"raw_size": 15437,
"comp_size": 100
}
When an event payload is read, the storage adapter or application deserializer detects __delta__ via is_delta_dict() and reconstructs the dictionary frame. If regular uncompressed dictionaries or primitives are stored, they are left untouched.
3. Usage Example
from eventsource.adapters.serialization import DeltaCodec, DeltaPayload
codec = DeltaCodec(max_chain_length=32, min_payload_size=128)
# 1. First event revision (stored as a standalone baseline)
v1_doc = {"document_id": "doc-001", "version": 1, "content": "Initial draft of long text..." * 50}
p1 = codec.compress(v1_doc)
assert p1.is_delta is False
# 2. Second event revision (stored as a delta against v1)
v2_doc = {"document_id": "doc-001", "version": 2, "content": "Initial draft of long text (with an edit)..." * 50}
p2 = codec.compress(v2_doc, base_payload=v1_doc, base_version=1)
assert p2.is_delta is True
# 3. Transparent decompression
restored_v1 = codec.decompress_to_dict(p1)
restored_v2 = codec.decompress_to_dict(p2, base_payload=restored_v1)
assert restored_v2 == v2_doc
Decompression Limits
DeltaCodec(max_decompressed_bytes=...) caps the output of decompress. The default is 64 MiB (64 1024 1024 bytes). Lower the limit when you read frames from a store that you do not fully trust.
decompress raises DeltaIntegrityError in these cases:
raw_sizeis negative or exceedsmax_decompressed_bytes. The codec checks this before it decompresses anything.- The decompressed output is longer than
raw_size. Decompression stops at the bound. - The decompressed output length differs from
raw_size. - The compressed stream is truncated, or it has data after the end of the stream.
Checksums
A frame stores two checksums. They are checked in this order:
- CRC32 of the compressed bytes.
decompresscomputes the CRC32 ofcompressed_bytesand compares it withpayload_checksum. This check runs before any inflate. It covers the stored frame data (a delta or a standalone compressed payload). - SHA-256 of the decompressed bytes. After inflate,
decompresscomputes the SHA-256 of the raw payload and compares it withcontent_hash. It covers the original payload. A wrongbase_payloadon a delta frame fails this check, or fails earlier withDeltaIntegrityErrorwhen zlib rejects the dictionary.
Both checks detect accidental corruption, such as bit flips or truncated writes. They do not authenticate a frame. The checksums are not keyed, and anyone who can write a frame can write matching checksums. The checks do not cover:
- the
is_deltaflag,base_version, andraw_sizefields, except as the bounds described above; - the identity of the base payload. The caller passes
base_payload, and the codec does not check its version.
Do not use these checksums as a signature. To detect tampering, sign or MAC the frames at the storage layer.
4. Benchmark Results
Measured across 30 sequential document revisions with small text edits:
- Uncompressed Total: ~600 KB
- Compressed Total: ~15 KB
- Compression Ratio: >35x reduction compared to verbatim payloads
- Reconstruction Overhead: <2ms worst-case reconstruction time per frame