Observability: Wiring OpenTelemetry Tracing

Every eventsource component that does I/O — buses, snapshot stores, repositories, projections, subscription runners, migration coordinators, and distributed locks — accepts an enable_tracing flag and emits OpenTelemetry spans when tracing is turned on. (The event store adapters -- InMemoryEventStore, SQLiteEventStore, PostgreSQLEventStore -- do not currently accept enable_tracing; use a traced bus or repository for the examples below.) OpenTelemetry itself is an optional dependency: without it installed, OTEL_AVAILABLE is False, every component falls back to a NullTracer, and the tracing code paths cost nothing.

This guide shows you how to:

Tracing is off-by-default and fails soft by design. A missing OpenTelemetry install, a provider configured too late, or enable_tracing=False all produce the same outcome — no spans — and never an error. That makes "no spans appeared" the normal symptom of every misconfiguration, so the troubleshooting section at the end walks through distinguishing the causes.

Everything below imports from the top-level eventsource.observability package; eventsource.observability.tracing and .tracer are the implementing modules but you should not need to import from them directly.

Before you start

You need:

Two defaults are worth internalizing before you debug anything:

Everything is imported from the top-level package:


from eventsource.observability import (
    OTEL_AVAILABLE,
    MockTracer,
    NullTracer,
    OpenTelemetryTracer,
    SpanKindEnum,
    Tracer,
    create_tracer,
    get_tracer,
    should_trace,
    traced,
)

The ATTR_* constants (ATTR_AGGREGATE_ID, ATTR_EVENT_TYPE, ATTR_SUBSCRIPTION_NAME, …) are re-exported from the same place, so from eventsource.observability import ATTR_AGGREGATE_ID works even though they are defined in eventsource.observability.attributes.

To follow the testing section you also need the repo's test suite checked out: the in-memory exporter fixtures live in tests/integration/observability/conftest.py and are not shipped as part of the installed package. MockTracer and NullTracer, by contrast, are public API and available to your own tests without any OpenTelemetry install at all.

Install the telemetry extra (`eventsource-py[telemetry]` or `[all]`)

The telemetry extra adds exactly two packages:


telemetry = [
    "opentelemetry-api>=1.16.0,<2.0",
    "opentelemetry-sdk>=1.16.0,<2.0",
]

Install it alongside whatever backends you already use:


# just tracing
pip install 'eventsource-py[telemetry]'

# tracing plus a couple of backends
pip install 'eventsource-py[postgresql,redis,telemetry]'

# everything: postgresql, sqlite, redis, rabbitmq, kafka, telemetry
pip install 'eventsource-py[all]'

With uv:


uv add 'eventsource-py[telemetry]'

Working inside a checkout of this repo, uv sync --all-extras installs the telemetry extra along with every other optional dependency, which is what the integration tests under tests/integration/observability/ assume.

Why the API *and* the SDK

opentelemetry-api is the part eventsource itself imports — from opentelemetry import trace at the top of observability/tracing.py is the only import that decides OTEL_AVAILABLE. On its own, the API package is a set of no-op stubs: trace.get_tracer(...) hands back a proxy tracer that creates non-recording spans, so OTEL_AVAILABLE would be True, create_tracer() would return a real OpenTelemetryTracer, and you would still see no output anywhere.

opentelemetry-sdk supplies the machinery that turns those calls into recorded spans: TracerProvider, span processors, and the console/in-memory exporters. Both are in the extra precisely because "API only" is the most confusing possible state to end up in — the counters move but nothing is emitted.

The extra does not include an exporter for your backend

The SDK ships ConsoleSpanExporter and InMemorySpanExporter, which is enough for the local-development and testing sections of this guide. Sending spans to a collector, Jaeger, Tempo, Honeycomb, or any other backend needs a separate exporter distribution that eventsource deliberately does not pin for you:


pip install opentelemetry-exporter-otlp

Install that only when you get to OTLP exporter for a collector or backend; if import opentelemetry.exporter.otlp raises ModuleNotFoundError, this is the package you are missing, and it is unrelated to OTEL_AVAILABLE.

Verify the install


from eventsource.observability import OTEL_AVAILABLE, create_tracer

print(OTEL_AVAILABLE)                                # True
print(type(create_tracer(__name__, True)).__name__)  # OpenTelemetryTracer
print(type(create_tracer(__name__, False)).__name__) # NullTracer

Two True/OpenTelemetryTracer lines mean the gate in observability/tracing.py is open. They do not mean spans are being recorded — that needs a real TracerProvider, which is Step 1. If the first line prints False, the extra did not land in the interpreter you are running; see OTEL_AVAILABLE is False despite install.

Nothing breaks if you skip the extra entirely. OTEL_AVAILABLE stays False, every create_tracer() call returns a NullTracer regardless of enable_tracing, and @traced-decorated methods run their bodies with no span overhead. Tracing is additive; leaving it uninstalled is a supported configuration, not a degraded one.

Step 1: Configure a real TracerProvider in your application

eventsource never configures OpenTelemetry for you. OpenTelemetryTracer.__init__ does exactly one thing:


from opentelemetry import trace

self._tracer = trace.get_tracer(tracer_name)

trace.get_tracer() asks the global tracer provider for a tracer. Until your application installs an SDK TracerProvider, that global is a ProxyTracerProvider whose spans are non-recording — they are created, they cost almost nothing, and they go nowhere. Installing a provider is the step that turns those calls into exported spans.

Do it once, at process startup, before you construct stores, buses, or repositories:


from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider

provider = TracerProvider(
    resource=Resource.create({"service.name": "orders-api"}),
)
trace.set_tracer_provider(provider)

A provider with no span processor still records nothing useful, so the exporter choice below is part of the same step. Two practical constraints:

Console exporter for local development

The SDK's ConsoleSpanExporter needs no collector and no extra install — it is the fastest way to confirm that spans are real:


from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor

provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)

SimpleSpanProcessor exports each span synchronously as it ends, so output appears in the order operations complete — good for a terminal, bad for throughput. ConsoleSpanExporter writes to sys.stdout unless you pass it a different out= stream. Use BatchSpanProcessor for anything but local poking.

With that in place, any traced component starts emitting:


import asyncio
from uuid import uuid4

from eventsource.adapters.memory import InMemorySnapshotStore

async def main() -> None:
    store = InMemorySnapshotStore(enable_tracing=True)
    await store.get_snapshot(uuid4(), "Order")

asyncio.run(main())
provider.shutdown()

That prints one JSON span named eventsource.snapshot.get, carrying the aggregate.id and aggregate.type attributes the store sets plus the trace and span IDs. Every traced component follows the same idiom — buses, repositories, projections, and the migration machinery each emit their own eventsource.* spans the same way.

enable_tracing=True is written out above for clarity, but it is already the default; the line that changed the behavior is set_tracer_provider(). If nothing prints, the gate is still closed somewhere — work through Troubleshooting.

OTLP exporter for a collector or backend

For anything shared — a collector, Jaeger, Tempo, Honeycomb — install the exporter distribution, which is not part of the telemetry extra:


pip install opentelemetry-exporter-otlp

Then swap the console exporter for OTLP and the simple processor for a batching one:


from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider(
    resource=Resource.create(
        {
            "service.name": "orders-api",
            "deployment.environment": "production",
        }
    ),
)
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317"))
)
trace.set_tracer_provider(provider)

Notes that matter in production:

None of this touches eventsource. Once a recording provider is installed, every component you construct with enable_tracing=True — the default — emits into it, which is what the next steps unpack.

Step 2: Understand the no-op default

Nothing you did in Step 1 touched eventsource. That is deliberate: the library decides per component instance, at construction time, whether it holds a real tracer or a no-op one, and it makes that decision from two inputs — whether OpenTelemetry could be imported, and whether the caller passed enable_tracing=True. Both must be true.

The decision is made once, in the constructor, and cached on the instance. There is no runtime re-check, no global "tracing on" switch you can flip later, and no error path — a component built before the SDK was configured simply carries a NullTracer for the rest of its life. Knowing this makes every "no spans" symptom diagnosable, so it is worth reading the three pieces in order.

`OTEL_AVAILABLE` — the import-time gate in `observability/tracing.py`

OTEL_AVAILABLE is a module-level boolean set exactly once, when eventsource.observability.tracing is first imported:


try:
    from opentelemetry import trace

    OTEL_AVAILABLE = True
except ImportError:
    OTEL_AVAILABLE = False
    trace = None  # type: ignore[assignment]

This is the single source of truth for OpenTelemetry availability across the whole library. Nothing else re-runs the import: observability/tracer.py does from eventsource.observability.tracing import OTEL_AVAILABLE, and the backends that need their own OpenTelemetry symbols — bus/kafka.py, bus/rabbitmq.py — import the flag from eventsource.observability and derive their local PROPAGATION_AVAILABLE from it rather than defining a second gate. There are unit tests (tests/unit/bus/test_eventbus_tracing_patterns.py, tests/unit/bus/test_rabbitmq_tracing.py) that assert no module ever writes its own OTEL_AVAILABLE = True/False.

Three consequences follow:

```python from unittest.mock import patch

from eventsource.observability import tracer as tracer_module

with patch.object(tracer_module, "OTEL_AVAILABLE", False): ... # create_tracer() now returns NullTracer ```

`should_trace(enable_tracing)` — combining component config with availability

should_trace is a one-line helper that ANDs the per-component flag with the import-time gate:


def should_trace(enable_tracing: bool) -> bool:
    return enable_tracing and OTEL_AVAILABLE

That is the whole predicate, and it is the rule the rest of the library follows: a component traces only when its own configuration asks for it and OpenTelemetry is importable. Neither input can override the other.

Use it in your own code when you want to skip work that only exists to feed a span — building an attribute dict, serializing an ID, counting a collection:


from eventsource.observability import should_trace

if should_trace(self._enable_tracing):
    attributes = {ATTR_EVENT_COUNT: len(events)}
else:
    attributes = None

For that specific guard-before-you-compute pattern the Tracer.enabled property is usually the better choice — tracer.enabled is False on NullTracer and True on OpenTelemetryTracer, so it answers the same question without you having to carry the boolean around. should_trace remains useful where you have a raw enable_tracing argument in hand and no tracer object yet, such as inside a factory or a constructor before the tracer is built.

`create_tracer(name, enable_tracing)` — returns `OpenTelemetryTracer` or `NullTracer`

create_tracer is the factory every component actually calls. It applies the same predicate and returns a concrete tracer:


def create_tracer(name: str, enable_tracing: bool = True) -> Tracer:
    if enable_tracing and OTEL_AVAILABLE:
        return OpenTelemetryTracer(name)
    return NullTracer()

It always returns a Tracer — never None. That is the point of the no-op default: call sites never branch on whether tracing exists, they just call self._tracer.span(...) and let the implementation decide whether anything happens. NullTracer.span() and .span_with_kind() are @contextlib.contextmanager functions that yield None, and NullTracer.start_span() returns None, so the disabled path costs one generator and nothing else.

The name argument becomes the OpenTelemetry instrumentation scope name via trace.get_tracer(tracer_name). Components pass their own module's __name__, so spans from the in-memory snapshot store are scoped to eventsource.adapters.memory.snapshots, spans from the in-memory event bus to eventsource.adapters.memory.bus, and so on. This is the scope your backend shows as the instrumentation library — it is not the span name.

Every instrumented component follows the identical two-line idiom in its constructor. From adapters/memory/snapshots.py:


self._tracer = tracer or create_tracer(__name__, enable_tracing)
self._enable_tracing = self._tracer.enabled

Read that second line carefully, because it explains most of the library's behavior:

Two tracers exist beyond these. MockTracer records (name, attributes) tuples into a spans list and reports enabled as True — it needs no OpenTelemetry at all, which is why it is the default tool for unit tests (Step 6). OpenTelemetryTracer can also be constructed directly, but it imports opentelemetry in __init__ and will raise ImportError if the extra is missing; create_tracer exists so you never have to guard that yourself.

You can see the whole gate resolve in three lines:


from eventsource.observability import OTEL_AVAILABLE, create_tracer, should_trace

print(OTEL_AVAILABLE)                                    # depends on your install
print(should_trace(True))                                # same value as above
print(type(create_tracer(__name__, True)).__name__)      # OpenTelemetryTracer | NullTracer
print(create_tracer(__name__, False).enabled)            # always False

With that model in place, Step 3 is just a matter of knowing which constructors take the flag.