Kafka Event Bus Metrics & Observability
KafkaEventBus provides native OpenTelemetry metrics for throughput, latency, consumer group lag, rebalances, and dead-letter queues. These metrics follow OpenTelemetry semantic conventions for messaging systems and allow monitoring production event streaming with Prometheus and Grafana.
Enabling Metrics
Metrics are enabled by default on KafkaEventBusConfig. Ensure that OpenTelemetry is installed (pip install "eventsource-py[all]") and your OpenTelemetry SDK MeterProvider is configured:
from eventsource.adapters.kafka import KafkaEventBus, KafkaEventBusConfig
config = KafkaEventBusConfig(
bootstrap_servers="kafka:9092",
topic_prefix="events",
consumer_group="order-projections",
enable_metrics=True, # Default: True
)
bus = KafkaEventBus(config=config, event_registry=registry)
await bus.connect()
await bus.start_consuming()
When enable_metrics=True and OpenTelemetry is present, KafkaEventBus registers metric instruments with the global MeterProvider under the meter name "eventsource.bus.kafka".
Metric Instruments Catalog
Counter Metrics
| Metric Name | Unit | Attributes / Labels | Description |
|---|---|---|---|
kafka.eventbus.messages.published |
messages |
messaging.system, messaging.destination, event.type |
Total count of messages successfully published to Kafka. |
kafka.eventbus.messages.consumed |
messages |
messaging.system, messaging.destination, messaging.kafka.partition, event.type |
Total count of messages consumed and dispatched to handlers. |
kafka.eventbus.handler.invocations |
invocations |
handler.name, event.type |
Number of times an event handler was invoked. |
kafka.eventbus.handler.errors |
errors |
handler.name, event.type, error.type |
Number of exceptions raised by event handlers during dispatch. |
kafka.eventbus.messages.dlq |
messages |
dlq.reason, error.type |
Messages routed to the dead-letter queue after retries exhausted. |
kafka.eventbus.connection.errors |
errors |
error.type |
Connection dropped or broker connectivity errors. |
kafka.eventbus.reconnections |
attempts |
_(none)_ | Total reconnection attempts initiated by the bus. |
kafka.eventbus.rebalances |
rebalances |
messaging.kafka.consumer_group |
Number of consumer group partition rebalance events. |
kafka.eventbus.publish.errors |
errors |
messaging.system, messaging.destination, event.type, error.type |
Number of failed publish attempts. |
Histogram Metrics
| Metric Name | Unit | Attributes / Labels | Description |
|---|---|---|---|
kafka.eventbus.publish.duration |
ms |
messaging.destination |
End-to-end latency to acknowledge a publish request. |
kafka.eventbus.consume.duration |
ms |
messaging.destination |
Duration from message delivery to completion of all handlers. |
kafka.eventbus.handler.duration |
ms |
handler.name, event.type |
Execution duration of an individual handler callable. |
kafka.eventbus.batch.size |
messages |
_(none)_ | Distribution of message counts in batch publish() calls. |
Observable Gauge Metrics
| Metric Name | Unit | Attributes / Labels | Description |
|---|---|---|---|
kafka.eventbus.connections.active |
1 |
messaging.system |
Connection status (1 = connected, 0 = disconnected). |
kafka.eventbus.consumer.lag |
messages |
messaging.kafka.consumer_group, messaging.destination, messaging.kafka.partition |
Number of uncommitted messages behind the partition high-water mark. |
PromQL Queries for Monitoring & Alerting
1. Throughput (Published vs Consumed Rate)
# Published events per second by event type
sum by (event_type) (rate(kafka_eventbus_messages_published_total[5m]))
# Consumed events per second by topic
sum by (messaging_destination) (rate(kafka_eventbus_messages_consumed_total[5m]))
2. Error Rate & Handler Failures
# Handler error rate percentage
sum(rate(kafka_eventbus_handler_errors_total[5m]))
/
sum(rate(kafka_eventbus_handler_invocations_total[5m])) * 100
3. P99 Latency
# Handler execution 99th percentile latency (seconds)
histogram_quantile(0.99, sum by (le, handler_name) (rate(kafka_eventbus_handler_duration_milliseconds_bucket[5m]))) / 1000
# Publish 99th percentile latency
histogram_quantile(0.99, sum by (le) (rate(kafka_eventbus_publish_duration_milliseconds_bucket[5m])))
4. Consumer Group Lag per Partition
# Total lag across consumer group
sum by (messaging_kafka_consumer_group) (kafka_eventbus_consumer_lag)
# Partitions with highest lag
topk(5, kafka_eventbus_consumer_lag)
Recommended Prometheus Alerting Rules
groups:
- name: kafka-eventbus-alerts
rules:
- alert: KafkaConsumerLagSpike
expr: sum by (messaging_kafka_consumer_group) (kafka_eventbus_consumer_lag) > 5000
for: 5m
labels:
severity: warning
annotations:
summary: "Kafka consumer group {{ $labels.messaging_kafka_consumer_group }} lagging behind"
description: "Consumer group has accumulated more than 5,000 unconsumed events for over 5 minutes."
- alert: KafkaHighDLQRate
expr: sum(rate(kafka_eventbus_messages_dlq_total[5m])) > 0.1
for: 2m
labels:
severity: critical
annotations:
summary: "Messages are being sent to DLQ"
description: "Dead-letter queue rate is > 0.1 msg/sec. Inspect failing events immediately."
- alert: KafkaRebalanceStorm
expr: sum by (messaging_kafka_consumer_group) (rate(kafka_eventbus_rebalances_total[10m])) > 0.05
for: 5m
labels:
severity: warning
annotations:
summary: "Frequent Kafka rebalances detected"
description: "Consumer group {{ $labels.messaging_kafka_consumer_group }} is experiencing frequent rebalances."
- alert: KafkaBrokerDisconnected
expr: kafka_eventbus_connections_active == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Kafka Event Bus disconnected"
description: "Kafka connection is down for more than 1 minute."