How to check if data is being pulled from kafka topic by supervisor?


Is there any way we can verify if data is loaded from Kafka topic by supervisor. ? Also is it possible to monitor if supervisors or if the subsequent tasks are failing ?

Supervisor status report provide us with “latest offsets” field. Can this be used to monitor the supervisors ?


Druid has a metrics system:

We monitor the ingest/kafka/lag metric from the overlord to ensure that Druid isn’t getting behind.

My colleague asked recently about monitoring for task failures at

The conclusion was that there isn’t a direct metric for it, so she added a task that polls the Druid completed task list API to our own monitoring setup (as described in the thread).