How to check if data is being pulled from kafka topic by supervisor?

Hello,

Is there any way we can verify if data is loaded from Kafka topic by supervisor. ? Also is it possible to monitor if supervisors or if the subsequent tasks are failing ?

Supervisor status report provide us with “latest offsets” field. Can this be used to monitor the supervisors ?

Thanks

Druid has a metrics system: http://druid.io/docs/latest/operations/metrics.html

We monitor the ingest/kafka/lag metric from the overlord to ensure that Druid isn’t getting behind.

My colleague asked recently about monitoring for task failures at https://groups.google.com/d/msg/druid-user/Qqne_FuuTGk/j7uAIUVvBwAJ

The conclusion was that there isn’t a direct metric for it, so she added a task that polls the Druid completed task list API to our own monitoring setup (as described in the thread).

–dave