How to monitor consumer lag in Kafka Indexing Service

Apache Druid Q&A Google Groups
#1

Hi,

I am using Kafka Indexing Service to ingest data into druid(0.9.2). I would like to monitor the consumer lag metrics for indexing Kafka consumer.

I initially thought I would use kafka.consumer lag metrics but later looked at Kafka zookeeper and realized that Kafka Indexing Service doesn’t commit offsets to Kafka. I am assuming that offsets are stored in Druid’s metastore table.

Druid on the other end doesn’t emit any consumer lag metric(http://druid.io/docs/latest/operations/metrics.html).

Is there any other way to monitor consumer lag? Another idea which comes to my mind is to pass auto commit properties in “consumerProperties” section of Supervisor spec but not sure, that might lead to some side effects in Kafka ingestion.

Thanks

Anuraj

#2

Druid does the tracking in its own metadata store, so that’s why you don’t see them in Kafka. The supervisor status report has the current offsets under “currentOffsets” for each task. Check out “Get Supervisor Status Report” on http://druid.io/docs/latest/development/extensions-core/kafka-ingestion.html.

#3

Thanks for the info Gian. I think “Get Supervisor Status Report” is good for our current monitoring use case.