Exposing Kafka Ingestion Lag

Hello!

Could anyone please suggest what the one need to do to expose ingest/kafka/lag on overload/peon?

I’ve configured statsd emitter both on overload and middle manager and added kafka-indexing-service extension. However, I still cannot get the lag metrics to get exposed.

druid.extensions.loadList=[“statsd-emitter”, “druid-hdfs-storage”, “druid-kafka-indexing-service”, “druid-histogram”, “druid-datasketches”, “druid-lookups-cached-global”, “mysql-metadata-storage”]

druid.extensions.directory=/opt/druid/extensions

druid.emitter=statsd

druid.emitter.statsd.hostname=localhost

druid.emitter.statsd.port=8670

druid.emitter.statsd.dimensionMapPath=/opt/druid/conf/druid/statdsMetricsMapping.json

Highly appreciate any suggestions.

Hi Sayat,

Have you enabled metric emission? Please see: http://druid.io/docs/latest/configuration/index.html#enabling-metrics

Best,

Caroline

Small update on my side. Apprenlty, there’s no need to include RealtimeMontoring to incldue into metrics.monitor list. Kafka lag get exposed in kafaka indexing supervisor. To expose it one need:

  1. kafka indexing extension, without it kafka ingestion won’t work

  2. ensure that overlord has configured emitter i.e. druid.emitter property.

KafkaSupervisor will emit lag with peridocity that was defined in configs(or use the default one)