Ingestion Metrics docs talk about realtime node config

Hiya,

I’m looking to have the Kafka consumer lag emitted in the logs. I’ve seen that the lag is emitted to the metric ingest/kafka/lag (https://github.com/pjain1/druid/blob/master/extensions-core/kafka-indexing-service/src/main/java/io/druid/indexing/kafka/supervisor/KafkaSupervisor.java#L2153) but when I check that metric in the metrics documentation http://druid.io/docs/latest/operations/metrics.html it mentions that ingestion metrics ‘…are only available if the RealtimeMetricsMonitor is included in the monitors list for the Realtime node. These metrics are deltas for each emission period.’

IIRC realtime nodes are a deprecated system and we’re using the Kafka Indexing Service. So then, what node config needs updating to output the kafka topic consumer lag messages?

(I’m aware of the supervisor status, but that doesn’t fit well into our metrics systems)

Thanks,

Dyana

be good if I actually linked the latest code correctly rather than on old fork :frowning:

Thanks for the question, our docs there will need to be updated.

For ingest/kafka/lag, the emission is not tied to a Monitor, so it should always be emitted by the supervisor.

For the other ingestion metrics like event counts, the Monitor object that handles those metrics will always be installed for Kafka Indexing Service tasks and no additional config is needed:

https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java#L329

Hi,

Reviving this thread since I have a related question: is it possible to monitor ingest/** metrics in batch indexing as well? I wasn’t able to see them when using the logging emitter and I’m not sure what extra config is required.

Thanks,

Eran

Hi

I hope Metrics is enable and being emitted. Are you not seeing the metrics listed under “Indexing Service” in the following?

https://druid.apache.org/docs/latest/operations/metrics.html

Thanks & Regards

Venkat

I can see metrics emitted. Haven’t checked if the task metrics are emitted.

I’m trying to measure the ingestion throughput, so I’m interested in the ingestion metrics listed right above “Indexing Service” in the metrics documentation. Are those available at all, or is there another method to measure ingestion throughput/latency in batch and stream indexing?

Thanks,

Eran.