Kafka And Druid Error

Hi Team,

Getting below error while ingesting data from kafka to druid in HDP 3.0.1

Please find attached code and commands

tail -500f /home/upglead1/examples/wikiticker-2015-09-12-sampled.json | /usr/hdp/3.0.1.0-187/kafka/bin/kafka-console-producer.sh --broker-list ipaddress:6667,ipaddress:6667 --topic kafkadruid1 --security-protocol SASL_PLAINTEXT

/usr/hdp/3.0.1.0-187/kafka/bin/kafka-console-consumer.sh --bootstrap-server ipaddress:6667,ipaddress6667 --topic kafkadruid1 --from-beginning --security-protocol SASL_PLAINTEXT

This both commands will work

Please can any one help me on this.

2019-02-18T08:39:44,616 WARN [KafkaSupervisor-kafkadruid01-Reporting-0] io.druid.indexing.kafka.supervisor.KafkaSupervisor - Unable to compute Kafka lag

io.druid.java.util.common.ISE: Latest offsets from Kafka have not been fetched

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.lambda$emitLag$19(KafkaSupervisor.java:2132) ~[?:?]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_112]

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_112]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_112]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_112]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]

2019-02-18T08:39:49,618 WARN [KafkaSupervisor-kafkadruid01] io.druid.indexing.kafka.supervisor.KafkaSupervisor - No such topic [kafkadruid1] found, list of discovered topics [[ATLAS_HOOK, ATLAS_ENTITIES]]

2019-02-18T08:39:49,619 INFO [KafkaSupervisor-kafkadruid01] io.druid.indexing.kafka.supervisor.KafkaSupervisor - {id=‘kafkadruid01’, generationTime=2019-02-18T08:39:49.619Z, payload={dataSource=‘kafkadruid01’, topic=‘kafkadruid1’, partitions=0, replicas=1, durationSeconds=3600, active=, publishing=}}

2019-02-18T08:39:53,709 INFO [TaskQueue-StorageSync] io.druid.indexing.overlord.TaskQueue - Synced 0 tasks from storage (0 tasks added, 0 tasks removed).

2019-02-18T08:40:04,618 WARN [KafkaSupervisor-kafkadruid01-Reporting-0] io.druid.indexing.kafka.supervisor.KafkaSupervisor - Exception while getting current/latest offsets

io.druid.java.util.common.ISE: Could not retrieve partitions for topic [kafkadruid1]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.updateLatestOffsetsFromKafka(KafkaSupervisor.java:2082) ~[?:?]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.lambda$updateCurrentAndLatestOffsets$24(KafkaSupervisor.java:2190) ~[?:?]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_112]

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_112]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_112]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_112]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]

2019-02-18T08:40:19,619 WARN [KafkaSupervisor-kafkadruid01] io.druid.indexing.kafka.supervisor.KafkaSupervisor - No such topic [kafkadruid1] found, list of discovered topics [[ATLAS_HOOK, ATLAS_ENTITIES]]

2019-02-18T08:40:19,620 INFO [KafkaSupervisor-kafkadruid01] io.druid.indexing.kafka.supervisor.KafkaSupervisor - {id=‘kafkadruid01’, generationTime=2019-02-18T08:40:19.620Z, payload={dataSource=‘kafkadruid01’, topic=‘kafkadruid1’, partitions=0, replicas=1, durationSeconds=3600, active=, publishing=}}

2019-02-18T08:40:34,621 WARN [KafkaSupervisor-kafkadruid01-Reporting-0] io.druid.indexing.kafka.supervisor.KafkaSupervisor - Exception while getting current/latest offsets

io.druid.java.util.common.ISE: Could not retrieve partitions for topic [kafkadruid1]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.updateLatestOffsetsFromKafka(KafkaSupervisor.java:2082) ~[?:?]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.lambda$updateCurrentAndLatestOffsets$24(KafkaSupervisor.java:2190) ~[?:?]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_112]

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_112]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_112]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_112]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]

2019-02-18T08:40:44,616 WARN [KafkaSupervisor-kafkadruid01-Reporting-0] io.druid.indexing.kafka.supervisor.KafkaSupervisor - Unable to compute Kafka lag

io.druid.java.util.common.ISE: Latest offsets from Kafka have not been fetched

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.lambda$emitLag$19(KafkaSupervisor.java:2132) ~[?:?]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_112]

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_112]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_112]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_112]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]

2019-02-18T08:40:49,617 WARN [KafkaSupervisor-kafkadruid01] io.druid.indexing.kafka.supervisor.KafkaSupervisor - No such topic [kafkadruid1] found, list of discovered topics [[ATLAS_HOOK, ATLAS_ENTITIES]]

2019-02-18T08:40:49,620 INFO [KafkaSupervisor-kafkadruid01] io.druid.indexing.kafka.supervisor.KafkaSupervisor - {id=‘kafkadruid01’, generationTime=2019-02-18T08:40:49.620Z, payload={dataSource=‘kafkadruid01’, topic=‘kafkadruid1’, partitions=0, replicas=1, durationSeconds=3600, active=, publishing=}}

2019-02-18T08:40:53,709 INFO [TaskQueue-StorageSync] io.druid.indexing.overlord.TaskQueue - Synced 0 tasks from storage (0 tasks added, 0 tasks removed).

2019-02-18T08:41:04,618 WARN [KafkaSupervisor-kafkadruid01-Reporting-0] io.druid.indexing.kafka.supervisor.KafkaSupervisor - Exception while getting current/latest offsets

io.druid.java.util.common.ISE: Could not retrieve partitions for topic [kafkadruid1]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.updateLatestOffsetsFromKafka(KafkaSupervisor.java:2082) ~[?:?]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.lambda$updateCurrentAndLatestOffsets$24(KafkaSupervisor.java:2190) ~[?:?]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_112]

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_112]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_112]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_112]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]

2019-02-18T08:41:19,618 WARN [KafkaSupervisor-kafkadruid01] io.druid.indexing.kafka.supervisor.KafkaSupervisor - No such topic [kafkadruid1] found, list of discovered topics [[ATLAS_HOOK, ATLAS_ENTITIES]]

2019-02-18T08:41:19,619 INFO [KafkaSupervisor-kafkadruid01] io.druid.indexing.kafka.supervisor.KafkaSupervisor - {id=‘kafkadruid01’, generationTime=2019-02-18T08:41:19.619Z, payload={dataSource=‘kafkadruid01’, topic=‘kafkadruid1’, partitions=0, replicas=1, durationSeconds=3600, active=, publishing=}}

2019-02-18T08:41:34,619 WARN [KafkaSupervisor-kafkadruid01-Reporting-0] io.druid.indexing.kafka.supervisor.KafkaSupervisor - Exception while getting current/latest offsets

io.druid.java.util.common.ISE: Could not retrieve partitions for topic [kafkadruid1]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.updateLatestOffsetsFromKafka(KafkaSupervisor.java:2082) ~[?:?]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.lambda$updateCurrentAndLatestOffsets$24(KafkaSupervisor.java:2190) ~[?:?]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_112]

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_112]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_112]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_112]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]

wikipedia-kafka-supervisor1.json (1.1 KB)

Hi Team,

Can any one help me on this.

Thanks

Mohammad Layeeq

Hi Team,

Can any one help me on this error , I am not able figure it out.

Thanks

Mohammad

Hi Team,

Can somebody help me on this error.

Thanks

Hey Layeeq,

I am not an expert on druid but the errors looks like the topic “kafkadruid1” doesn’t exist. Can you verify if the topic exists and has data ?

Thanks !