Druid Kafka Supervisor not is not spawning task

I am trying to do kafka streaming to Druid, after creating supervisor its not spawning tasks which will listen to incoming messages.

I am following the sample tutorial given for loading Wikipedia page details.

Note : I was able to receive the data from consumer console listening via the same topic

--------------- This is the message coming in the log

org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - Could not fetch partitions for topic/stream [wikipedia22]
2019-06-28T13:18:46,615 WARN [KafkaSupervisor-wikipedia-Reporting-0] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - Exception while getting current/latest sequences
java.lang.RuntimeException: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.updateLatestOffsetsFromStream(SeekableStreamSupervisor.java:2652) ~[druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.lambda$updateCurrentAndLatestOffsets$37(SeekableStreamSupervisor.java:2609) ~[druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_171]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_171]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_171]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
2019-06-28T13:18:46,616 WARN [KafkaSupervisor-wikipedia-Reporting-0] org.apache.druid.indexing.kafka.supervisor.KafkaSupervisor - Unable to compute Kafka lag
org.apache.druid.java.util.common.ISE: Latest offsets from Kafka have not been fetched
at org.apache.druid.indexing.kafka.supervisor.KafkaSupervisor.lambda$emitLag$2(KafkaSupervisor.java:298) ~[?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_171]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_171]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_171]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
2019-06-28T13:18:47,936 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.DruidCoordinator - Metadata store not polled yet, skipping this run.
2019-06-28T13:18:52,936 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.DruidCoordinator - Metadata store not polled yet, skipping this run.
2019-06-28T13:18:57,936 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.DruidCoordinator - Metadata store not polled yet, skipping this run.
2019-06-28T13:19:02,166 WARN [DatabaseSegmentManager-Exec–0] org.apache.druid.metadata.SQLMetadataSegmentManager - No segments found in the database!
2019-06-28T13:19:02,472 INFO [DatabaseRuleManager-Exec–0] org.apache.druid.metadata.SQLMetadataRuleManager - Polled and found 1 rule(s) for 1 datasource(s)
2019-06-28T13:19:02,479 INFO [TaskQueue-StorageSync] org.apache.druid.indexing.overlord.TaskQueue - Synced 0 tasks from storage (0 tasks added, 0 tasks removed).
2019-06-28T13:19:02,937 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.DruidCoordinator - Metadata store not polled yet, skipping this run.
2019-06-28T13:19:07,937 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.DruidCoordinator - Metadata store not polled yet, skipping this run.
2019-06-28T13:19:12,937 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.DruidCoordinator - Metadata store not polled yet, skipping this run.
2019-06-28T13:19:17,938 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.DruidCoordinator - Metadata store not polled yet, skipping this run.
2019-06-28T13:19:22,938 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.DruidCoordinator - Metadata store not polled yet, skipping this run.
2019-06-28T13:19:27,939 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.DruidCoordinator - Metadata store not polled yet, skipping this run.
2019-06-28T13:19:32,939 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.DruidCoordinator - Metadata store not polled yet, skipping this run.
2019-06-28T13:19:37,939 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.DruidCoordinator - Metadata store not polled yet, skipping this run.

Hi Aravindan,

Have you confirmed that you’re able to reach all the Kafka brokers from the machine running the overlord (using either the Kafka console consumer or even something like telnet) ? Could you post the full logs from the overlord?

Hi David,

I was using older version of kafka. I downloaded new one and I am able to ingest to druid from kafka…

But I ran into weird issue now… the load data option on the menu bar in the druid console is not visible now…

Strange, could you post a screenshot? Might you be running a different version of Druid? The data loader was just recently added in 0.15.0.

Thanks for helping , looks like browser issue. cleared cache on my browser and it worked.

I have few questions on memory management in druid. Can you please let me know the details i am looking on the thread “Question on Memory management in Druid”