Overlord error logs when one streaming datasource ingesting

a streaming datasource was stopped while it was running, we don’t know why, other datasources don’t have same problems. so I ingested the data through curl -X POST -H ‘Content-Type: application/json’ -d @homepage_streaming_tw_ingest.json http://druid-overlord00:8090/druid/indexer/v1/supervisor, after a few minutes, there are some overlord logs like below

2018-10-15T12:21:21,684 WARN [KafkaSupervisor-shopee_tw_s0__campaign_homepage-Reporting-0] io.druid.indexing.kafka.supervisor.KafkaSupervisor - Exception while getting current/latest offsets

org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException

at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeThrowInterruptException(ConsumerNetworkClient.java:417) ~[?:?]

at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:242) ~[?:?]

at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:188) ~[?:?]

at org.apache.kafka.clients.consumer.internals.Fetcher.getTopicMetadata(Fetcher.java:282) ~[?:?]

at org.apache.kafka.clients.consumer.internals.Fetcher.getAllTopicMetadata(Fetcher.java:262) ~[?:?]

at org.apache.kafka.clients.consumer.KafkaConsumer.listTopics(KafkaConsumer.java:1370) ~[?:?]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.updateLatestOffsetsFromKafka(KafkaSupervisor.java:2077) ~[?:?]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.lambda$updateCurrentAndLatestOffsets$27(KafkaSupervisor.java:2188) ~[?:?]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_121]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_121]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_121]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

Caused by: java.lang.InterruptedException

… 15 more

we found there is still no running task related the datasource.So I would know how to solve the issue. we used version druid 0.12.2. anyone can help you?thank you

Hi,

the stack trace looks that the supervisor failed to get the list of topics from Kafka.

Would you please double check that the bootstrap servers in your supervisor spec are valid?

Thanks,

Jihoon