Problem with KafkaSupervisor after updating druid from 0.10.0 to 0.10.1

We are using Kafka Indexing Service (KafkaSupervisor) to pull data from existing Kafka topic.
This worked fine until I tried to update druid to 0.10.1 today (in out test-cluster).
Supervisor was running and also visible in the ‘new’ coordinator console, but no task were starting.
After searching the logs it was clear, there is a problem.
2017-10-12T11:15:56,991 WARN [KafkaSupervisor-analytics_raw_tracking_supervisor] io.druid.indexing.kafka.supervisor.KafkaSupervisor - Unable to get partition data from Kafka for brokers [hadoop11x:9092,hadoop11x:9092,hadoop11x:9092], are the brokers up?
org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeThrowInterruptException( ~[kafka-clients-]
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll( ~[kafka-clients-]
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll( ~[kafka-clients-]
at org.apache.kafka.clients.consumer.internals.Fetcher.getTopicMetadata( ~[kafka-clients-]
at org.apache.kafka.clients.consumer.internals.Fetcher.getAllTopicMetadata( ~[kafka-clients-]
at org.apache.kafka.clients.consumer.KafkaConsumer.listTopics( ~[kafka-clients-]
at io.druid.indexing.kafka.supervisor.KafkaSupervisor.updatePartitionDataFromKafka( [druid-kafka-indexing-service-0.10.1.jar:0.10.1]
at io.druid.indexing.kafka.supervisor.KafkaSupervisor.runInternal( [druid-kafka-indexing-service-0.10.1.jar:0.10.1]
at io.druid.indexing.kafka.supervisor.KafkaSupervisor$RunNotice.handle( [druid-kafka-indexing-service-0.10.1.jar:0.10.1]
at io.druid.indexing.kafka.supervisor.KafkaSupervisor$ [druid-kafka-indexing-service-0.10.1.jar:0.10.1]
at java.util.concurrent.Executors$ [?:1.8.0_111]
at [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker( [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$ [?:1.8.0_111]
at [?:1.8.0_111]
Caused by: java.lang.InterruptedException
… 15 more

Can anyone tell me what the problem is? Maybe Kafka client version?

We are also using Tranquility 0.8.2 to pull data from kafka, that worked fine after the update. Tranquility does not match
our requirements, because we want to use numeric dimensions which seams to be a problem with Tranquility.

(CDH5.7.1, Hadoop2.6.0, Kafka 0.9.0 (should be 0.10.0, but it worked before druid update))


Druid 0.10.1 uses Kafka-client version so probably you need to upgrade your Kafka cluster to Kafka 0.10.X or higher version.


We already discussed that and updateing Kafka is not a solution for us at the moment, because we would run into some troubles.

If we do an update, I will let you know if it fix our problem.