My name is Dan and this is my first post here, so please bare with me. I wanted to share with you an issue that we encountered with Kafka indexing service and maybe you’ll add more support for this particular case in a further version.
In short, we are using Kafka Indexing Service to ingest data from a Kafka topic into Druid - the problem was that we didn’t had any task spawn by the middle manager and the logs didn’t show anything unusual :
Enter 2017-03-15T10:14:38,778 INFO [KafkaSupervisor-xxx] io.druid.indexing.overlord.RemoteTaskRunner - Registered listener [KafkaSupervisor-xxx]
2017-03-15T10:14:39,035 INFO [KafkaSupervisor-xxx] io.druid.indexing.kafka.supervisor.KafkaSupervisor - New partition  discovered for topic [yyy], added to task group 
2017-03-15T10:14:39,036 INFO [KafkaSupervisor-xx] io.druid.indexing.kafka.supervisor.KafkaSupervisor - Creating new task group  for partitions 
2017-03-15T10:15:27,043 INFO [TaskQueue-StorageSync] io.druid.indexing.overlord.TaskQueue - Synced 0 tasks from storage (0 tasks added, 0 tasks removed).code here…
After debugging, we pinpoint the issue on the Kafka side: we had previously some test topics created by brokers which weren’t available anymore (AWS instances shut down)
We are on the version 0.9.2 and the process get stuck on KafkaSupervisor.getOffsetFromKafkaForPartition(), line 1380:
Going through the Kafka code, in AbstractCoordinator.ensureCoordinatorKnown() , I can see an exception swallowed - “The group coordinator is not available.” and apparently loop forever.
After cleaning up our Zookeeper, everything get back to normal, but it would be helpfull if more we get more information in logs