Some kafka index services can't ingest data

There are a lot of kafka index services, we have submit them successfully, but a few hours later, there are some kis tasks stopped ingesting data and no related running tasks, I had confirmed new data produced to these topics, what caused this issue? anyone have the same problems?thank you, the exception are below in overlord

2018-09-12T19:29:13,130 ERROR [KafkaSupervisor-shopee_my_s0__campaign_microsite] io.druid.indexing.kafka.supervisor.KafkaSupervisor - KafkaSupervisor[shopee_my_s0__campaign_microsite] failed to handle notice: {class=io.druid.indexing.kafka.supervisor.KafkaSupervisor, exceptionType=class java.lang.IllegalStateException, exceptionMessage=This consumer has already been closed., noticeClass=RunNotice}

java.lang.IllegalStateException: This consumer has already been closed.

at org.apache.kafka.clients.consumer.KafkaConsumer.ensureNotClosed(KafkaConsumer.java:1611) ~[kafka-clients-0.10.2.0.jar:?]

at org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:1622) ~[kafka-clients-0.10.2.0.jar:?]

at org.apache.kafka.clients.consumer.KafkaConsumer.assignment(KafkaConsumer.java:759) ~[kafka-clients-0.10.2.0.jar:?]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.getOffsetFromKafkaForPartition(KafkaSupervisor.java:1886) ~[druid-kafka-indexing-service-0.12.0.jar:0.12.0]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.getOffsetFromStorageForPartition(KafkaSupervisor.java:1839) ~[druid-kafka-indexing-service-0.12.0.jar:0.12.0]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.generateStartingOffsetsForPartitionGroup(KafkaSupervisor.java:1821) ~[druid-kafka-indexing-service-0.12.0.jar:0.12.0]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.createNewTasks(KafkaSupervisor.java:1712) ~[druid-kafka-indexing-service-0.12.0.jar:0.12.0]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.runInternal(KafkaSupervisor.java:825) ~[druid-kafka-indexing-service-0.12.0.jar:0.12.0]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor$RunNotice.handle(KafkaSupervisor.java:574) ~[druid-kafka-indexing-service-0.12.0.jar:0.12.0]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor$2.run(KafkaSupervisor.java:366) [druid-kafka-indexing-service-0.12.0.jar:0.12.0]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

2018-09-12T19:29:13,140 ERROR [KafkaSupervisor-shopee_ph_s0__campaign_microsite] io.druid.indexing.kafka.supervisor.KafkaSupervisor - No task in [[index_kafka_shopee_ph_s0__campaign_microsite_92c63c0ed0737cb_nehnbobo]] succeeded before the completion timeout elapsed [PT1800S]!: {class=io.druid.indexing.kafka.supervisor.KafkaSupervisor}

2018-09-12T19:29:13,140 ERROR [KafkaSupervisor-shopee_ph_s0__campaign_microsite] io.druid.indexing.kafka.supervisor.KafkaSupervisor - KafkaSupervisor[shopee_ph_s0__campaign_microsite] failed to handle notice: {class=io.druid.indexing.kafka.supervisor.KafkaSupervisor, exceptionType=class java.lang.NullPointerException, exceptionMessage=null, noticeClass=RunNotice}

java.lang.NullPointerException

This looks like some of Kafka ingestion bugs that were fixed after 0.12.0:

https://github.com/apache/incubator-druid/pull/5902

https://github.com/apache/incubator-druid/issues/5900

Can you try running 0.12.2 or 0.12.3-rc1?

ok. thank you so much, I have been using 0.12.0. As I see that, the codes only updated in the kafkaSupervisor, so I just need to update the druid-kafka-indexing-service-0.12.0.jar to druid-kafka-indexing-service-0.12.2.jar, anything else would be ok, is that right?

You should upgrade all the jars, since there are also changes to “indexing-service” and other Druid packages between 0.12.0 -> current version.

Thanks,

Jon

understand, thank you