Kafka Indexing tasks failing with "failed to pause" error

Hi,

While running Kafka Indexing service to ingest data from Kafka, i observed that some of the tasks are failing. There is no error in the log other than few lines in overlord log -

2018-02-13T08:36:01,703 INFO [KafkaIndexTaskClient-zai-v2-3] io.druid.indexing.kafka.KafkaIndexTaskClient - Still waiting for task [index_kafka_zai-v2_5c4ff842e794436_obebicli] to pause; will try again in [PT10S]

2018-02-13T08:36:01,703 INFO [KafkaIndexTaskClient-zai-v2-0] io.druid.indexing.kafka.KafkaIndexTaskClient - Still waiting for task [index_kafka_zai-v2_d22ae39248bd484_kggcalhf] to pause; will try again in [PT10S]

2018-02-13T08:36:31,724 ERROR [KafkaIndexTaskClient-zai-v2-3] io.druid.indexing.kafka.KafkaIndexTaskClient - Task [index_kafka_zai-v2_5c4ff842e794436_obebicli] failed to pause, aborting

2018-02-13T08:36:31,725 WARN [KafkaSupervisor-zai-v2-Worker-0] io.druid.indexing.kafka.supervisor.KafkaSupervisor - Task [index_kafka_zai-v2_5c4ff842e794436_obebicli] failed to respond to [pause] in a timely manner, killing task

2018-02-13T08:36:31,727 INFO [KafkaSupervisor-zai-v2-Worker-0] io.druid.indexing.overlord.RemoteTaskRunner - Sent shutdown message to worker: ip-10-40-16-14.eu-west-1.compute.internal:8091, status 200 OK, response: {“task”:“index_kafka_zai-v2_5c4ff842e794436_obebicli”}

Corresponding MiddleManager log -

2018-02-13T08:36:31,727 INFO [qtp1064315206-29] io.druid.indexing.overlord.ForkingTaskRunner - Killing process for task: index_kafka_zai-v2_5c4ff842e794436_obebicli

2018-02-13T08:36:31,769 INFO [qtp1064315206-21] io.druid.indexing.overlord.ForkingTaskRunner - Killing process for task: index_kafka_zai-v2_5c4ff842e794436_obebicli

2018-02-13T08:36:33,764 INFO [forking-task-runner-8] io.druid.indexing.overlord.ForkingTaskRunner - Exception caught during execution

java.io.IOException: Stream closed

    at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170) ~[?:1.8.0_161]

    at java.io.BufferedInputStream.read1(BufferedInputStream.java:291) ~[?:1.8.0_161]

    at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_161]

    at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[?:1.8.0_161]

    at com.google.common.io.ByteStreams.copy(ByteStreams.java:175) ~[guava-16.0.1.jar:?]

    at io.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:452) [druid-indexing-service-0.11.0.jar:0.11.0]

    at io.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:224) [druid-indexing-service-0.11.0.jar:0.11.0]

    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]

    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

2018-02-13T08:36:35,468 ERROR [forking-task-runner-8] io.druid.indexing.overlord.ForkingTaskRunner - Failed to delete task directory: {class=io.druid.indexing.overlord.ForkingTaskRunner, exceptionType=class java.io.IOException, exceptionMessage=Unable to delete directory /tmp/druid/persistent/task/index_kafka_zai-v2_5c4ff842e794436_obebicli/work/persist/zai-v2_2018-02-06T18:00:00.000Z_2018-02-06T19:00:00.000Z_2018-02-13T07:51:51.298Z_72/19/v9-tmp/tmpPeonFiles., taskDir=/tmp/druid/persistent/task/index_kafka_zai-v2_5c4ff842e794436_obebicli, task=index_kafka_zai-v2_5c4ff842e794436_obebicli}

java.io.IOException: Unable to delete directory /tmp/druid/persistent/task/index_kafka_zai-v2_5c4ff842e794436_obebicli/work/persist/zai-v2_2018-02-06T18:00:00.000Z_2018-02-06T19:00:00.000Z_2018-02-13T07:51:51.298Z_72/19/v9-tmp/tmpPeonFiles.

Other than this, there is no other error or info anywhere. Also, the same task was retried by KIS again and it was successful.

Can you help me figure out why is this happening or what is the error ?

Thanks,
Chaitanya