Unable to kill a running task of Kafka indexing service

Hi,

I ingested data using kafka indexing service. I shutdown that kafka indexing service and tried to kill running task. But the task doesn’t get killed. This task has no logs. I check overlord logs and it is retrying to kill this task for long time(more than couple of hours). Below are overlord logs. What could be the cause of this ? How to avoid it? And how to kill this task?

26/Feb/2018 16:17:52,185- TaskQueue: Asking taskRunner to clean up 1 tasks.

26/Feb/2018 16:17:52,185- ChannelResourceFactory: Generating: http://node.com:8091

26/Feb/2018 16:17:52,187- RemoteTaskRunner: Sent shutdown message to worker: node.com:8091, status 200 OK, response: {“task”:“index_kafka_foo_bar_f69bfd0b3fcf497_cgmmphnj”}

26/Feb/2018 16:18:52,185- TaskQueue: Synced 1 tasks from storage (0 tasks added, 0 tasks removed).

26/Feb/2018 16:18:52,185- TaskQueue: Asking taskRunner to clean up 1 tasks.

26/Feb/2018 16:18:52,185- ChannelResourceFactory: Generating: http://node.com:8091

26/Feb/2018 16:18:52,188- RemoteTaskRunner: Sent shutdown message to worker: node.com:8091, status 200 OK, response: {“task”:“index_kafka_foo_bar_f69bfd0b3fcf497_cgmmphnj”}

26/Feb/2018 16:19:52,185- TaskQueue: Synced 1 tasks from storage (0 tasks added, 0 tasks removed).

26/Feb/2018 16:19:52,185- TaskQueue: Asking taskRunner to clean up 1 tasks.

26/Feb/2018 16:19:52,186- ChannelResourceFactory: Generating: http://node.com:8091

26/Feb/2018 16:19:52,217- RemoteTaskRunner: Sent shutdown message to worker: node.com:8091, status 200 OK, response: {“task”:“index_kafka_foo_bar_f69bfd0b3fcf497_cgmmphnj”}

26/Feb/2018 16:20:52,185- TaskQueue: Synced 1 tasks from storage (0 tasks added, 0 tasks removed).

26/Feb/2018 16:20:52,185- TaskQueue: Asking taskRunner to clean up 1 tasks.

26/Feb/2018 16:20:52,185- ChannelResourceFactory: Generating: http://node.com:8091

26/Feb/2018 16:20:52,191- RemoteTaskRunner: Sent shutdown message to worker: node.com:8091, status 200 OK, response: {“task”:“index_kafka_foo_bar_f69bfd0b3fcf497_cgmmphnj”}

``

FYI: The ingest job had task duration as 1 hour.

Any suggestions are appreciated.

Thanks

Prashanth

FYI, I’m using Druid 0.11.0.

Hi Prashanth,

It looks like for some reason the worker is not shutting down the task like it promised. Do you see anything interesting in the middleManager log for that worker?