Kafka Indexing Service Tasks getting stopped unexpectedly while in RUNNING

We are using the KIS to ingest data from Kafka. We are seeing some of our tasks gracefully terminate while in the READING state:

2017-08-01T02:41:05,287 INFO [qtp1101231294-111] io.druid.indexing.kafka.KafkaIndexTask - Stopping gracefully (status: [READING])

The task does some work and then terminates with:

2017-08-01T02:41:06,718 INFO [task-runner-0-priority-0] io.druid.indexing.kafka.KafkaIndexTask - The task was asked to stop before completing

It doesn’t look like this task every Publishes. So when this happens, a new task is created, but it restarts from a previous offset and re-reads a lot data from the kafka partition.

The stopped task seems to have gotten as far as the following kafka offsets:

2017-08-01T02:41:06,685 INFO [events-incremental-persist] io.druid.segment.realtime.appenderator.AppenderatorImpl - Committing metadata[FiniteAppenderatorDriverMetadata{activeSegments={index_kafka_events_f73152525982c3a_0=[events_2017-08-01T02:00:00.000Z_2017-08-01T03:00:00.000Z_2017-08-01T02:24:31.964Z_6], index_kafka_events_f73152525982c3a_3=[events_2017-08-01T02:00:00.000Z_2017-08-01T03:00:00.000Z_2017-08-01T02:24:31.964Z_3], index_kafka_events_f73152525982c3a_6=[events_2017-08-01T02:00:00.000Z_2017-08-01T03:00:00.000Z_2017-08-01T02:24:31.964Z_7]}, lastSegmentIds={index_kafka_events_f73152525982c3a_0=events_2017-08-01T02:00:00.000Z_2017-08-01T03:00:00.000Z_2017-08-01T02:24:31.964Z_6, index_kafka_events_f73152525982c3a_3=events_2017-08-01T02:00:00.000Z_2017-08-01T03:00:00.000Z_2017-08-01T02:24:31.964Z_3, index_kafka_events_f73152525982c3a_6=events_2017-08-01T02:00:00.000Z_2017-08-01T03:00:00.000Z_2017-08-01T02:24:31.964Z_7}, callerMetadata={nextPartitions=KafkaPartitions{topic=‘druid_ingestion’, partitionOffsetMap={0=25298723, 3=31623366, 6=23190480}}}}] for sinks[events_2017-08-01T02:00:00.000Z_2017-08-01T03:00:00.000Z_2017-08-01T02:24:31.964Z_7:31, events_2017-08-01T02:00:00.000Z_2017-08-01T03:00:00.000Z_2017-08-01T02:24:31.964Z_6:31, events_2017-08-01T02:00:00.000Z_2017-08-01T03:00:00.000Z_2017-08-01T02:24:31.964Z_3:31].


While the new task that is created begin with the following in it’s payload, the offsets for the partitions are well in the past:

{
“ioConfig”: {
“type”: “kafka”,
“baseSequenceName”: “index_kafka_events_58c62471501c99f”,
“startPartitions”: {
“topic”: “druid_ingestion”,
“partitionOffsetMap”: {
“0”: 8474599,
“3”: 9987135,
“6”: 8366294
}
},
“endPartitions”: {
“topic”: “druid_ingestion”,
“partitionOffsetMap”: {
“0”: 9223372036854775807,
“3”: 9223372036854775807,
“6”: 9223372036854775807
}
},
“consumerProperties”: {
“bootstrap.servers”: “10.198.5.188:9207,10.198.4.80:10039,10.198.5.46:10019,10.198.6.242:9411”
},
“useTransaction”: true,
“pauseAfterRead”: false,
“minimumMessageTime”: null
},
“context”: null,
“groupId”: “index_kafka_events”,
“dataSource”: “events”
}

Any help on why the task may get terminated while running?

Thanks.