We are using the KIS to ingest data from Kafka. We are seeing some of our tasks gracefully terminate while in the READING state:
2017-08-01T02:41:05,287 INFO [qtp1101231294-111] io.druid.indexing.kafka.KafkaIndexTask - Stopping gracefully (status: [READING])
The task does some work and then terminates with:
2017-08-01T02:41:06,718 INFO [task-runner-0-priority-0] io.druid.indexing.kafka.KafkaIndexTask - The task was asked to stop before completing
It doesn’t look like this task every Publishes. So when this happens, a new task is created, but it restarts from a previous offset and re-reads a lot data from the kafka partition.
The stopped task seems to have gotten as far as the following kafka offsets:
2017-08-01T02:41:06,685 INFO [events-incremental-persist] io.druid.segment.realtime.appenderator.AppenderatorImpl - Committing metadata[FiniteAppenderatorDriverMetadata{activeSegments={index_kafka_events_f73152525982c3a_0=[events_2017-08-01T02:00:00.000Z_2017-08-01T03:00:00.000Z_2017-08-01T02:24:31.964Z_6], index_kafka_events_f73152525982c3a_3=[events_2017-08-01T02:00:00.000Z_2017-08-01T03:00:00.000Z_2017-08-01T02:24:31.964Z_3], index_kafka_events_f73152525982c3a_6=[events_2017-08-01T02:00:00.000Z_2017-08-01T03:00:00.000Z_2017-08-01T02:24:31.964Z_7]}, lastSegmentIds={index_kafka_events_f73152525982c3a_0=events_2017-08-01T02:00:00.000Z_2017-08-01T03:00:00.000Z_2017-08-01T02:24:31.964Z_6, index_kafka_events_f73152525982c3a_3=events_2017-08-01T02:00:00.000Z_2017-08-01T03:00:00.000Z_2017-08-01T02:24:31.964Z_3, index_kafka_events_f73152525982c3a_6=events_2017-08-01T02:00:00.000Z_2017-08-01T03:00:00.000Z_2017-08-01T02:24:31.964Z_7}, callerMetadata={nextPartitions=KafkaPartitions{topic=‘druid_ingestion’, partitionOffsetMap={0=25298723, 3=31623366, 6=23190480}}}}] for sinks[events_2017-08-01T02:00:00.000Z_2017-08-01T03:00:00.000Z_2017-08-01T02:24:31.964Z_7:31, events_2017-08-01T02:00:00.000Z_2017-08-01T03:00:00.000Z_2017-08-01T02:24:31.964Z_6:31, events_2017-08-01T02:00:00.000Z_2017-08-01T03:00:00.000Z_2017-08-01T02:24:31.964Z_3:31].
While the new task that is created begin with the following in it’s payload, the offsets for the partitions are well in the past:
{
“ioConfig”: {
“type”: “kafka”,
“baseSequenceName”: “index_kafka_events_58c62471501c99f”,
“startPartitions”: {
“topic”: “druid_ingestion”,
“partitionOffsetMap”: {
“0”: 8474599,
“3”: 9987135,
“6”: 8366294
}
},
“endPartitions”: {
“topic”: “druid_ingestion”,
“partitionOffsetMap”: {
“0”: 9223372036854775807,
“3”: 9223372036854775807,
“6”: 9223372036854775807
}
},
“consumerProperties”: {
“bootstrap.servers”: “10.198.5.188:9207,10.198.4.80:10039,10.198.5.46:10019,10.198.6.242:9411”
},
“useTransaction”: true,
“pauseAfterRead”: false,
“minimumMessageTime”: null
},
“context”: null,
“groupId”: “index_kafka_events”,
“dataSource”: “events”
}
Any help on why the task may get terminated while running?
Thanks.