0.13.0 Kafka Indexing task pausing forever if no data received in intermediateHandoffPeriod

I’ve been running a number of tests locally with Kafka indexing and I believe something similar to https://github.com/apache/incubator-druid/commit/638f50cb52c248f4408975d5fc7762cc9ce82d8e is still occurring.

I’ve set the intermediateHandoffPeriod to a low value PT5M while testing to see what handoffs and shards will look like.

If I send data in every < 5 minutes then the task continues indexing. However if there is a gap where no data lands during a hand off period I see this logged out in the task:

2019-01-30T14:18:16,507 INFO [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - Pausing ingestion until resumed
2019-01-30T14:18:16,512 INFO [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - Pausing ingestion until resumed

and any further data sent to the stream is not indexed. If the supervisor task is suspended and resumed, then the task starts reading and indexing the kafka stream from the last checkpoint, and no data loss is seen.

I’ll open a new issue for this if it’s not already a known problem

Dyana