Segment handoff stuck

We are ingesting data through tranquility in our Druid cluster. The indexing tasks are stuck. Looking at the logs, I see lines like these:

2019-02-06T22:15:00,004 INFO [ben_test_metrics-overseer-18] io.druid.segment.realtime.plumber.RealtimePlumber - Found [1] segments. Attempting to hand off segments that start before [1970-01-01T00:00:00.000Z].

2019-02-06T22:15:00,004 INFO [ben_test_metrics-overseer-18] io.druid.segment.realtime.plumber.RealtimePlumber - Skipping persist and merge for entry [1549404000000=…

2019-02-06T22:25:39,556 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2019-02-05T22:00:00.000Z/2019-02-05T23:00:00.000Z, version=‘2019-02-05T22:06:43.706Z’, partitionNumber=18}]]

Could someone help me understand why isn’t the handoff happening. And also, what does “Attempting to hand off segments that start before [1970-01-01T00:00:00.000Z]” mean?

The payload for the job looks like this:

“granularitySpec”: {

“type”: “uniform”,

“segmentGranularity”: “HOUR”,

“queryGranularity”: {

“type”: “duration”,

“duration”: 60000,

“origin”: “1970-01-01T00:00:00.000Z”

},

“rollup”: true,

“intervals”: null

}, “transformSpec”: {

“filter”: null,

“transforms”:

}

}, “tuningConfig”: {

“type”: “realtime”,

“maxRowsInMemory”: 1000000,

“intermediatePersistPeriod”: “PT5M”,

“windowPeriod”: “PT15M”,

“basePersistDirectory”: “/apps/druid/tmp/1549404389534-0”

Hi Samarth,

what version of druid were you running? And what was your rejectionPolicy?

what does “Attempting to hand off segments that start before [1970-01-01T00:00:00.000Z]” mean?

The ‘before’ timestamp is the start time of the window period. The realtime task will publish the segments outside of the window period.

This is calculated like this: https://github.com/apache/incubator-druid/blob/master/server/src/main/java/org/apache/druid/segment/realtime/plumber/RealtimePlumber.java#L818-L827.

BTW, if you’re ingesting from Kafka, please consider migrating to Kafka indexing service. It’s more powerful and stable than tranquility.

Jihoon

Hi Jihoon,

We are using druid 0.12.2. There is no rejection policy set.

"rejectionPolicy":{"type":"none"}

Any insights as to why the handoff may not be happening?

2019-02-06T22:25:39,556 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2019-02-05T22:00:00.000Z/2019-02-05T23:00:00.000Z, version=‘2019-02-05T22:06:43.706Z’, partitionNumber=18}]]

Thanks, but I’m not sure why handoff didn’t happen.

Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2019-02-05T22:00:00.000Z/2019-02-05T23:00:00.000Z, version=‘2019-02-05T22:06:43.706Z’, partitionNumber=18}]]

Handoff would complete once any historical loads this segment in its local disk. The coordinator is responsible for assigning segments to historicals.

Do you see any errors about this segment in the coordinator log?

Jihoon