Published segments aren't being handed off

The tasks that the supervisor is creating never finish, and sometimes the historical node will stop in ambari. The tasks will get to “Still waiting for handoff for Segments…” and then just stall and nothing happens. I’m not really familiar with all this so I don’t really know where to start or where to look

2018-08-13T20:22:02,353 INFO [publish-0] io.druid.segment.realtime.appenderator.AppenderatorDriver - Published segments, awaiting handoff.

2018-08-13T20:22:02,356 INFO [task-runner-0-priority-0] io.druid.segment.realtime.appenderator.AppenderatorDriver - Register handoff of segments: [[AweFormSubmit-IndexService_2018-08-13T15:00:00.000Z_2018-08-13T16:00:00.000Z_2018-08-13T19:02:08.263Z_27, AweFormSubmit-IndexService_2018-08-13T15:00:00.000Z_2018-08-13T16:00:00.000Z_2018-08-13T19:02:08.263Z_29, AweFormSubmit-IndexService_2018-08-13T15:00:00.000Z_2018-08-13T16:00:00.000Z_2018-08-13T19:02:08.263Z_28, AweFormSubmit-IndexService_2018-08-13T16:00:00.000Z_2018-08-13T17:00:00.000Z_2018-08-13T20:21:10.250Z_2, AweFormSubmit-IndexService_2018-08-13T16:00:00.000Z_2018-08-13T17:00:00.000Z_2018-08-13T20:21:10.250Z_1, AweFormSubmit-IndexService_2018-08-13T16:00:00.000Z_2018-08-13T17:00:00.000Z_2018-08-13T20:21:10.250Z]]

2018-08-13T20:22:02,358 INFO [task-runner-0-priority-0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Adding SegmentHandoffCallback for dataSource[AweFormSubmit-IndexService] Segment[SegmentDescriptor{interval=2018-08-13T15:00:00.000Z/2018-08-13T16:00:00.000Z, version=‘2018-08-13T19:02:08.263Z’, partitionNumber=27}]

2018-08-13T20:22:02,358 INFO [task-runner-0-priority-0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Adding SegmentHandoffCallback for dataSource[AweFormSubmit-IndexService] Segment[SegmentDescriptor{interval=2018-08-13T15:00:00.000Z/2018-08-13T16:00:00.000Z, version=‘2018-08-13T19:02:08.263Z’, partitionNumber=29}]

2018-08-13T20:22:02,358 INFO [task-runner-0-priority-0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Adding SegmentHandoffCallback for dataSource[AweFormSubmit-IndexService] Segment[SegmentDescriptor{interval=2018-08-13T15:00:00.000Z/2018-08-13T16:00:00.000Z, version=‘2018-08-13T19:02:08.263Z’, partitionNumber=28}]

2018-08-13T20:22:02,358 INFO [task-runner-0-priority-0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Adding SegmentHandoffCallback for dataSource[AweFormSubmit-IndexService] Segment[SegmentDescriptor{interval=2018-08-13T16:00:00.000Z/2018-08-13T17:00:00.000Z, version=‘2018-08-13T20:21:10.250Z’, partitionNumber=2}]

2018-08-13T20:22:02,358 INFO [task-runner-0-priority-0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Adding SegmentHandoffCallback for dataSource[AweFormSubmit-IndexService] Segment[SegmentDescriptor{interval=2018-08-13T16:00:00.000Z/2018-08-13T17:00:00.000Z, version=‘2018-08-13T20:21:10.250Z’, partitionNumber=1}]

2018-08-13T20:22:02,358 INFO [task-runner-0-priority-0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Adding SegmentHandoffCallback for dataSource[AweFormSubmit-IndexService] Segment[SegmentDescriptor{interval=2018-08-13T16:00:00.000Z/2018-08-13T17:00:00.000Z, version=‘2018-08-13T20:21:10.250Z’, partitionNumber=0}]

2018-08-13T20:22:58,363 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://dl-data008-uat.xxxxxxxxxxx.com:8081

2018-08-13T20:22:58,364 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://dl-data008-uat.xxxxxxxxxxx.com:8081

2018-08-13T20:22:58,364 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://dl-data008-uat.xxxxxxxxxxx.com:8081

2018-08-13T20:22:58,364 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://dl-data008-uat.xxxxxxxxxxx.com:8081

2018-08-13T20:22:58,365 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://dl-data008-uat.xxxxxxxxxxx.com:8081

2018-08-13T20:22:58,365 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://dl-data008-uat.xxxxxxxxxxx.com:8081

2018-08-13T20:22:58,365 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://dl-data008-uat.xxxxxxxxxxx.com:8081

2018-08-13T20:22:58,366 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://dl-data008-uat.xxxxxxxxxxx.com:8081

2018-08-13T20:22:58,366 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://dl-data008-uat.xxxxxxxxxxx.com:8081

2018-08-13T20:22:58,366 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://dl-data008-uat.xxxxxxxxxxx.com:8081

2018-08-13T20:22:58,366 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://dl-data008-uat.xxxxxxxxxxx.com:8081

2018-08-13T20:22:58,366 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://dl-data008-uat.xxxxxxxxxxx.com:8081

2018-08-13T20:22:58,367 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://dl-data008-uat.xxxxxxxxxxx.com:8081

2018-08-13T20:22:58,368 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://dl-data008-uat.xxxxxxxxxxx.com:8081

2018-08-13T20:22:58,368 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://dl-data008-uat.xxxxxxxxxxx.com:8081

2018-08-13T20:22:58,368 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://dl-data008-uat.xxxxxxxxxxx.com:8081

2018-08-13T20:22:58,369 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://dl-data008-uat.xxxxxxxxxxx.com:8081

2018-08-13T20:22:58,370 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://dl-data008-uat.xxxxxxxxxxx.com:8081

2018-08-13T20:22:58,370 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://dl-data008-uat.xxxxxxxxxxx.com:8081

2018-08-13T20:22:58,370 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://dl-data008-uat.xxxxxxxxxxx.com:8081

2018-08-13T20:22:58,450 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2018-08-13T15:00:00.000Z/2018-08-13T16:00:00.000Z, version=‘2018-08-13T19:02:08.263Z’, partitionNumber=27}, SegmentDescriptor{interval=2018-08-13T16:00:00.000Z/2018-08-13T17:00:00.000Z, version=‘2018-08-13T20:21:10.250Z’, partitionNumber=0}, SegmentDescriptor{interval=2018-08-13T16:00:00.000Z/2018-08-13T17:00:00.000Z, version=‘2018-08-13T20:21:10.250Z’, partitionNumber=1}, SegmentDescriptor{interval=2018-08-13T15:00:00.000Z/2018-08-13T16:00:00.000Z, version=‘2018-08-13T19:02:08.263Z’, partitionNumber=28}, SegmentDescriptor{interval=2018-08-13T15:00:00.000Z/2018-08-13T16:00:00.000Z, version=‘2018-08-13T19:02:08.263Z’, partitionNumber=29}, SegmentDescriptor{interval=2018-08-13T16:00:00.000Z/2018-08-13T17:00:00.000Z, version=‘2018-08-13T20:21:10.250Z’, partitionNumber=2}]]

2018-08-13T20:23:58,398 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2018-08-13T15:00:00.000Z/2018-08-13T16:00:00.000Z, version=‘2018-08-13T19:02:08.263Z’, partitionNumber=27}, SegmentDescriptor{interval=2018-08-13T16:00:00.000Z/2018-08-13T17:00:00.000Z, version=‘2018-08-13T20:21:10.250Z’, partitionNumber=0}, SegmentDescriptor{interval=2018-08-13T16:00:00.000Z/2018-08-13T17:00:00.000Z, version=‘2018-08-13T20:21:10.250Z’, partitionNumber=1}, SegmentDescriptor{interval=2018-08-13T15:00:00.000Z/2018-08-13T16:00:00.000Z, version=‘2018-08-13T19:02:08.263Z’, partitionNumber=28}, SegmentDescriptor{interval=2018-08-13T15:00:00.000Z/2018-08-13T16:00:00.000Z, version=‘2018-08-13T19:02:08.263Z’, partitionNumber=29}, SegmentDescriptor{interval=2018-08-13T16:00:00.000Z/2018-08-13T17:00:00.000Z, version=‘2018-08-13T20:21:10.250Z’, partitionNumber=2}]]

2018-08-13T20:24:58,395 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2018-08-13T15:00:00.000Z/2018-08-13T16:00:00.000Z, version=‘2018-08-13T19:02:08.263Z’, partitionNumber=27}, SegmentDescriptor{interval=2018-08-13T16:00:00.000Z/2018-08-13T17:00:00.000Z, version=‘2018-08-13T20:21:10.250Z’, partitionNumber=0}, SegmentDescriptor{interval=2018-08-13T16:00:00.000Z/2018-08-13T17:00:00.000Z, version=‘2018-08-13T20:21:10.250Z’, partitionNumber=1}, SegmentDescriptor{interval=2018-08-13T15:00:00.000Z/2018-08-13T16:00:00.000Z, version=‘2018-08-13T19:02:08.263Z’, partitionNumber=28}, SegmentDescriptor{interval=2018-08-13T15:00:00.000Z/2018-08-13T16:00:00.000Z, version=‘2018-08-13T19:02:08.263Z’, partitionNumber=29}, SegmentDescriptor{interval=2018-08-13T16:00:00.000Z/2018-08-13T17:00:00.000Z, version=‘2018-08-13T20:21:10.250Z’, partitionNumber=2}]]

2018-08-13T20:25:58,397 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2018-08-13T15:00:00.000Z/2018-08-13T16:00:00.000Z, version=‘2018-08-13T19:02:08.263Z’, partitionNumber=27}, SegmentDescriptor{interval=2018-08-13T16:00:00.000Z/2018-08-13T17:00:00.000Z, version=‘2018-08-13T20:21:10.250Z’, partitionNumber=0}, SegmentDescriptor{interval=2018-08-13T16:00:00.000Z/2018-08-13T17:00:00.000Z, version=‘2018-08-13T20:21:10.250Z’, partitionNumber=1}, SegmentDescriptor{interval=2018-08-13T15:00:00.000Z/2018-08-13T16:00:00.000Z, version=‘2018-08-13T19:02:08.263Z’, partitionNumber=28}, SegmentDescriptor{interval=2018-08-13T15:00:00.000Z/2018-08-13T16:00:00.000Z, version=‘2018-08-13T19:02:08.263Z’, partitionNumber=29}, SegmentDescriptor{interval=2018-08-13T16:00:00.000Z/2018-08-13T17:00:00.000Z, version=‘2018-08-13T20:21:10.250Z’, partitionNumber=2}]]

``

Hi,
Looks like your historical node going down is causing the handoff to NOT complete.

Check for any errors in the historical node logs that is going down.

Even when the historical node stays up, the handoff doesn’t complete. Could it still be the cause?

There can be multiple reasons for handoff not working.
Please check the steps mentioned here - http://druid.io/docs/latest/ingestion/faq.html (See Section - My stream ingest is not handing segments off)

Hopefully it will help in finding the root cause.

I had the same problem because I set up the datasource rules. One rule was to keep the data for one month and the other was to delete all the rules

在 2018年8月14日星期二 UTC+8上午4:35:52,Mark Shannon写道: