making segments in datasource visible again

Hi Druid users,
I have a following situation:
I had realtime crash due to Full GC allocation Failure and end up with “holes” of data. the following is what I have:

  1. I have a segment pushed deep storage:
    /user/druid/prod_my_events/20161016T000000.000Z_20161016T060000.000Z
  2. I do not have segment as seen by coordinator:
    curl myServer:9082/druid/coordinator/v1/datasources/prod_my_events/segments/prod_my_events_2016-10-16T00:00:00.000Z_2016-10-16T06:00:00.000Z_2016-10-16T00:00:00.000Z
    —> 204 No Content
  3. But I have metadata of this segment as seen by coordinator:
    curl myServer:9082/druid/coordinator/v1/metadata/datasources/prod_my_events/segments/prod_my_events_2016-10-16T00:00:00.000Z_2016-10-16T06:00:00.000Z_2016-10-16T00:00:00.000Z
    {“dataSource”:“prod_my_events”,“interval”:“2016-10-16T00:00:00.000Z/2016-10-16T06:00:00.000Z”,“version”:“2016-10-16T00:00:00.000Z”,“loadSpec”:{“type”:“hdfs”,“path”:"/user/druid/prod_my_events/20161016T000000.000Z_20161016T060000.000Z/2016-10-16T00_00_00.000Z/0/index.zip"},“dimensions”:"…",“metrics”:“rows”,“shardSpec”:{“type”:“none”},“binaryVersion”:9,“size”:166760441,“identifier”:“prod_my_events_2016-10-16T00:00:00.000Z_2016-10-16T06:00:00.000Z_2016-10-16T00:00:00.000Z”}
    I try to enable the segment without success by
    curl -X POST myServer:9082/druid/coordinator/v1/datasources/prod_my_events/segments/prod_my_events_2016-10-16T00:00:00.000Z_2016-10-16T06:00:00.000Z_2016-10-16T00:00:00.000Z

Any suggestions?

Thank in advance!!!
Best,
Pawel

have you tried restarting the realtime node with increasing the heap size ?
FWIW, when a realtime node restarts, it tries to load data which is already persisted onto the disk from previous run.

Hi Nishant,

I tried to restart with increased heap size, but the realtime did not managed to load data already persisted in /tmp from previous run.
I ended up with several hours hole (segment is 6 hours). The adjacent segments are present with absent data on their boundaries.