Index task creation took long time after restart of middleManager/ tranquility/overlord

Here is my situation…
I had to restart the services to address “java.lang.OutOfMemoryError: unable to create new native thread” by setting ulimit - u on middleManager nodes ( Hope this will resole the issue) .

After restart of middleManager, Tranquility and Overloard services the realtime index task creation took very long time in spite f the fact that my incoming data is current. Following is my tranquility log

2017-05-09 15:47:13,734 [ClusteredBeam-ZkFuturePool-25921be3-2765-4e93-9f56-b02f29d775fc] INFO c.m.tranquility.beam.ClusteredBeam - Global latestCloseTime[2017-05-09T15:00:00.000Z] for identifier[druid:overlord/WalmartLabsLiveStream] has moved past timestamp[2017-05-09T15:00:00.000Z], not creating merged beam

2017-05-09 15:47:13,738 [ClusteredBeam-ZkFuturePool-25921be3-2765-4e93-9f56-b02f29d775fc] INFO c.m.tranquility.beam.ClusteredBeam - Turns out we decided not to actually make beams for identifier[druid:overlord/WalmartLabsLiveStream] timestamp[2017-05-09T15:00:00.000Z]. Returning None.

Under completed task ( On Coordinator/Overload console) I see all hung tasks as failed and I assume its a expected because of service restarts.

My question is, why it took so long to create new realtime index task for fresh/latest in coming data ?, is it trying to recover the failed task(s)?. If Yes, can I override this feature ?

Thx and Rgds


ulimit setting resolved the “unable to create new native thread” issue.