Druid coordinator behaviour on historical startup

Hi ,
I am using druid-0.12.3.My understanding is that when we stop a historical node, the coordinator treats the segments present on that node as missing
and places them on loadqueue path of other historicals in zookeeper ,so that they can load it.
On historical node startup, it tries to memory map the segments present in it’s local directory.

The issue I have observed is after the historical node has successfully joined the cluster and the data is completely loaded in druid(by checking the coordinator UI for any red or yellow mark on datasources)
I still see that the coordinator is trying to load segments(very large number equal to the size of the historical) on to the cluster.This leads to an increase in the handoff time of ingestion tasks and some also fail.
Restarting the Master coordinator fixes this issue.
Is there any solution other than restarting the coordinator ?

Sharath,

Once all historicals are up and they still have the segments, that will be relayed back to the coordinator and coordinator eventually drops the segments already present in historicals from the queue. How long was the queue keeping the segments to be loaded?

Rommel Garcia

Hi,

sorry for the delay.I have collected metrics related to the coordinator like “segment/loadQueue/count” etc…It took around 3-4 hours for the value to reach 0.
After checking the particular historical node logs, I saw that it was receiving load requests from coordinator and it was checking whether if the segment already exists in the cluster and skipped it if does .