Historical Node showing on Coordinator UI after long time post druid restart

Hi,

When we restart druid on historical Node which has around 10 TB of data . Its taking around 2 to 3 hours to show up on coordinator UI . I am seeing loading segments happening from historical logs .

INFO [Segment-Load-Startup-8] io.druid.server.coordination.SegmentLoadDropHandler - Loading segment[11846/35805]

Please let me know how can we make this loading segment process faster .

Is this Loading segments is from Deep storage or from disk to memory ?

Hi,

The historical won’t be visible to the coordinator while it’s still booting up. So the historical loads segments from it’s local disk, not deep storage on boot up and this can take some time depending upon how many segments it has to load and disk speed. Do you see any exceptions in your historical logs or OOM errors ?

Hi Surekha,

Thanks for your reply. I don’t see any exceptions but just wanted to check how can i speed up the operations is there any parameter or that sort

Hi,

Just wondering if you are compacting the segments. That keeps a check on the growing number of segments, and may also help reducing the total segment counts if there are multiple segments for the same interval.

Hi Krishna,

May I know which version of druid you are on?

You can use druid.announcer.type as http . By default this is zk. From 0.14 onwards this is changed to druid.serverview.type

I am sharing a part of communication between me and my colleague Muthu. Thanks Muthu for sharing this awsome feature. Sorry I am lazy to type it again .

Using HTTP based segment discovery on Coordinators and Brokers.

To use HTTP based segment discovery on Coordinators and Brokers, please set this setting in the coordinator and broker runtime properties. And only restart the coordinator and brokers

druid.serverview.type=http

NOTE - the druid.announcer.type is changed to druid.serverview.type from 0.14.0 version onwards.

Once you make the above change, the coordinator and broker will use http to talk to historical nodes and realtime nodes to figure out the segments they are loading.

Using HTTP based segment load/drop management at Coordinators

Please set the below property in coordinator runtime properties to enable segment load and drop management via http (instead of zk)

druid.coordinator.loadqueuepeon.type=http

Optionally the below parameter can be set for processing multiple load/drop requests in parallel

druid.coordinator.loadqueuepeon.http.batchSize

Using HTTP based Task management at Overlord

If you need overlord also not to do ZK based task management. Following can be done. Again note this can be ZK and the above two can still be in http.

Set the below parameter in overlord runtime properties

druid.indexer.runner.type=httpRemote

NOTE

  • httpRemote is an experimental feature based on our doc.
  • All Overlords must be using same taskrunner type config at all times. So, you would update the configuration at all Overlords, then stop all of them, and then start all of them.

Thanks Tijo for the update , we are on .12 version

Hi Krishna,

0.12 is pretty old. The latest version is 0.18 and there has been a lot of improvement since 0.12.

Is there any chance you can upgrade any version post 0.15

Thanks

We are working on upgrading to 0.16

If you have any can you please share some use cases i need to consider while upgrading