Load rule forever

Hi guys,

What means the loadForever load rule?

Does it mean that it will load as many segments as fit in the segments cache? If yes, what’s the order? Are they loaded in any order?

Thanks, Vadim.

Also, if i have no granularity (raw data) and loadForever rule without replication, will the coordinator balance the segments accros histo nodes?

Hi Vadim,

Yes, the loadForever load rule means the coordinator will issue requests to historical nodes such that all the segments in the metadata segment list for all of time are loaded. This would be in contrast to rules that only load segments that match a certain time interval or period.

If you run out of disk space in your segment cache, Druid will stop loading segments and start throwing alerts. This is a bad situation as Druid is only able to serve queries from segments available in cache and not directly from deep storage. You will want to monitor the capacity of your historicals to make sure you have sufficient space to handle the load, and if you need more space, simply start up more historical nodes and the coordinator will detect and rebalance automatically.

To answer your order question, in my experience, the coordinator seems to load segments in reverse chronological order, but again, make sure you have sufficient capacity for your segment cache as partial loading is not a supported scenario.

To your second question - yes, the coordinator has algorithms to distribute segments across the cluster of historical nodes to optimize query time. If you had replication, the coordinator would also take this into account by ensuring that segment replicas are loaded on different nodes.